1717051 - Transparently manage Mach virtualenv at run-time

Status: RESOLVED FIXED

(Firefox Build System :: Mach Core, enhancement, P2)

Description

[Mitchell Hentges [:mhentges] 🦀]

1. There are pieces of the mach virtualenv which are needed for specific commands (zstandard, psutil). Move these into their associated command virtualenv.
2. Instead of having a separate ./mach create-mach-environment -f, manage the mach virtualenv more transparently:
   • Bootstrap Mach by start with just vendored code, build the Mach virtualenv, install native dependencies, activate it, then continue running Mach.
   • Automatically re-create the venv it if we're using a different Python version

Comment 1

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1717051: Remove obsolete python module paths

The three removed paths don't exist in-repo, and after a cursory glance
they don't appear to be populated dynamically.

Note that the removal of the six path for WPT is different: it's
technically just incorrect, and should amended to point to
$WPT/tools/third_party/six. However, Python only allows a single
instance of a library to exist in import scope, and we're already
consuming six from the Firefox-wide vendored 3rd-party libs.

Depends on D119687

Comment 2

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1717051: Move first-party module list outside of requirement def

1st-party and 3rd-party module references are two different concerns.
This is most clear when considering that, for any given revision, every
command is going to want the 1st-party modules contained in that
revision - they (shouldn't) ever want, say, mozversioncontrol from
last year.

However, it's completely viable for two different commands to depend on
different (perhaps even conflicting) versions of a third_party module
(such as six).

So, we centralize the definition of 1st-party modules accordingly.

Depends on D119825

Comment 3

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1717051: Only use stdlib for installing pip packages

_install_pip_package() may be run from populate(), which
is invoked from a child Python process that doesn't
have in-tree Python modules in its sys.path.

An alternate solution would be to add in-tree modules
to the sys.path, but that seemed more costly than
simply using tempfile and shutil.

Depends on D119826

Comment 4

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1717051: Move global mach dependencies to requirements definition

After removing optional in Bug 1712804, we need to add a variant back
here because there's fallible dependencies. However, I've tweaked the
re-introduction of the feature to require a specific repercussion
message as well. This seemed like a decent tradeoff - the developer
becomes aware that the failure is bad, it has repercussions, but it's
not a blocking issue. Additionally, since we're printing pip's output,
the developer will be able to see the underlying error causing the
warning.

I also added comment functionality to requirements definitions to allow
adjacent documentation of why some requirements are fallible. (Related:
I'm looking forward to mach_bootstrap not needing to parse
requirements definitions. Almost there!)

Finally, in preparation for review: I didn't make
FalliblePypiSpecifier extend PypiSpecifier because I figured that
the benefit of flexibility (easier to allow implementations to diverge
without needing to untangle an inheritance relationship) was larger than
the cost of needing to add properties to both specifiers.
If we wanted re-use, I'd probably have FalliblePypiSpecifier contain
a PypiSpecifier, but then you have to reach deeper into the object to
get data, so shrug.

Depends on D119834

Comment 5

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Existing_Mach_bootstrap_with_requirements.png

Comment 6

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Target_mach_bootstrap_with_requirements.png

Comment 7

[Mitchell Hentges [:mhentges] 🦀]

Attached some images (originally diagrammed for this patch) to help illustrate the target of this bug.

Comment 8

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1717051: Remove "install-moz-phab" from nativecmds

It was in nativecmds to make finding pip3 easier.
However, since we're removing the distinction between "regular" and
"native" cmds (Mach will create and activate its venv naturally during
initialization), we need to start pruning the nativecmds list.

Depends on D119835

Comment 9

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1717051: Move arm64 rosetta check to bootstrap

It was originally added to "create-mach-environment" because that would
eventually be called by "./mach bootstrap".
However, as "create-mach-environment" is going away, this moves it
closer to the command that it's meant to impact.

Depends on D120398

Comment 10

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1717051: Consolidate state_dir creation

We had logic in both mach_bootstrap and the Mach Bootstrapper to
create the state_dir.

This joins them and has the added benefit of creating the state dir
earlier in the Mach lifecycle (as will be needed for early instantiation
of the Mach virtualenv).

Depends on D120399

Comment 11

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1717051: Automatically create and activate Mach virtualenv

TODO

Depends on D120400

Comment 12

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1717051: Always populate Mach pths

Now that we no longer patch sys.path when using the Mach virtualenv,
the populate_local_paths option is no longer ever False. So, it can
be inlined as True.

Depends on D120401

Comment 13

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1717051: Update `pkg_resources`, `sys.executable` when activating venvs

pkg_resources parses the environment state when it's first imported.
However, when we activate an existing virtualenv, pkg_resources isn't
automatically updated.

The current symptom of this is that the PyCharm debugger causes failures
when a virtualenv boundary is passed, because it imports pkg_resources
very early in the debugging lifecycle.

Depends on D120402

Comment 14

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1717051: Set sys.executable to Mach virtualenv

Warning: this solution may not be stable as more dependencies are added.
However, at the moment, it seems like the best set of tradeoffs.

Glean creates a child Python process that needs to be able to import glean. So, it creates that process from sys.executable with the
assumption that "sys.executable has the current Glean module imported,
so it must be able to do it again."

However, sys.executable is not changed when dynamically activating
virtualenvs from an existing Python process. So, without this change,
sys.executable is the system Python, which isn't guaranteed to have
Glean.

Unfortunately, this solution only works because we currently only have
one dependency using sys.executable. If we have this case in the
future with a dependency in a command virtualenv, then this solution
won't work anymore.

This leads us to an alternative solution (which isn't currently
adopted): we could install all Mach-level dependencies into every
command virtualenv. This has some benefits:

• Any dependency of Mach or any command can start subprocesses from
  sys.executable, and it'll always run from a virtualenv with that
  dependency installed.
• We're already going to have to validate that the dependencies of
  commands don't conflict with Mach dependencies. The current planned
  solution is to do that validation only in CI, but this would simplify
  that situation.
• The virtualenv developers have stated that dynamically activating a
  virtualenv from an existing Python process isn't "switching"
  virtualenvs, but rather adding its packages to the current
  environment. Installing Mach dependencies into all command virtualenvs
  would align with the additive nature here.

However, there's one big downside: performance. Installing modules,
especially native modules, can be slow. Having to set up glean,
zstandard and psutil into all command virtualenvs is inefficient and
will waste developer time. So, until it's needed, the manual
sys.executable management appears to be the more effective solution.

Depends on D120404

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6597db537fa2
Remove obsolete python module paths r=ahal
https://hg.mozilla.org/integration/autoland/rev/4f32438176a0
Only use stdlib for installing pip packages r=ahal
https://hg.mozilla.org/integration/autoland/rev/dbdd100c60d0
Move global mach dependencies to requirements definition r=ahal

Comment 16

[Cosmin Sabou [:CosminS]]

Backed out for busting the gecko decision task.

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&revision=dbdd100c60d0200f4ebb32c5af6794b4daf6d3fa&selectedTaskRun=brUZkaLtQpSXtB3jn7d0_A.0

Failure log: https://treeherder.mozilla.org/logviewer?job_id=345811009&repo=autoland

Backout link: https://hg.mozilla.org/integration/autoland/rev/7b7a049fb3e3979546f707a50e12dc9aacf59f78

Comment 17

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1717051: Rename "mach_bootstrap.py" to "mach_initialize.py"

We've overloaded "bootstrap" to mean three different things:

• The "standalone bootstrap script": python/mozboot/bin/bootstrap.py.
  This is to freshly clone a new repo, then run ./mach bootstrap.
• ./mach bootstrap: Install necessary dependencies and set up the
  system for development.
• "Mach bootstrap": do the work Mach needs before it can run commands.

By using the term "initialize" instead, perhaps we can remove
ambiguity when discussing Mach.

I'm not attached to the name (or this change at all), but I'm interested
in reviewer thoughts :)

Depends on D120398

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d1333fe69cf4
Remove obsolete python module paths r=ahal
https://hg.mozilla.org/integration/autoland/rev/605651561f2a
Only use stdlib for installing pip packages r=ahal
https://hg.mozilla.org/integration/autoland/rev/fb9919b26f30
Move global mach dependencies to requirements definition r=ahal

Comment 19

[Sandor Molnar[:smolnar]]

Backed out for causing win debug build bustages

Backout link: https://hg.mozilla.org/integration/autoland/rev/d716918916ac1a4993a616f07d501f9b64be467d

Push with failures

Failure log

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b522a5b7a10e
Remove obsolete python module paths r=ahal
https://hg.mozilla.org/integration/autoland/rev/d8de22311420
Only use stdlib for installing pip packages r=ahal
https://hg.mozilla.org/integration/autoland/rev/1a12fa683af8
Move global mach dependencies to requirements definition r=ahal

Comment 21

[Natalia Csoregi [:nataliaCs]]

Backed out for causing gecko decision failures.

• backout: https://hg.mozilla.org/integration/autoland/rev/eae583a154ad574dde8c4203f3402ae3f980c494

• push: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&revision=1a12fa683af84dbb2b8cb79765ff2d9ee4d21288

• failure log: https://treeherder.mozilla.org/logviewer?job_id=346814802&repo=autoland&lineNumber=634

[task 2021-07-29T23:02:40.686Z] Writing docker-contexts/ubuntu1804-test.tar.gz for docker image ubuntu1804-test
[task 2021-07-29T23:02:40.686Z] Error loading tasks for kind docker-image:
[task 2021-07-29T23:02:40.687Z] Traceback (most recent call last):
[task 2021-07-29T23:02:40.687Z]   File "/builds/worker/checkouts/gecko/taskcluster/mach_commands.py", line 354, in taskgraph_decision
[task 2021-07-29T23:02:40.687Z]     ret = taskgraph.decision.taskgraph_decision(options)
[task 2021-07-29T23:02:40.687Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/decision.py", line 227, in taskgraph_decision
[task 2021-07-29T23:02:40.687Z]     full_task_json = tgg.full_task_graph.to_json()
[task 2021-07-29T23:02:40.687Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 174, in full_task_graph
[task 2021-07-29T23:02:40.687Z]     return self._run_until("full_task_graph")
[task 2021-07-29T23:02:40.687Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 435, in _run_until
[task 2021-07-29T23:02:40.687Z]     k, v = next(self._run)
[task 2021-07-29T23:02:40.687Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 312, in _run
[task 2021-07-29T23:02:40.687Z]     self._write_artifacts,
[task 2021-07-29T23:02:40.687Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 92, in load_tasks
[task 2021-07-29T23:02:40.687Z]     for task_dict in transforms(trans_config, inputs)
[task 2021-07-29T23:02:40.687Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/generator.py", line 80, in <listcomp>
[task 2021-07-29T23:02:40.687Z]     Task(
[task 2021-07-29T23:02:40.687Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/task.py", line 2120, in check_run_task_caches
[task 2021-07-29T23:02:40.687Z]     for task in tasks:
[task 2021-07-29T23:02:40.687Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/task.py", line 2028, in check_task_dependencies
[task 2021-07-29T23:02:40.687Z]     for task in tasks:
[task 2021-07-29T23:02:40.687Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/task.py", line 2014, in check_task_identifiers
[task 2021-07-29T23:02:40.687Z]     for task in tasks:
[task 2021-07-29T23:02:40.687Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/task.py", line 1995, in chain_of_trust
[task 2021-07-29T23:02:40.687Z]     for task in tasks:
[task 2021-07-29T23:02:40.687Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/task.py", line 1786, in build_task
[task 2021-07-29T23:02:40.687Z]     for task in tasks:
[task 2021-07-29T23:02:40.687Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/task.py", line 1777, in try_task_config_routes
[task 2021-07-29T23:02:40.687Z]     for task in tasks:
[task 2021-07-29T23:02:40.687Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/task.py", line 1767, in try_task_config_chemspill_prio
[task 2021-07-29T23:02:40.687Z]     for task in tasks:
[task 2021-07-29T23:02:40.687Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/task.py", line 1757, in try_task_config_env
[task 2021-07-29T23:02:40.688Z]     for task in tasks:
[task 2021-07-29T23:02:40.688Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/task.py", line 1719, in add_index_routes
[task 2021-07-29T23:02:40.688Z]     for task in tasks:
[task 2021-07-29T23:02:40.688Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/task.py", line 1514, in validate
[task 2021-07-29T23:02:40.688Z]     for task in tasks:
[task 2021-07-29T23:02:40.688Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/task.py", line 1491, in task_name_from_label
[task 2021-07-29T23:02:40.688Z]     for task in tasks:
[task 2021-07-29T23:02:40.688Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/task.py", line 1446, in set_defaults
[task 2021-07-29T23:02:40.688Z]     for task in tasks:
[task 2021-07-29T23:02:40.688Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/task.py", line 1425, in set_implementation
[task 2021-07-29T23:02:40.688Z]     for task in tasks:
[task 2021-07-29T23:02:40.688Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/cached_tasks.py", line 64, in cache_task
[task 2021-07-29T23:02:40.688Z]     for task in order_tasks(config, tasks):
[task 2021-07-29T23:02:40.688Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/cached_tasks.py", line 19, in order_tasks
[task 2021-07-29T23:02:40.688Z]     pending = deque(tasks)
[task 2021-07-29T23:02:40.688Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/transforms/docker_image.py", line 108, in fill_template
[task 2021-07-29T23:02:40.688Z]     GECKO, context_path, context_file, image_name, args
[task 2021-07-29T23:02:40.688Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/util/docker.py", line 216, in create_context_tar
[task 2021-07-29T23:02:40.688Z]     args=args,
[task 2021-07-29T23:02:40.688Z]   File "/builds/worker/checkouts/gecko/taskcluster/taskgraph/util/docker.py", line 252, in stream_context_tar
[task 2021-07-29T23:02:40.688Z]     raise Exception("extra include path does not exist: %s" % p)
[task 2021-07-29T23:02:40.688Z] Exception: extra include path does not exist: build/psutil_requirements.txt
[taskcluster 2021-07-29 23:02:41.087Z] === Task Finished ===
[taskcluster 2021-07-29 23:02:41.994Z] Unsuccessful task run with exit code: 1 completed in 23.475 seconds

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/732724b94715
Remove obsolete python module paths r=ahal
https://hg.mozilla.org/integration/autoland/rev/06ae6038c823
Only use stdlib for installing pip packages r=ahal
https://hg.mozilla.org/integration/autoland/rev/d3f2270f5d80
Move global mach dependencies to requirements definition r=ahal

Comment 23

[Noemi Erli[:noemi_erli]]

This was backout out for causing initial backout conflict, apologize for the inconvenience.

Backed out per glandium's request, 7 changesets (Bug 1720215, Bug 1717645, Bug 1717051) for causing build bustage and conflicting backout

Log: https://treeherder.mozilla.org/logviewer?job_id=346816196&repo=autoland&lineNumber=987

Backout: https://hg.mozilla.org/integration/autoland/rev/791ec1c34fb986a24cfcda7c4d98af416299a024

Comment 24

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ae118a8af444
Remove obsolete python module paths r=ahal
https://hg.mozilla.org/integration/autoland/rev/1a328c15cdc0
Only use stdlib for installing pip packages r=ahal
https://hg.mozilla.org/integration/autoland/rev/cb770866fd3d
Move global mach dependencies to requirements definition r=ahal

Comment 25

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/ae118a8af444
https://hg.mozilla.org/mozilla-central/rev/1a328c15cdc0
https://hg.mozilla.org/mozilla-central/rev/cb770866fd3d

Comment 26

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1717051: Spell `glean-sdk` with a dash, not an underscore

The Glean SDK was traditionally represented in-tree as glean_sdk.
However, on PyPI and in pip list, it appears as glean-sdk (pip was
doing the translation implicitly and internally).

This is breaking our virtualenv state-checking code, since it thinks
that glean_sdk isn't installed.

Since glean_sdk was marked as pypi-optional, this hasn't affected us
yet, but it will when we bump the Glean SDK version, because developers'
virtualenvs won't be detected as out-of-date.

Comment 27

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5d72777ab5ab
Spell `glean-sdk` with a dash, not an underscore r=ahal

Comment 28

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/5d72777ab5ab

Comment 29

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1717051: Remove obsolete virtualenv code

The "six"-related comment is no longer relevant, and here isn't used.

Depends on D120402

Comment 30

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1717051: Validate if optional dependencies are out-of-date

Though it's acceptable for an optional dependency to not be installed,
it is a problem if it's installed to the wrong version.

Depends on D122885

Comment 31

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1717051: Attempt to reinstall optional dependencies during bootstrap

Currently, developers don't have a way to have the Mach virtualenv
re-attempt to install optional dependencies (such as glean).

As part of ./mach bootstrap (the general catch-all "re-create my dev
environment" command), we should retry installing optional dependencies.

This also matches the "glean isn't installed" error message
recommendation.

Depends on D123241

Comment 32

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/38f8d0738109
Rename "mach_bootstrap.py" to "mach_initialize.py" r=firefox-build-system-reviewers,glandium
https://hg.mozilla.org/integration/autoland/rev/312e5766e25d
Validate if optional dependencies are out-of-date r=ahal
https://hg.mozilla.org/integration/autoland/rev/b526de57970f
Move arm64 rosetta check to bootstrap r=firefox-build-system-reviewers,glandium
https://hg.mozilla.org/integration/autoland/rev/a046d63a60f9
Consolidate state_dir creation r=firefox-build-system-reviewers,glandium
https://hg.mozilla.org/integration/autoland/rev/d42dd3032386
Remove obsolete virtualenv code r=ahal

Comment 33

[Marian-Vasile Laza]

https://hg.mozilla.org/mozilla-central/rev/38f8d0738109
https://hg.mozilla.org/mozilla-central/rev/312e5766e25d
https://hg.mozilla.org/mozilla-central/rev/b526de57970f
https://hg.mozilla.org/mozilla-central/rev/a046d63a60f9
https://hg.mozilla.org/mozilla-central/rev/d42dd3032386

Comment 34

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1717051: Reuse "requirements" logic in mach_initialize

Rather than re-implementing it as search_path(), use the existing
MachEnvRequirements tool to parse mach_virtualenv_requirements.txt

Depends on D125910

Comment 35

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2aef162b9a1b
Reuse "requirements" logic in mach_initialize r=ahal

Comment 36

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f54ef8a8e3a3
Reuse "requirements" logic in mach_initialize r=ahal

Comment 37

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/f54ef8a8e3a3

Comment 38

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/894fc83e19a0
Always populate Mach pths r=firefox-build-system-reviewers,glandium

Comment 39

[Marian-Vasile Laza]

https://hg.mozilla.org/mozilla-central/rev/894fc83e19a0

Comment 40

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Pulsebot from comment #38)

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/894fc83e19a0
Always populate Mach pths r=firefox-build-system-reviewers,glandium

With this patch landed I'm not able to run any mach command anymore. The failure is:

➜ mach build
Traceback (most recent call last):
  File "/Users/henrik/code/gecko/mach", line 167, in <module>
    main(sys.argv[1:])
  File "/Users/henrik/code/gecko/mach", line 159, in main
    mach = check_and_get_mach(os.path.dirname(os.path.realpath(__file__)))
  File "/Users/henrik/code/gecko/mach", line 146, in check_and_get_mach
    return load_mach(dir_path, mach_path)
  File "/Users/henrik/code/gecko/mach", line 134, in load_mach
    return mach_initialize.initialize(dir_path)
  File "/Users/henrik/code/gecko/build/mach_initialize.py", line 302, in initialize
    import mach.main
  File "/Users/henrik/code/gecko/python/mach/mach/main.py", line 33, in <module>
    from .dispatcher import CommandAction
  File "/Users/henrik/code/gecko/python/mach/mach/dispatcher.py", line 20, in <module>
    from .decorators import SettingsProvider
  File "/Users/henrik/code/gecko/python/mach/mach/decorators.py", line 13, in <module>
    from mozbuild.base import MachCommandBase
ModuleNotFoundError: No module named 'mozbuild'

I requested a backout from the sheriffs.

Comment 41

[Cristina Cozmuta (:CrissCozmuta)]

Backed out for causing busting macth commands. a=backout
Backout link : https://hg.mozilla.org/mozilla-central/rev/26cd1746e43de0cf2cbcb9897cf19a0ee4a2da88

Comment 42

[Mitchell Hentges [:mhentges] 🦀]

Hmm, your mach virtualenv needed to be updated, but we don't have an up_to_date() check happening unless you run ./mach create-mach-environment or ./mach bootstrap.
I'll add an up_to_date() check on startup, then re-land this.

Comment 43

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c6fd83ff13e7
Always populate Mach pths r=firefox-build-system-reviewers,glandium

Comment 44

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/c6fd83ff13e7

Comment 45

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Mitchell Hentges [:mhentges] 🦀 from comment #42)

Hmm, your mach virtualenv needed to be updated, but we don't have an up_to_date() check happening unless you run ./mach create-mach-environment or ./mach bootstrap.
I'll add an up_to_date() check on startup, then re-land this.

Thanks for adding the extra check. It's working fine now and prompts me to run create-mach-environment. Maybe this could be done automatically?

Comment 46

[Mitchell Hentges [:mhentges] 🦀]

Good call, I'm working my way towards it. It's in the patch stack :)

Comment 49

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1717051: Allow in-process Mach site activation

Depends on D129694

Comment 50

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0745ee826522
Allow in-process Mach site activation r=ahal
https://hg.mozilla.org/integration/autoland/rev/d6b4de673f49
Update `pkg_resources`, `sys.executable` when activating venvs r=ahal
https://hg.mozilla.org/integration/autoland/rev/6d2196ff584c
Remove "install-moz-phab" from nativecmds r=ahal
https://hg.mozilla.org/integration/autoland/rev/fc578092665f
Automatically create and activate Mach virtualenv r=ahal

Comment 51

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/0745ee826522
https://hg.mozilla.org/mozilla-central/rev/d6b4de673f49
https://hg.mozilla.org/mozilla-central/rev/6d2196ff584c
https://hg.mozilla.org/mozilla-central/rev/fc578092665f

Comment 52

[Mitchell Hentges [:mhentges] 🦀]

Mission accomplished 😎

Comment 53

[Phabricator Automation]

Comment on attachment 9237261 [details]
Bug 1717051: Attempt to reinstall optional dependencies during bootstrap

Revision D123242 was moved to bug 1747837. Setting attachment 9237261 [details] to obsolete.

Comment 55

[Phabricator Automation]

Comment on attachment 9237261 [details]
Bug 1717051: Attempt to reinstall optional dependencies during bootstrap

Revision D123242 was moved to bug 1747837. Setting attachment 9237261 [details] to obsolete.