1635481 - ./mach configure breaks with ModuleNotFoundError: No module named 'mozfile'

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Axel Hecht [:Pike]]

Attachment: stack shown by mach configure

I'm currently unable to run ./mach configure, and hg bisect attributes that to https://hg.mozilla.org/mozilla-central/rev/407894bc5f9c from bug 1627163. I'm on macos 10.14.

I'm dying on ModuleNotFoundError: No module named 'mozfile'. Stack attached.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1627163

Comment 2

[Ricky Stewart]

I can't reproduce this.

1. Are you still encountering this issue today after rebasing?
2. What version of the code are you running here? This line of the log confuses me because this hasn't been line 344 of init.configure for a few weeks as far as I can tell from the log.

 0:05.13   File "/src/central/mozilla-central/build/moz.configure/init.configure", line 344, in virtualenv_python3
 0:05.13     python, version = find_python3_executable(min_version='3.5.0')

Comment 3

[Axel Hecht [:Pike]]

Attachment: output of `mach configure`

I just updated to try, and pruned things really hard:

```
axelhecht-7njgh6:mozilla-central axelhecht$ hg status -i
I .gradle/5.1.1/executionHistory/executionHistory.bin
I .gradle/5.1.1/executionHistory/executionHistory.lock
I .gradle/5.1.1/fileChanges/last-build.bin
I .gradle/5.1.1/fileContent/fileContent.lock
I .gradle/5.1.1/fileHashes/fileHashes.bin
I .gradle/5.1.1/fileHashes/fileHashes.lock
I .gradle/5.1.1/fileHashes/resourceHashesCache.bin
I .gradle/5.1.1/gc.properties
I .gradle/5.1.1/javaCompile/classAnalysis.bin
I .gradle/5.1.1/javaCompile/jarAnalysis.bin
I .gradle/5.1.1/javaCompile/javaCompile.lock
I .gradle/5.1.1/javaCompile/taskHistory.bin
I .gradle/buildOutputCleanup/buildOutputCleanup.lock
I .gradle/buildOutputCleanup/cache.properties
I .gradle/buildOutputCleanup/outputFiles.bin
I .gradle/vcs-1/gc.properties
I .mozconfig
I .vscode/settings.json
I build/__pycache__/mach_bootstrap.cpython-37.pyc
I js/src/configure
I js/src/old-configure
I mozconfig-artifact-fx
I mozconfig-fennec
I mozconfig-fx
I mozconfig-tb
I obj-firefox/.mozbuild/last_log.json
I testing/mozharness/external_tools/__pycache__/robustcheckout.cpython-37.pyc
```

`hg status` was empty.

`./mach configure` still fails, see the last attachment.

Comment 4

[Axel Hecht [:Pike]]

Source status:

axelhecht-7njgh6:mozilla-central axelhecht$ hg log -r .
changeset:   598360:ec6c8f37b1fa
tag:         tip
fxtree:      central
user:        Tomislav Jovanovic <[e-mail]>
date:        Wed May 06 17:08:11 2020 +0000
summary:     Bug 1635637 - Add contextId to MessageSender, prevent onConnect in same Context r=robwu a=RyanVM

Comment 5

[Ricky Stewart]

I also asked you this on chat.mozilla, but for posterity:

Can you clobber and then do a ./mach -v configure? The extra log output would be helpful.
After you've done so, forward me the contents of your $objdir/config.log please, in addition the the shell output. I basically need to do a bunch of sanity checking right now, because this issue doesn't make a great deal of sense to me.

So what I'm seeing is that everything is progressing as I would expect (configure creates the Python 3 virtualenv and then re-executes configure inside of it, which is what it's meant to do). mozfile is supposed to be importable from within the virtualenv, though (we add it in build/virtualenv_packages.txt). So the first thing I would check is that the re-execution is working as expected and you're actually ending up inside the virtualenv, which the log will help confirm. If that all looks good then your virtualenv is evidently not being set up correctly and that would be the next line of inquiry.

Comment 6

[Axel Hecht [:Pike]]

Attachment: ./mach -v configure

So this is what the output of ./mach -v configure looks like, not a whole lot more.

config.log is

DEBUG: python3: running with pid 19356
DEBUG: python3: sys.executable: '/src/central/mozilla-central/obj-firefox/_virtualenvs/init_py3/bin/python'
DEBUG: python3: executable from configuration: None

The virtualenv is clearly not right:

axelhecht-7njgh6:mozilla-central axelhecht$ ls obj-firefox/_virtualenvs/init_py3/lib/python3.7/site-packages/
__pycache__			pip-19.3.1.dist-info		setuptools-41.6.0.dist-info
easy_install.py			pkg_resources			wheel
pip				setuptools			wheel-0.33.6.dist-info

Comment 7

[Mike Hommey [:glandium]]

did you try mach clobber --full?

Comment 8

[Axel Hecht [:Pike]]

OK, time for drinks.

The installation command in https://searchfox.org/mozilla-central/rev/dc4560dcaafd79375b9411fdbbaaebb0a59a93ac/python/mozbuild/mozbuild/virtualenv.py#509 doesn't do anything if called from mach.
If I call it manually, well, it does.

And then ./mach configure passes.

This is a sibling of bug 1607470, same trick worked again. Patch coming up.

Comment 9

[Axel Hecht [:Pike]]

Attachment: Bug 1635481, workaround python and virtualenv, r=#build

Comment 10

[Axel Hecht [:Pike]]

Re clobber, I did a manual clobber, in the sense of "remove all stuff that's found by hg status -i". I resorted to that when that command had leftover python caches after a ./mach clobber python.

Re the patch, this is the second time I use this hack, and I'm not sure if this is a fix or a whackamole.

Comment 11

[Axel Hecht [:Pike]]

FWIW, there's mention of __PYENV_LAUNCHER__ on the 3.7.7 release notes: https://docs.python.org/3.7/whatsnew/changelog.html#python-3-7-7-final

Comment 12

[Axel Hecht [:Pike]]

Correction, the release notes are for unreleased 3.7.x so far. I haven't nailed down why only I'm hit with this.

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9e00d7167e30
workaround python and virtualenv, r=firefox-build-system-reviewers,rstewart

Comment 14

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/9e00d7167e30

Comment 15

[BugBot [:suhaib / :marco/ :calixte]]

The patch landed in nightly and beta is affected.
:rstewart, is this bug important enough to require an uplift?
If not please set status_beta to wontfix.

For more information, please visit auto_nag documentation.

Comment 16

[Ricky Stewart]

Comment on attachment 9146427 [details]
Bug 1635481, workaround python and virtualenv, r=#build

Beta/Release Uplift Approval Request

• User impact if declined: configure can fail on macOS infrequently. However, when this is encountered, it is persistent and difficult or impossible to fix.
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This doesn't change anything functionally about the way the build works except a little tweak to make sure that virtualenv creation works 100% of the time on macOS.
• String changes made/needed:

Comment 17

[Pascal Chevrel:pascalc (PTO until April 26)]

Comment on attachment 9146427 [details]
Bug 1635481, workaround python and virtualenv, r=#build

Looks safe and no possible impact on end-users, uplift approved for beta, thanks.

Comment 18

[Julien Cristau [:jcristau] (back April 22)]

https://hg.mozilla.org/releases/mozilla-beta/rev/21862397abfa