Setting PYTHON3 in mozconfig causes unconditional configure failure
Categories
(Firefox Build System :: General, defect, P3)
Tracking
(firefox78 fixed)
| Tracking | Status | |
|---|---|---|
| firefox78 | --- | fixed |
People
(Reporter: rstewart, Assigned: glandium)
Details
Attachments
(2 files, 1 obsolete file)
This happens when I add either export PYTHON3=... or ac_add_options PYTHON3=... in my mozconfig.
rickystewart-a5lvdq:mozilla-unified rickystewart$ ./mach configure
0:00.87 Clobber not needed.
0:00.87 Adding make options from /Users/rickystewart/src/mozilla-unified/mozconfig
MOZ_OBJDIR=/Users/rickystewart/src/mozilla-unified/obj-x86_64-apple-darwin19.2.0
OBJDIR=/Users/rickystewart/src/mozilla-unified/obj-x86_64-apple-darwin19.2.0
FOUND_MOZCONFIG=/Users/rickystewart/src/mozilla-unified/mozconfig
export FOUND_MOZCONFIG
0:00.88 /usr/bin/make -f client.mk -s configure
0:00.89 cd /Users/rickystewart/src/mozilla-unified/obj-x86_64-apple-darwin19.2.0
0:00.89 /Users/rickystewart/src/mozilla-unified/configure
0:01.34 Creating Python 3 environment
0:01.43 Using base prefix '/Library/Frameworks/Python.framework/Versions/3.5'
0:01.44 New python executable in /Users/rickystewart/src/mozilla-unified/obj-x86_64-apple-darwin19.2.0/_virtualenvs/init_py3/bin/python3.5
0:01.44 Not overwriting existing python script /Users/rickystewart/src/mozilla-unified/obj-x86_64-apple-darwin19.2.0/_virtualenvs/init_py3/bin/python (you must use /Users/rickystewart/src/mozilla-unified/obj-x86_64-apple-darwin19.2.0/_virtualenvs/init_py3/bin/python3.5)
0:02.13 Installing setuptools, pip, wheel...
0:04.77 done.
0:05.21 b'running build_ext\ncopying build/lib.macosx-10.6-intel-3.5/psutil/_psutil_osx.cpython-35m-darwin.so -> psutil\ncopying build/lib.macosx-10.6-intel-3.5/psutil/_psutil_posix.cpython-35m-darwin.so -> psutil\n'
0:05.22 Error processing command. Ignoring because optional. (optional:packages.txt:comm/build/virtualenv_packages.txt)
0:05.24 Re-executing in the virtualenv
0:05.73 Adding configure options from /Users/rickystewart/src/mozilla-unified/mozconfig
0:05.73 --with-macos-sdk=/Users/rickystewart/src/MacOSX-SDKs/MacOSX10.14.sdk
0:05.73 PYTHON3=/usr/local/bin/python3.5
0:05.73 checking for vcs source checkout... git
0:05.78 checking for a shell... /bin/sh
0:05.88 checking for host system type... x86_64-apple-darwin19.2.0
0:05.88 checking for target system type... x86_64-apple-darwin19.2.0
0:06.30 checking whether cross compiling... no
0:06.41 Traceback (most recent call last):
0:06.41 File "/Users/rickystewart/src/mozilla-unified/configure.py", line 181, in <module>
0:06.41 sys.exit(main(sys.argv))
0:06.41 File "/Users/rickystewart/src/mozilla-unified/configure.py", line 52, in main
0:06.41 sandbox.run(os.path.join(os.path.dirname(__file__), 'moz.configure'))
0:06.41 File "/Users/rickystewart/src/mozilla-unified/python/mozbuild/mozbuild/configure/__init__.py", line 494, in run
0:06.41 raise InvalidOptionError(msg)
0:06.41 mozbuild.configure.options.InvalidOptionError: Unknown option: PYTHON3
0:06.47 *** Fix above errors and then restart with "./mach build"
0:06.47 make: *** [configure] Error 1
|
Reporter | |
Updated•5 years ago
|
|
|
Reporter | |
Comment 1•5 years ago
|
||
Configuration values defined above early_options in init.configure are those that can't be configured in mozconfig. As far as I can tell there is nothing wrong in principle with setting PYTHON3 in mozconfig, so here we just bump early_options above PYTHON3 configuration.
Side note, diagnosing this failure took a long time. The error message that this produced ("unknown option PYTHON3") is useless and the underlying algos being extremely mutable, I ended up having to spend a lot of time in the debugger and monitor all the underlying changes to a bunch of mutable data structures to determine where PYTHON3 was getting lost. A better error message would be good here but I don't know how I would begin adding it.
|
||
Updated•5 years ago
|
|
||
Comment 3•5 years ago
|
||
Backed out for build bustages.
Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=303114171&repo=autoland&lineNumber=771
Backout: https://hg.mozilla.org/integration/autoland/rev/8dba8b454ae100a6e7f5e2ea2745f6c19c928d7a
|
Assignee | |
Comment 4•5 years ago
|
||
What went wrong with this change is that mozconfig is handled once, which injects its data, then the virtualenv is initialized and configure re-executed, which re-handles mozconfig, with interesting results, e.g. screwing up variables like LIB or CFLAGS because of additional msys->win32 transitions.
Without the change, mozconfig is handled once and no variables are altered.
Now, looking back at comment 0, and looking at virtualenv_python3, setting PYTHON3 in the mozconfig is meant to be handled already. The handling was added explicitly back in bug 1253502... and it worked (back then, it was for PYTHON, though).
A bisect later, this turned out to break with bug 1264527, but that's not entirely helpful to understand what's going on.
So, what's going on:
- Python configure has a record of all the options it's meant to be handling. When each of them is handled, they are removed from the record. Once they are all handled, what's supposed to be left in the record is unknown options. That's the meaning of the error in comment 0.
- But the mozconfig injection thing makes things a little fishier: Any option that has been handled before the injection are not in the record anymore (as intended), but the mozconfig injection adds new records for them, and they're not re-handled. So they end up remaining at the end.
The above is true for --help, DIST, MOZCONFIG, OLD_CONFIGURE, EXTERNAL_SOURCE_DIR, CONFIG_SHELL... and PYTHON3.
|
|
Assignee | |
Comment 5•5 years ago
|
||
The configure sandbox has a list of arguments to handle, and removes
each of them when it resolves the corresponding options through e.g. a
@depends. When the configure sandbox is finished, what's supposed to
be left is unknown options.
The mozconfig injections adds elements to that list of arguments to
handle. The problem is that by the time the mozconfig injection happens,
some early options have already been handled by the sandbox and won't be
re-handled. Which means by the end of configure, the arguments are still
there, and the sandbox throws an error because it thinks they are for
unknown options.
Things were actually working before bug 1264527, essentially because we
had an explicit list of mozconfig variables that would be injected, and
that didn't include those early options.
So what we now do is to not actually inject those early options from
mozconfig.
|
|
Assignee | |
Updated•5 years ago
|
|
|
Reporter | |
Updated•5 years ago
|
|
|
Assignee | |
Comment 7•5 years ago
|
||
|
||
Comment 9•5 years ago
|
||
| bugherder | ||
https://hg.mozilla.org/mozilla-central/rev/70b3b48718bd
https://hg.mozilla.org/mozilla-central/rev/f395ce024843
|
|
Reporter | |
Comment 10•5 years ago
|
||
This was never actually closed.
|
|
Assignee | |
Comment 11•5 years ago
|
||
(In reply to Ricky Stewart from comment #10)
This was never actually closed.
Huh? The landed patches should have fixed it.
|
|
Reporter | |
Comment 12•5 years ago
|
||
The issue is still observable at HEAD. Revision D75635 might have fixed it but it was never landed (last I checked a week or two back, it was still causing issues in asan Windows build in CI). I just haven't gotten around to diagnosing it further.
|
|
Assignee | |
Comment 13•5 years ago
|
||
I totally confirm setting PYTHON3 in mozconfig works since those landings. Can you be more specific?
|
|
Assignee | |
Comment 14•5 years ago
|
||
(And file a new bug)
|
|
||
Updated•5 years ago
|
|
|
Reporter | |
Comment 15•5 years ago
|
||
facepalm don't worry about this, this was user error on my part. It failed and because I'm so used to seeing it failing I didn't even look at the error message to see it was a completely different thing. Whoops.
Description
•