mach try should use Python 3
Categories
(Developer Infrastructure :: Try, task, P3)
Tracking
(firefox78 fixed)
Tracking | Status | |
---|---|---|
firefox78 | --- | fixed |
People
(Reporter: saschanaz, Assigned: mhentges)
References
Details
Attachments
(6 files, 9 obsolete files)
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review |
Bundling 2 changesets
Bundling 2 manifests
Bundling 2 revisions of 2 files
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: recorded push in pushlog
remote: added 2 changesets with 2 changes to 2 files (+1 heads)
remote:
remote: View your changes here:
remote: https://hg.mozilla.org/try/rev/2c5986831eefbeee1d911c3571baa443ceedd831
remote: https://hg.mozilla.org/try/rev/47a717b18abdac828299a6577648caa2bb9f0ce5
remote:
remote: Follow the progress of your build on Treeherder:
remote: https://treeherder.mozilla.org/#/jobs?repo=try&revision=47a717b18abdac828299a6577648caa2bb9f0ce5
remote: recorded changegroup in replication log in 0.010s
To hg::ssh://hg.mozilla.org/try
* [new branch] HEAD -> branches/default/tip
Exception TypeError: TypeError("'NoneType' object is not callable",) in <bound method _NonClosingTextIOWrapper.__del__ of <_io.TextIOWrapper encoding='utf-16-le'>> ignor
ed
Pushing succeeds anyway, so AFAIK this doesn't block anything.
Reporter | ||
Updated•5 years ago
|
Comment 1•5 years ago
|
||
Did you go out of your way to run mach try
with Python 3? It's in the Python 2-only list in the mach
script, so errors are going to happen.
In light of that I'll re-name and re-classify the bug.
Updated•5 years ago
|
Updated•5 years ago
|
Comment 2•5 years ago
|
||
The component has been changed since the backlog priority was decided, so we're resetting it.
For more information, please visit auto_nag documentation.
Reporter | ||
Updated•5 years ago
|
Assignee | ||
Updated•5 years ago
|
Assignee | ||
Comment 3•5 years ago
|
||
Which try
subcommand did you use, and how did you invoke it with Python 3?
I've run ./mach try empty
and ./mach try auto
, and though there's py3-specific errors, I haven't been able to reproduce the TypeError
you see there.
Reporter | ||
Comment 4•5 years ago
•
|
||
I think I did python3 mach try chooser
(as the commit message says so) but yes, it now instead generates another fatal TypeError:
Error running mach:
['try', 'chooser']
The error occurred in mach itself. This is likely a bug in mach itself or a
fundamental problem with a loaded module.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.
If filing a bug, please include the full output of mach, including this error
message.
The details of the failure are as follows:
TypeError: Unicode-objects must be encoded before hashing
File "c:\Users\Kagami\Documents\GitHub\gecko-dev\python/mach\mach\main.py", line 360, in run
return self._run(argv)
File "c:\Users\Kagami\Documents\GitHub\gecko-dev\python/mach\mach\main.py", line 416, in _run
args = parser.parse_args(argv)
File "c:\mozilla-build\python38\lib\argparse.py", line 1768, in parse_args
args, argv = self.parse_known_args(args, namespace)
File "c:\mozilla-build\python38\lib\argparse.py", line 1800, in parse_known_args
namespace, args = self._parse_known_args(args, namespace)
File "c:\mozilla-build\python38\lib\argparse.py", line 2009, in _parse_known_args
stop_index = consume_positionals(start_index)
File "c:\mozilla-build\python38\lib\argparse.py", line 1965, in consume_positionals
take_action(action, args)
File "c:\mozilla-build\python38\lib\argparse.py", line 1874, in take_action
action(self, namespace, argument_values, option_string)
File "c:\Users\Kagami\Documents\GitHub\gecko-dev\python/mach\mach\dispatcher.py", line 181, in __call__
if handler.parser:
File "c:\Users\Kagami\Documents\GitHub\gecko-dev\python/mach\mach\decorators.py", line 87, in parser
self._parser = self._parser()
File "c:\Users\Kagami\Documents\GitHub\gecko-dev\tools/tryselect/mach_commands.py", line 37, in __call__
mod = importlib.import_module('tryselect.selectors.{}'.format(self.selector))
File "c:\mozilla-build\python38\lib\importlib\__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 783, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "c:\Users\Kagami\Documents\GitHub\gecko-dev\tools\tryselect\selectors\chooser\__init__.py", line 12, in <module>
from tryselect.push import check_working_directory, generate_try_task_config, push_to_try
File "c:\Users\Kagami\Documents\GitHub\gecko-dev\build/mach_bootstrap.py", line 435, in __call__
module = self._original_import(name, globals, locals, fromlist, level)
File "c:\Users\Kagami\Documents\GitHub\gecko-dev\tools\tryselect\push.py", line 52, in <module>
history_path = os.path.join(get_state_dir(srcdir=True), 'history', 'try_task_configs.json')
File "c:\Users\Kagami\Documents\GitHub\gecko-dev\python/mozboot\mozboot\util.py", line 35, in get_state_dir
srcdir_hash = hashlib.sha256(srcdir).hexdigest()[:12]
Assignee | ||
Comment 5•5 years ago
|
||
I'm making decent progress with py3-ification for ./mach try
.
While migrating ./mach try chooser
, I've found out that the error mentioned in the original comment:
Exception TypeError: TypeError("'NoneType' object is not callable",) in <bound method _NonClosingTextIOWrapper.__del__ of <_io.TextIOWrapper encoding='utf-16-le'>> ignored
Seems to occur regardless of Python version, and it's caused by the temporary flask server we spool up for choosing try
task.
The error doesn't seem to be harming anything (it's printed once ./mach
finishes running and Python does process-end logic), but I'd still like to invest a bit of time to see why it's popping up
Assignee | ||
Updated•5 years ago
|
Assignee | ||
Comment 6•5 years ago
|
||
Assignee | ||
Comment 7•5 years ago
|
||
This is a graft of py3-compat migrations occurring in-tree.
Note that this isn't a complete migration, it just adds compatibility for the code paths being worked on in bug 1621960.
Assignee | ||
Comment 8•5 years ago
|
||
As |./mach| commands are migrated to python3, "taskgraph" needs to be compatible while still working with python2.
This patch migrates several iter*() calls and python2-specific imports to work with "six" instead.
Note that there's still python2-specific parts of taskgraph, I'm just modifying the pieces that are affecting
the code paths that I'm currently migrating (in this case, |./mach try|
Assignee | ||
Comment 9•5 years ago
|
||
|./mach try| subcommands are now compatible with both python 2 and 3.
Hand-tested with many combinations of subcommand and subcommand flags.
Comment 10•5 years ago
|
||
Thanks for doing this! You'll need to update where the tests run for some of the directories (added comments to related revisions).
Also make sure you run the little-known "cram" tests for ./mach try:
$ ./mach cramtest tools/tryselect
if there are failures, use:
$ ./mach cramtest tools/tryselect -i
and select "Yes" if the output differences are expected.
Comment 11•5 years ago
|
||
Comment 12•5 years ago
|
||
bugherder |
Assignee | ||
Comment 13•5 years ago
|
||
Whoops, sorry, there's two patches needed here
Updated•5 years ago
|
Comment 14•5 years ago
|
||
Comment 15•5 years ago
|
||
bugherder |
Assignee | ||
Comment 17•5 years ago
|
||
Assignee | ||
Comment 18•5 years ago
|
||
Depends on D77960
Assignee | ||
Comment 19•5 years ago
|
||
Depends on D77961
Assignee | ||
Comment 20•5 years ago
|
||
Depends on D77962
Assignee | ||
Comment 21•5 years ago
|
||
Depends on D77963
Assignee | ||
Comment 22•5 years ago
|
||
Also causes EasyServer to print a backtrace when it encounters an error.
Depends on D77964
Assignee | ||
Comment 23•5 years ago
|
||
Depends on D77965
Assignee | ||
Comment 24•5 years ago
|
||
By explicitly specifying that subparsers.required = True
, we encourage argparse to provide a useful message,
rather than "'NoneType' is not callable", when no arguments are provided.
Depends on D77966
Updated•5 years ago
|
Assignee | ||
Comment 25•5 years ago
|
||
On Windows, the IDL files were being read with the "charmap" codec, but quotes like “ were causing errors.
By specifying "utf8" instead, the read() succeeds.
Updated•5 years ago
|
Updated•5 years ago
|
Comment 26•5 years ago
|
||
Updated•5 years ago
|
Updated•5 years ago
|
Comment 27•5 years ago
|
||
Depends on D78159
Assignee | ||
Comment 28•5 years ago
|
||
Assignee | ||
Comment 29•5 years ago
|
||
As part of the migration of mach python
to Python 3, I did some regression testing.
All grep-able usages of mach python
have been tested (as well as one non-grep-able usage noted by nthomas, thanks!).
Almost all invocations that work with Python 2 are now working with Python 3.
There's only one exception, and it is reported here and has been configured to run with Python 2 until resolved.
There are a few usages that don't work with Python 2. To limit the scope of the work here, I haven't investigated these further, but they are listed here:
./mach python build/pgo/genpgocert.py
:- Linux:
TypeError: unsupported operand type(s) for &=: 'str' and 'int'
- Windows:
%1 is not a valid win32 application
- Macos: (functional)
- Linux:
./mach python testing/mozharness/scripts/android_wrench.py --config testing/mozharness/configs/android/wrench.py
- Linux:
Missing emulator engine program for 'x86_64' CPU'
- I created and ran an AVD with an
x86_64
CPU by hand and it worked. Unsure what's happening here
- I created and ran an AVD with an
- Windows:
ps: illegal option -A
- Macos:
- Python 2:
invalid command code m
. Probably assuming that it's Linuxsed
, not Macossed
? - Python 3: Looks for
.mozbuild/android-sdk-linux
(instead of.mozbuild/android-sdk-macosx
), can't find it and fails
- Python 2:
- Linux:
./mach python js/src/vm/make_opcode_doc.py
- Linux + Windows + Macos:
- Python 2: Syntax error on
print(..., file=sys.stderr)
- Python 3: Needs
markdown
library, but it's not vendored
- Python 2: Syntax error on
- Linux + Windows + Macos:
TL;DR: All usages I found that worked before the migration will continue to work after the migration.
Comment 30•5 years ago
|
||
One thing that doesn't work with mach python
as python3 is --no-virtualenv
, because of not one but two enum
modules in virtualenv_packages.txt that conflict with python3's and break the regex module:
$ python3 ./mach python --no-virtualenv -c 'import re'
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/lib/python3.7/re.py", line 143, in <module>
class RegexFlag(enum.IntFlag):
AttributeError: module 'enum' has no attribute 'IntFlag'
Comment 31•5 years ago
|
||
Oh, it's actually fixed by https://phabricator.services.mozilla.com/D77960
Comment 32•5 years ago
|
||
bugherder |
Assignee | ||
Comment 33•5 years ago
|
||
Updated•5 years ago
|
Comment 34•5 years ago
|
||
Comment 35•5 years ago
|
||
bugherder |
Comment 36•5 years ago
|
||
Comment 37•5 years ago
|
||
bugherder |
Comment 38•5 years ago
|
||
Comment on attachment 9153677 [details]
Bug 1632429: "enum34" and "enum" virtualenv packages are now only available to the python2 env r?rstewart
Revision D77960 was moved to bug 1621960. Setting attachment 9153677 [details] to obsolete.
Comment 39•5 years ago
|
||
Comment on attachment 9153678 [details]
Bug 1632429: Improves python 3 compatibility of mozharness r?aki
Revision D77961 was moved to bug 1621960. Setting attachment 9153678 [details] to obsolete.
Comment 40•5 years ago
|
||
Comment on attachment 9153679 [details]
Bug 1632429: Makes mozrelease compatible with python 3 r?aki
Revision D77962 was moved to bug 1621960. Setting attachment 9153679 [details] to obsolete.
Comment 41•5 years ago
|
||
Comment on attachment 9153681 [details]
Bug 1632429: Improves update-verify compatibility with python 3 r?nthomas
Revision D77964 was moved to bug 1621960. Setting attachment 9153681 [details] to obsolete.
Comment 42•5 years ago
|
||
Comment on attachment 9153682 [details]
Bug 1632429: Improves mozhttpd compatibility with Python 3 r?egao
Revision D77965 was moved to bug 1621960. Setting attachment 9153682 [details] to obsolete.
Comment 43•5 years ago
|
||
Comment on attachment 9153683 [details]
Bug 1632429: Improves genpgocert compatibility with Python 3 r?rstewart
Revision D77966 was moved to bug 1621960. Setting attachment 9153683 [details] to obsolete.
Comment 44•5 years ago
|
||
Comment on attachment 9153684 [details]
Bug 1632429: Improves error when "profile" is called without any arguments in py3 r?ahal
Revision D77967 was moved to bug 1621960. Setting attachment 9153684 [details] to obsolete.
Comment 45•5 years ago
|
||
Comment on attachment 9154021 [details]
Bug 1632429: Change |mach python| default from Python 2 to Python 3 r?rstewart
Revision D78181 was moved to bug 1621960. Setting attachment 9154021 [details] to obsolete.
Comment 46•5 years ago
|
||
Comment 47•5 years ago
|
||
Comment 48•5 years ago
|
||
bugherder |
Comment 49•5 years ago
|
||
bugherder |
Comment 50•5 years ago
|
||
It would be good to get D78345 in esr78. Mitch, can you do an approval request for it?
Comment 51•5 years ago
•
|
||
D77960 + D78345, actually
Assignee | ||
Comment 52•5 years ago
•
|
||
Missing D77960 may definitely cause issues within ESR78 since we've flipped some commands to use Python 3 by default, but does it make sense to uplift D78345? Haven't we had duplicate enum libraries in previous ESRs?
I'll set up an approval request for D77960 right away.
Assignee | ||
Comment 53•5 years ago
|
||
D78345 hasn't been uplifted, but I haven't heard any issues yet, so I'm going to clear the NI.
Updated•3 years ago
|
Description
•