Closed Bug 1632429 Opened 7 months ago Closed 7 months ago

mach try should use Python 3

Categories

(Firefox Build System :: Try, task, P3)

x86_64
Windows 10
task

Tracking

(firefox78 fixed)

RESOLVED FIXED
mozilla78
Tracking Status
firefox78 --- fixed

People

(Reporter: saschanaz, Assigned: mhentges)

References

Details

Attachments

(6 files, 9 obsolete files)

47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
Bundling 2 changesets
Bundling 2 manifests
Bundling 2 revisions of 2 files
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: recorded push in pushlog
remote: added 2 changesets with 2 changes to 2 files (+1 heads)
remote:
remote: View your changes here:
remote:   https://hg.mozilla.org/try/rev/2c5986831eefbeee1d911c3571baa443ceedd831
remote:   https://hg.mozilla.org/try/rev/47a717b18abdac828299a6577648caa2bb9f0ce5
remote:
remote: Follow the progress of your build on Treeherder:
remote:   https://treeherder.mozilla.org/#/jobs?repo=try&revision=47a717b18abdac828299a6577648caa2bb9f0ce5
remote: recorded changegroup in replication log in 0.010s
To hg::ssh://hg.mozilla.org/try
 * [new branch]                HEAD -> branches/default/tip
Exception TypeError: TypeError("'NoneType' object is not callable",) in <bound method _NonClosingTextIOWrapper.__del__ of <_io.TextIOWrapper encoding='utf-16-le'>> ignor
ed

Pushing succeeds anyway, so AFAIK this doesn't block anything.

Summary: mach try logs TypeError on Windows → mach try logs TypeError on Windows Python 3

Did you go out of your way to run mach try with Python 3? It's in the Python 2-only list in the mach script, so errors are going to happen.

In light of that I'll re-name and re-classify the bug.

Priority: -- → P3
Type: defect → task
Summary: mach try logs TypeError on Windows Python 3 → mach try should use Python 3
Component: Mach Core → Try

The component has been changed since the backlog priority was decided, so we're resetting it.
For more information, please visit auto_nag documentation.

Priority: P3 → --
Priority: -- → P3
Assignee: nobody → mhentges

Which try subcommand did you use, and how did you invoke it with Python 3?
I've run ./mach try empty and ./mach try auto, and though there's py3-specific errors, I haven't been able to reproduce the TypeError you see there.

Flags: needinfo?(krosylight)

I think I did python3 mach try chooser (as the commit message says so) but yes, it now instead generates another fatal TypeError:

Error running mach:

    ['try', 'chooser']

The error occurred in mach itself. This is likely a bug in mach itself or a
fundamental problem with a loaded module.

You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

TypeError: Unicode-objects must be encoded before hashing

  File "c:\Users\Kagami\Documents\GitHub\gecko-dev\python/mach\mach\main.py", line 360, in run
    return self._run(argv)
  File "c:\Users\Kagami\Documents\GitHub\gecko-dev\python/mach\mach\main.py", line 416, in _run
    args = parser.parse_args(argv)
  File "c:\mozilla-build\python38\lib\argparse.py", line 1768, in parse_args
    args, argv = self.parse_known_args(args, namespace)
  File "c:\mozilla-build\python38\lib\argparse.py", line 1800, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "c:\mozilla-build\python38\lib\argparse.py", line 2009, in _parse_known_args
    stop_index = consume_positionals(start_index)
  File "c:\mozilla-build\python38\lib\argparse.py", line 1965, in consume_positionals
    take_action(action, args)
  File "c:\mozilla-build\python38\lib\argparse.py", line 1874, in take_action
    action(self, namespace, argument_values, option_string)
  File "c:\Users\Kagami\Documents\GitHub\gecko-dev\python/mach\mach\dispatcher.py", line 181, in __call__
    if handler.parser:
  File "c:\Users\Kagami\Documents\GitHub\gecko-dev\python/mach\mach\decorators.py", line 87, in parser
    self._parser = self._parser()
  File "c:\Users\Kagami\Documents\GitHub\gecko-dev\tools/tryselect/mach_commands.py", line 37, in __call__
    mod = importlib.import_module('tryselect.selectors.{}'.format(self.selector))
  File "c:\mozilla-build\python38\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "c:\Users\Kagami\Documents\GitHub\gecko-dev\tools\tryselect\selectors\chooser\__init__.py", line 12, in <module>
    from tryselect.push import check_working_directory, generate_try_task_config, push_to_try
  File "c:\Users\Kagami\Documents\GitHub\gecko-dev\build/mach_bootstrap.py", line 435, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "c:\Users\Kagami\Documents\GitHub\gecko-dev\tools\tryselect\push.py", line 52, in <module>
    history_path = os.path.join(get_state_dir(srcdir=True), 'history', 'try_task_configs.json')
  File "c:\Users\Kagami\Documents\GitHub\gecko-dev\python/mozboot\mozboot\util.py", line 35, in get_state_dir
    srcdir_hash = hashlib.sha256(srcdir).hexdigest()[:12]
Flags: needinfo?(krosylight)

I'm making decent progress with py3-ification for ./mach try.
While migrating ./mach try chooser, I've found out that the error mentioned in the original comment:

Exception TypeError: TypeError("'NoneType' object is not callable",) in <bound method _NonClosingTextIOWrapper.__del__ of <_io.TextIOWrapper encoding='utf-16-le'>> ignored

Seems to occur regardless of Python version, and it's caused by the temporary flask server we spool up for choosing try task.
The error doesn't seem to be harming anything (it's printed once ./mach finishes running and Python does process-end logic), but I'd still like to invest a bit of time to see why it's popping up

Status: NEW → ASSIGNED

The exception from the original comment:

Exception TypeError: TypeError("'NoneType' object is not callable",) in <bound method _NonClosingTextIOWrapper.__del__ of <_io.TextIOWrapper encoding='utf-16-le'>> ignor ed

Is reported here and fixed here

Depends on: 1569115

This is a graft of py3-compat migrations occurring in-tree.

Note that this isn't a complete migration, it just adds compatibility for the code paths being worked on in bug 1621960.

As |./mach| commands are migrated to python3, "taskgraph" needs to be compatible while still working with python2.

This patch migrates several iter*() calls and python2-specific imports to work with "six" instead.

Note that there's still python2-specific parts of taskgraph, I'm just modifying the pieces that are affecting
the code paths that I'm currently migrating (in this case, |./mach try|

|./mach try| subcommands are now compatible with both python 2 and 3.

Hand-tested with many combinations of subcommand and subcommand flags.

Thanks for doing this! You'll need to update where the tests run for some of the directories (added comments to related revisions).

Also make sure you run the little-known "cram" tests for ./mach try:

$ ./mach cramtest tools/tryselect

if there are failures, use:

$ ./mach cramtest tools/tryselect -i

and select "Yes" if the output differences are expected.

Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/9dc9a089ab43
Improve py3-compatiblity of taskgraph; r=tomprince,ahal
Regressions: 1636209
Status: ASSIGNED → RESOLVED
Closed: 7 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla78

Whoops, sorry, there's two patches needed here

Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/f119ea4ef010
Migrate |./mach try| to python 3; r=rstewart,ahal
Regressions: 1637041
Status: REOPENED → RESOLVED
Closed: 7 months ago7 months ago
Resolution: --- → FIXED
See Also: → 1637261
Regressions: 1637305
Regressions: 1637333
Regressions: 1637915
Regressions: 1637911
Duplicate of this bug: 1606193
Regressions: 1641134
Regressions: 1642517

Depends on D77961

Also causes EasyServer to print a backtrace when it encounters an error.

Depends on D77964

By explicitly specifying that subparsers.required = True, we encourage argparse to provide a useful message,
rather than "'NoneType' is not callable", when no arguments are provided.

Depends on D77966

Attachment #9153684 - Attachment description: Bug 1632429: Improves error when "profile" is called without any arguments r?ahal → Bug 1632429: Improves error when "profile" is called without any arguments in py3 r?ahal

On Windows, the IDL files were being read with the "charmap" codec, but quotes like “ were causing errors.
By specifying "utf8" instead, the read() succeeds.

Attachment #9153677 - Attachment description: Bug 1632429: Resolves error importing "argparse" while using |python3 mach python| r?rstewart → Bug 1632429: "enum34" and "enum" virtualenv packages are now only available to the python2 env r?rstewart
Attachment #9153680 - Attachment is obsolete: true
Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/adadc27a0266
"enum34" and "enum" virtualenv packages are now only available to the python2 env r=rstewart
https://hg.mozilla.org/integration/autoland/rev/d1a3ae7eb7da
Improves python 3 compatibility of mozharness r=aki
https://hg.mozilla.org/integration/autoland/rev/c30caac7237f
Makes mozrelease compatible with python 3 r=aki
Attachment #9153681 - Attachment description: Bug 1632429: Improves update-verify compatibility with python 3 r?tomprince → Bug 1632429: Improves update-verify compatibility with python 3 r?nthomas
Attachment #9153682 - Attachment description: Bug 1632429: Improves mozhttpd compatibility with Python 3 → Bug 1632429: Improves mozhttpd compatibility with Python 3 r?egao
See Also: → 1643198

As part of the migration of mach python to Python 3, I did some regression testing.
All grep-able usages of mach python have been tested (as well as one non-grep-able usage noted by nthomas, thanks!).

Almost all invocations that work with Python 2 are now working with Python 3.
There's only one exception, and it is reported here and has been configured to run with Python 2 until resolved.

There are a few usages that don't work with Python 2. To limit the scope of the work here, I haven't investigated these further, but they are listed here:

  • ./mach python build/pgo/genpgocert.py:
    • Linux: TypeError: unsupported operand type(s) for &=: 'str' and 'int'
    • Windows: %1 is not a valid win32 application
    • Macos: (functional)
  • ./mach python testing/mozharness/scripts/android_wrench.py --config testing/mozharness/configs/android/wrench.py
    • Linux: Missing emulator engine program for 'x86_64' CPU'
      • I created and ran an AVD with an x86_64 CPU by hand and it worked. Unsure what's happening here
    • Windows: ps: illegal option -A
    • Macos:
      • Python 2: invalid command code m. Probably assuming that it's Linux sed, not Macos sed?
      • Python 3: Looks for .mozbuild/android-sdk-linux (instead of .mozbuild/android-sdk-macosx), can't find it and fails
  • ./mach python js/src/vm/make_opcode_doc.py
    • Linux + Windows + Macos:
      • Python 2: Syntax error on print(..., file=sys.stderr)
      • Python 3: Needs markdown library, but it's not vendored

TL;DR: All usages I found that worked before the migration will continue to work after the migration.

One thing that doesn't work with mach python as python3 is --no-virtualenv, because of not one but two enum modules in virtualenv_packages.txt that conflict with python3's and break the regex module:

$ python3 ./mach python --no-virtualenv -c 'import re'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.7/re.py", line 143, in <module>
    class RegexFlag(enum.IntFlag):
AttributeError: module 'enum' has no attribute 'IntFlag'
Attachment #9154279 - Attachment description: Bug 1632429: removes redundant enum library r?glandium → Bug 1632429: remove redundant enum library r?glandium
Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/4864b9629ca7
GeneratePureDOMFunctions no longer throws an error when run with Python 3 on Windows r=loganfsmyth
Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/afdc5717644b
Improves update-verify compatibility with python 3 r=nthomas

Comment on attachment 9153677 [details]
Bug 1632429: "enum34" and "enum" virtualenv packages are now only available to the python2 env r?rstewart

Revision D77960 was moved to bug 1621960. Setting attachment 9153677 [details] to obsolete.

Attachment #9153677 - Attachment is obsolete: true

Comment on attachment 9153678 [details]
Bug 1632429: Improves python 3 compatibility of mozharness r?aki

Revision D77961 was moved to bug 1621960. Setting attachment 9153678 [details] to obsolete.

Attachment #9153678 - Attachment is obsolete: true

Comment on attachment 9153679 [details]
Bug 1632429: Makes mozrelease compatible with python 3 r?aki

Revision D77962 was moved to bug 1621960. Setting attachment 9153679 [details] to obsolete.

Attachment #9153679 - Attachment is obsolete: true

Comment on attachment 9153681 [details]
Bug 1632429: Improves update-verify compatibility with python 3 r?nthomas

Revision D77964 was moved to bug 1621960. Setting attachment 9153681 [details] to obsolete.

Attachment #9153681 - Attachment is obsolete: true

Comment on attachment 9153682 [details]
Bug 1632429: Improves mozhttpd compatibility with Python 3 r?egao

Revision D77965 was moved to bug 1621960. Setting attachment 9153682 [details] to obsolete.

Attachment #9153682 - Attachment is obsolete: true

Comment on attachment 9153683 [details]
Bug 1632429: Improves genpgocert compatibility with Python 3 r?rstewart

Revision D77966 was moved to bug 1621960. Setting attachment 9153683 [details] to obsolete.

Attachment #9153683 - Attachment is obsolete: true

Comment on attachment 9153684 [details]
Bug 1632429: Improves error when "profile" is called without any arguments in py3 r?ahal

Revision D77967 was moved to bug 1621960. Setting attachment 9153684 [details] to obsolete.

Attachment #9153684 - Attachment is obsolete: true

Comment on attachment 9154021 [details]
Bug 1632429: Change |mach python| default from Python 2 to Python 3 r?rstewart

Revision D78181 was moved to bug 1621960. Setting attachment 9154021 [details] to obsolete.

Attachment #9154021 - Attachment is obsolete: true
Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/8526c9dfc05a
remove redundant enum library r=glandium,jgraham
Pushed by egao@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/f156ef6b0f03
bump mozhttpd version to 0.7.1 r=gbrown

It would be good to get D78345 in esr78. Mitch, can you do an approval request for it?

Flags: needinfo?(mhentges)

D77960 + D78345, actually

Missing D77960 may definitely cause issues within ESR78 since we've flipped some commands to use Python 3 by default, but does it make sense to uplift D78345? Haven't we had duplicate enum libraries in previous ESRs?

I'll set up an approval request for D77960 right away.

Flags: needinfo?(mhentges) → needinfo?(mh+mozilla)
See Also: → 1595808

D78345 hasn't been uplifted, but I haven't heard any issues yet, so I'm going to clear the NI.

Flags: needinfo?(mh+mozilla)
You need to log in before you can comment on or make changes to this bug.