1632429 - mach try should use Python 3

Status: RESOLVED FIXED

(Developer Infrastructure :: Try, task, P3)

Description

[Kagami Rosylight [:saschanaz] (they/them)]

Bundling 2 changesets
Bundling 2 manifests
Bundling 2 revisions of 2 files
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: recorded push in pushlog
remote: added 2 changesets with 2 changes to 2 files (+1 heads)
remote:
remote: View your changes here:
remote:   https://hg.mozilla.org/try/rev/2c5986831eefbeee1d911c3571baa443ceedd831
remote:   https://hg.mozilla.org/try/rev/47a717b18abdac828299a6577648caa2bb9f0ce5
remote:
remote: Follow the progress of your build on Treeherder:
remote:   https://treeherder.mozilla.org/#/jobs?repo=try&revision=47a717b18abdac828299a6577648caa2bb9f0ce5
remote: recorded changegroup in replication log in 0.010s
To hg::ssh://hg.mozilla.org/try
 * [new branch]                HEAD -> branches/default/tip
Exception TypeError: TypeError("'NoneType' object is not callable",) in <bound method _NonClosingTextIOWrapper.__del__ of <_io.TextIOWrapper encoding='utf-16-le'>> ignor
ed

Pushing succeeds anyway, so AFAIK this doesn't block anything.

Comment 1

[Ricky Stewart]

Did you go out of your way to run mach try with Python 3? It's in the Python 2-only list in the mach script, so errors are going to happen.

In light of that I'll re-name and re-classify the bug.

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

The component has been changed since the backlog priority was decided, so we're resetting it.
For more information, please visit auto_nag documentation.

Comment 3

[Mitchell Hentges [:mhentges] 🦀]

Which try subcommand did you use, and how did you invoke it with Python 3?
I've run ./mach try empty and ./mach try auto, and though there's py3-specific errors, I haven't been able to reproduce the TypeError you see there.

Comment 4

[Kagami Rosylight [:saschanaz] (they/them)]

I think I did python3 mach try chooser (as the commit message says so) but yes, it now instead generates another fatal TypeError:

Error running mach:

    ['try', 'chooser']

The error occurred in mach itself. This is likely a bug in mach itself or a
fundamental problem with a loaded module.

You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

TypeError: Unicode-objects must be encoded before hashing

  File "c:\Users\Kagami\Documents\GitHub\gecko-dev\python/mach\mach\main.py", line 360, in run
    return self._run(argv)
  File "c:\Users\Kagami\Documents\GitHub\gecko-dev\python/mach\mach\main.py", line 416, in _run
    args = parser.parse_args(argv)
  File "c:\mozilla-build\python38\lib\argparse.py", line 1768, in parse_args
    args, argv = self.parse_known_args(args, namespace)
  File "c:\mozilla-build\python38\lib\argparse.py", line 1800, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "c:\mozilla-build\python38\lib\argparse.py", line 2009, in _parse_known_args
    stop_index = consume_positionals(start_index)
  File "c:\mozilla-build\python38\lib\argparse.py", line 1965, in consume_positionals
    take_action(action, args)
  File "c:\mozilla-build\python38\lib\argparse.py", line 1874, in take_action
    action(self, namespace, argument_values, option_string)
  File "c:\Users\Kagami\Documents\GitHub\gecko-dev\python/mach\mach\dispatcher.py", line 181, in __call__
    if handler.parser:
  File "c:\Users\Kagami\Documents\GitHub\gecko-dev\python/mach\mach\decorators.py", line 87, in parser
    self._parser = self._parser()
  File "c:\Users\Kagami\Documents\GitHub\gecko-dev\tools/tryselect/mach_commands.py", line 37, in __call__
    mod = importlib.import_module('tryselect.selectors.{}'.format(self.selector))
  File "c:\mozilla-build\python38\lib\importlib\__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "c:\Users\Kagami\Documents\GitHub\gecko-dev\tools\tryselect\selectors\chooser\__init__.py", line 12, in <module>
    from tryselect.push import check_working_directory, generate_try_task_config, push_to_try
  File "c:\Users\Kagami\Documents\GitHub\gecko-dev\build/mach_bootstrap.py", line 435, in __call__
    module = self._original_import(name, globals, locals, fromlist, level)
  File "c:\Users\Kagami\Documents\GitHub\gecko-dev\tools\tryselect\push.py", line 52, in <module>
    history_path = os.path.join(get_state_dir(srcdir=True), 'history', 'try_task_configs.json')
  File "c:\Users\Kagami\Documents\GitHub\gecko-dev\python/mozboot\mozboot\util.py", line 35, in get_state_dir
    srcdir_hash = hashlib.sha256(srcdir).hexdigest()[:12]

Comment 5

[Mitchell Hentges [:mhentges] 🦀]

I'm making decent progress with py3-ification for ./mach try.
While migrating ./mach try chooser, I've found out that the error mentioned in the original comment:

Exception TypeError: TypeError("'NoneType' object is not callable",) in <bound method _NonClosingTextIOWrapper.__del__ of <_io.TextIOWrapper encoding='utf-16-le'>> ignored

Seems to occur regardless of Python version, and it's caused by the temporary flask server we spool up for choosing try task.
The error doesn't seem to be harming anything (it's printed once ./mach finishes running and Python does process-end logic), but I'd still like to invest a bit of time to see why it's popping up

Comment 6

[Mitchell Hentges [:mhentges] 🦀]

The exception from the original comment:

Exception TypeError: TypeError("'NoneType' object is not callable",) in <bound method _NonClosingTextIOWrapper.__del__ of <_io.TextIOWrapper encoding='utf-16-le'>> ignor ed

Is reported here and fixed here

Comment 7

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1632429: Improve py3-compatiblity of taskgraph; r?tomprince

This is a graft of py3-compat migrations occurring in-tree.

Note that this isn't a complete migration, it just adds compatibility for the code paths being worked on in bug 1621960.

Comment 8

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1632429: Improve py3-compatiblity of taskgraph; r?tomprince

As |./mach| commands are migrated to python3, "taskgraph" needs to be compatible while still working with python2.

This patch migrates several iter*() calls and python2-specific imports to work with "six" instead.

Note that there's still python2-specific parts of taskgraph, I'm just modifying the pieces that are affecting
the code paths that I'm currently migrating (in this case, |./mach try|

Comment 9

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1632429: Migrate |./mach try| to python 3; r?rstewart

|./mach try| subcommands are now compatible with both python 2 and 3.

Hand-tested with many combinations of subcommand and subcommand flags.

Comment 10

[Andrew Halberstadt [:ahal]]

Thanks for doing this! You'll need to update where the tests run for some of the directories (added comments to related revisions).

Also make sure you run the little-known "cram" tests for ./mach try:

$ ./mach cramtest tools/tryselect

if there are failures, use:

$ ./mach cramtest tools/tryselect -i

and select "Yes" if the output differences are expected.

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9dc9a089ab43
Improve py3-compatiblity of taskgraph; r=tomprince,ahal

Comment 12

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/9dc9a089ab43

Comment 13

[Mitchell Hentges [:mhentges] 🦀]

Whoops, sorry, there's two patches needed here

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f119ea4ef010
Migrate |./mach try| to python 3; r=rstewart,ahal

Comment 15

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/f119ea4ef010

Comment 17

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1632429: "enum34" and "enum" virtualenv packages are now only available to the python2 env r?rstewart

Comment 18

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1632429: Improves python 3 compatibility of mozharness r?aki

Depends on D77960

Comment 19

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1632429: Makes mozrelease compatible with python 3 r?aki

Depends on D77961

Comment 20

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1632429: Improves mozdevice compatibility with python 3 r?bc

Depends on D77962

Comment 21

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1632429: Improves update-verify compatibility with python 3 r?nthomas

Depends on D77963

Comment 22

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1632429: Improves mozhttpd compatibility with Python 3 r?egao

Also causes EasyServer to print a backtrace when it encounters an error.

Depends on D77964

Comment 23

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1632429: Improves genpgocert compatibility with Python 3 r?rstewart

Depends on D77965

Comment 24

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1632429: Improves error when "profile" is called without any arguments in py3 r?ahal

By explicitly specifying that subparsers.required = True, we encourage argparse to provide a useful message,
rather than "'NoneType' is not callable", when no arguments are provided.

Depends on D77966

Comment 25

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1632429: GeneratePureDOMFunctions no longer throws an error when run with Python 3 on Windows

On Windows, the IDL files were being read with the "charmap" codec, but quotes like “ were causing errors.
By specifying "utf8" instead, the read() succeeds.

Comment 26

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/adadc27a0266
"enum34" and "enum" virtualenv packages are now only available to the python2 env r=rstewart
https://hg.mozilla.org/integration/autoland/rev/d1a3ae7eb7da
Improves python 3 compatibility of mozharness r=aki
https://hg.mozilla.org/integration/autoland/rev/c30caac7237f
Makes mozrelease compatible with python 3 r=aki

Comment 27

[Edwin Takahashi (:egao | infrequent contributor)]

Attachment: Bug 1632429 - bump mozhttpd version to 0.7.1

Depends on D78159

Comment 28

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1632429: Change |mach python| default from Python 2 to Python 3 r?rstewart

Comment 29

[Mitchell Hentges [:mhentges] 🦀]

As part of the migration of mach python to Python 3, I did some regression testing.
All grep-able usages of mach python have been tested (as well as one non-grep-able usage noted by nthomas, thanks!).

Almost all invocations that work with Python 2 are now working with Python 3.
There's only one exception, and it is reported here and has been configured to run with Python 2 until resolved.

There are a few usages that don't work with Python 2. To limit the scope of the work here, I haven't investigated these further, but they are listed here:

• ./mach python build/pgo/genpgocert.py:
  • Linux: TypeError: unsupported operand type(s) for &=: 'str' and 'int'
  • Windows: %1 is not a valid win32 application
  • Macos: (functional)
• ./mach python testing/mozharness/scripts/android_wrench.py --config testing/mozharness/configs/android/wrench.py
  • Linux: Missing emulator engine program for 'x86_64' CPU'
    • I created and ran an AVD with an x86_64 CPU by hand and it worked. Unsure what's happening here
  • Windows: ps: illegal option -A
  • Macos:
    • Python 2: invalid command code m. Probably assuming that it's Linux sed, not Macos sed?
    • Python 3: Looks for .mozbuild/android-sdk-linux (instead of .mozbuild/android-sdk-macosx), can't find it and fails
• ./mach python js/src/vm/make_opcode_doc.py
  • Linux + Windows + Macos:
    • Python 2: Syntax error on print(..., file=sys.stderr)
    • Python 3: Needs markdown library, but it's not vendored

TL;DR: All usages I found that worked before the migration will continue to work after the migration.

Comment 30

[Mike Hommey [:glandium]]

One thing that doesn't work with mach python as python3 is --no-virtualenv, because of not one but two enum modules in virtualenv_packages.txt that conflict with python3's and break the regex module:

$ python3 ./mach python --no-virtualenv -c 'import re'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.7/re.py", line 143, in <module>
    class RegexFlag(enum.IntFlag):
AttributeError: module 'enum' has no attribute 'IntFlag'

Comment 31

[Mike Hommey [:glandium]]

Oh, it's actually fixed by https://phabricator.services.mozilla.com/D77960

Comment 32

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/adadc27a0266
https://hg.mozilla.org/mozilla-central/rev/d1a3ae7eb7da
https://hg.mozilla.org/mozilla-central/rev/c30caac7237f

Comment 33

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1632429: remove redundant enum library r?glandium

Comment 34

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4864b9629ca7
GeneratePureDOMFunctions no longer throws an error when run with Python 3 on Windows r=loganfsmyth

Comment 35

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/4864b9629ca7

Comment 36

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/afdc5717644b
Improves update-verify compatibility with python 3 r=nthomas

Comment 37

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/afdc5717644b

Comment 38

[Phabricator Automation]

Comment on attachment 9153677 [details]
Bug 1632429: "enum34" and "enum" virtualenv packages are now only available to the python2 env r?rstewart

Revision D77960 was moved to bug 1621960. Setting attachment 9153677 [details] to obsolete.

Comment 39

[Phabricator Automation]

Comment on attachment 9153678 [details]
Bug 1632429: Improves python 3 compatibility of mozharness r?aki

Revision D77961 was moved to bug 1621960. Setting attachment 9153678 [details] to obsolete.

Comment 40

[Phabricator Automation]

Comment on attachment 9153679 [details]
Bug 1632429: Makes mozrelease compatible with python 3 r?aki

Revision D77962 was moved to bug 1621960. Setting attachment 9153679 [details] to obsolete.

Comment 41

[Phabricator Automation]

Comment on attachment 9153681 [details]
Bug 1632429: Improves update-verify compatibility with python 3 r?nthomas

Revision D77964 was moved to bug 1621960. Setting attachment 9153681 [details] to obsolete.

Comment 42

[Phabricator Automation]

Comment on attachment 9153682 [details]
Bug 1632429: Improves mozhttpd compatibility with Python 3 r?egao

Revision D77965 was moved to bug 1621960. Setting attachment 9153682 [details] to obsolete.

Comment 43

[Phabricator Automation]

Comment on attachment 9153683 [details]
Bug 1632429: Improves genpgocert compatibility with Python 3 r?rstewart

Revision D77966 was moved to bug 1621960. Setting attachment 9153683 [details] to obsolete.

Comment 44

[Phabricator Automation]

Comment on attachment 9153684 [details]
Bug 1632429: Improves error when "profile" is called without any arguments in py3 r?ahal

Revision D77967 was moved to bug 1621960. Setting attachment 9153684 [details] to obsolete.

Comment 45

[Phabricator Automation]

Comment on attachment 9154021 [details]
Bug 1632429: Change |mach python| default from Python 2 to Python 3 r?rstewart

Revision D78181 was moved to bug 1621960. Setting attachment 9154021 [details] to obsolete.

Comment 46

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8526c9dfc05a
remove redundant enum library r=glandium,jgraham

Comment 47

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f156ef6b0f03
bump mozhttpd version to 0.7.1 r=gbrown

Comment 48

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/8526c9dfc05a

Comment 49

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/f156ef6b0f03

Comment 50

[Mike Hommey [:glandium]]

It would be good to get D78345 in esr78. Mitch, can you do an approval request for it?

Comment 51

[Mike Hommey [:glandium]]

D77960 + D78345, actually

Comment 52

[Mitchell Hentges [:mhentges] 🦀]

Missing D77960 may definitely cause issues within ESR78 since we've flipped some commands to use Python 3 by default, but does it make sense to uplift D78345? Haven't we had duplicate enum libraries in previous ESRs?

I'll set up an approval request for D77960 right away.

Comment 53

[Mitchell Hentges [:mhentges] 🦀]

D78345 hasn't been uplifted, but I haven't heard any issues yet, so I'm going to clear the NI.