1621960 - Run `mach python` with Python 3

Status: RESOLVED FIXED

(Firefox Build System :: General, task, P3)

Description

[Andrew Halberstadt [:ahal]]

Currently mach python still uses Python 2. If you want to run the Python 3 environment you have to run something like $ python3 mach python. We should switch this around as the Python 3 environment is slowly becoming the main one (even if it isn't there yet).

Currently making this switch results in some traceback in the mach's logging:

--- Logging error ---
Traceback (most recent call last):
  File "/home/ahal/.pyenv/versions/3.7.6/lib/python3.7/logging/__init__.py", line 1025, in emit
    msg = self.format(record)
  File "/home/ahal/.pyenv/versions/3.7.6/lib/python3.7/logging/__init__.py", line 869, in format
    return fmt.format(record)
  File "/home/ahal/dev/mozilla-central/python/mach/mach/logging.py", line 50, in format
    return json.dumps([record.created, action, params])
  File "/home/ahal/.pyenv/versions/3.7.6/lib/python3.7/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/home/ahal/.pyenv/versions/3.7.6/lib/python3.7/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/home/ahal/.pyenv/versions/3.7.6/lib/python3.7/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
TypeError: keys must be str, int, float, bool or None, not bytes
Call stack:
  File "mach", line 264, in <module>
    main(sys.argv[1:])
  File "mach", line 260, in main
    sys.exit(mach.run(args))
  File "/home/ahal/dev/mozilla-central/python/mach/mach/main.py", line 360, in run
    return self._run(argv)
  File "/home/ahal/dev/mozilla-central/python/mach/mach/main.py", line 464, in _run
    **vars(args.command_args))
  File "/home/ahal/dev/mozilla-central/python/mach/mach/registrar.py", line 109, in _run_command_handler
    result = fn(**kwargs)
  File "/home/ahal/dev/mozilla-central/python/mach_commands.py", line 90, in python
    append_env=append_env)
  File "/home/ahal/dev/mozilla-central/python/mach/mach/mixin/process.py", line 104, in run_process
    self.log(logging.DEBUG, 'process', {'env': use_env}, 'Environment: {env}')
  File "/home/ahal/dev/mozilla-central/python/mach/mach/mixin/logging.py", line 54, in log
    extra={'action': action, 'params': params})
Message: 'Environment: {env}'
Arguments: ()
Python 3.7.6 (default, Feb 27 2020, 09:08:27) 
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

The console still works fine afterwards, but we'll need to figure out why it's throwing that.

Comment 1

[Justin Wood (:Callek)]

From my matrix comments....

THIS is the issue for that traceback in py3 - https://searchfox.org/mozilla-central/source/python/mach_commands.py#59
(at least py3.6)
and of course https://searchfox.org/mozilla-central/source/python/mach_commands.py#64 as well
and we have a similar sort of shim at https://searchfox.org/mozilla-central/source/mach#206 (I couldn't find any others in my brief search).

Comment 2

[Andrew Halberstadt [:ahal]]

Easy fix, it just needs to wrap the env with this function:
https://searchfox.org/mozilla-central/source/python/mozbuild/mozbuild/util.py#1491

Comment 3

[BugBot [:suhaib / :marco/ :calixte]]

Bugbug thinks this bug should belong to this component, but please revert this change in case of error.

Comment 4

[Ricky Stewart]

Hey Andrew -- did you ever get around to throwing a patch together for this? If not I'll assign to someone else.

Comment 5

[Andrew Halberstadt [:ahal]]

Nope, I never started working on this (is unassigned atm).

Comment 6

[Shubham Kumar [:imskr]]

Can I work on this?

Comment 7

[Mitchell Hentges [:mhentges] 🦀]

Hey Shubham, this issue will be tricky to do without having unintended consequences elsewhere in the codebase (there's some sneaky calls to ./mach python that use py3-incompatible code, and tracking these all down might be easier after having some familiarity with the codebase).

I'll probably grab this ticket, but there'll probably be some other more accessible py3-conversion tickets :)

Comment 8

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1621960: Improve py3-compatiblity of taskgraph; r?tomprince

This is a graft of py3-compat migrations occurring in-tree.

Note that this isn't a complete migration, it just adds compatibility for the code paths being worked on in bug 1621960.

Comment 9

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1621960: Improve py3-compatiblity of taskgraph; r?tomprince

As |./mach| commands are migrated to python3, "taskgraph" needs to be compatible while still working with python2.

This patch migrates several iter*() calls and python2-specific imports to work with "six" instead.

Note that there's still python2-specific parts of taskgraph, I'm just modifying the pieces that are affecting
the code paths that I'm currently migrating (in this case, |./mach try|

Depends on D73099

Comment 10

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1621960: Migrate |./mach try| to python 3; r?rstewart

|./mach try| subcommands are now compatible with both python 2 and 3.

Hand-tested with many combinations of subcommand and subcommand flags.

Depends on D73397

Comment 11

[Phabricator Automation]

Comment on attachment 9144918 [details]
Bug 1621960: Improve py3-compatiblity of taskgraph; r?tomprince

Revision D73396 was moved to bug 1632429. Setting attachment 9144918 [details] to obsolete.

Comment 12

[Phabricator Automation]

Comment on attachment 9144921 [details]
Bug 1621960: Improve py3-compatiblity of taskgraph; r?tomprince

Revision D73397 was moved to bug 1632429. Setting attachment 9144921 [details] to obsolete.

Comment 13

[Phabricator Automation]

Comment on attachment 9144922 [details]
Bug 1621960: Migrate |./mach try| to python 3; r?rstewart

Revision D73398 was moved to bug 1632429. Setting attachment 9144922 [details] to obsolete.

Comment 14

[Mitchell Hentges [:mhentges] 🦀]

Wrong bug ID, sorry about that folks

Comment 15

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1621960: "enum34" and "enum" virtualenv packages are now only available to the python2 env r?rstewart

When running |mach python| with python 3, it was failing when importing "enum", since it would get the python 2 backport. By making these packages only
available to python 2, we allow python 3 to import standard library packages properly.

Comment 16

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1621960: Improves python 3 compatibility of mozharness r?aki

Depends on D77960

Comment 17

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1621960: Makes mozrelease compatible with python 3 r?aki

Depends on D77961

Comment 18

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1621960: Improves update-verify compatibility with python 3 r?nthomas

Depends on D77962

Comment 19

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1621960: Improves mozhttpd compatibility with Python 3 r?egao

Depends on D77964

Comment 20

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1621960: Improves genpgocert compatibility with Python 3 r?rstewart

Depends on D77965

Comment 21

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1621960: Improves error when "profile" is called without any arguments in py3 r?ahal

By explicitly specifying that subparsers.required = True, we encourage argparse to provide a useful message,
rather than "'NoneType' is not callable", when no arguments are provided.

Depends on D77966

Comment 22

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1621960: Change |mach python| default from Python 2 to Python 3 r?rstewart

Depends on D77967

Comment 23

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d7eb101e5df7
Improves mozhttpd compatibility with Python 3 r=egao
https://hg.mozilla.org/integration/autoland/rev/3e2cbe4dd777
Improves genpgocert compatibility with Python 3 r=rstewart
https://hg.mozilla.org/integration/autoland/rev/0e4661c9061d
Improves error when "profile" is called without any arguments in py3 r=ahal
https://hg.mozilla.org/integration/autoland/rev/f36a95234a5f
Change |mach python| default from Python 2 to Python 3 r=rstewart

Comment 24

[Noemi Erli[:noemi_erli]]

Backed out 4 changesets (Bug 1621960) for causing Android bustages CLOSED TREE

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&selectedTaskRun=KQiBZcoWSsW0L3Nq6kbpfQ-0&resultStatus=testfailed%2Cbusted%2Cexception&revision=f36a95234a5fca1a8eab7565ab888ed628376db8

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=305672347&repo=autoland&lineNumber=67

Backout: https://hg.mozilla.org/integration/autoland/rev/80d2d3b254ca0999ec5b9c377ff2228c3b470876

Comment 25

[Noemi Erli[:noemi_erli]]

Later update: this is also happening on other platforms too: https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&resultStatus=testfailed%2Cbusted%2Cexception&revision=f36a95234a5fca1a8eab7565ab888ed628376db8&searchStr=UgsB

Comment 26

[Mitchell Hentges [:mhentges] 🦀]

There was a usage of ./mach python that I wasn't aware of that depends on a python2-specific import.
I'm going to re-land the other changes, then I'll investigate the python2-specific import.

Comment 27

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/88d426f7019e
Improves mozhttpd compatibility with Python 3 r=egao
https://hg.mozilla.org/integration/autoland/rev/73349d7e362e
Improves genpgocert compatibility with Python 3 r=rstewart
https://hg.mozilla.org/integration/autoland/rev/f8456bcdfdb7
Improves error when "profile" is called without any arguments in py3 r=ahal

Comment 28

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/88d426f7019e
https://hg.mozilla.org/mozilla-central/rev/73349d7e362e
https://hg.mozilla.org/mozilla-central/rev/f8456bcdfdb7

Comment 29

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1621960: upload_generated_sources uses py3-compatible import r?rstewart

Comment 30

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1177ebce20f8
upload_generated_sources uses py3-compatible import r=rstewart

Comment 31

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3b1827c74e34
Change |mach python| default from Python 2 to Python 3 r=rstewart

Comment 32

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/1177ebce20f8

Comment 33

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/3b1827c74e34

Comment 34

[Mitchell Hentges [:mhentges] 🦀]

Comment on attachment 9155345 [details]
Bug 1621960: "enum34" and "enum" virtualenv packages are now only available to the python2 env r?rstewart

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration: Some scripts in-tree now run with Python 3 by default, but the vendored enum libraries directly impede Python 3's enum implementation.
• User impact if declined: This may cause issues when downstream is running different ./mach commands with ESR 78.
• Fix Landed on Version: 79
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): * We know that we don't want the 3rd party enum libraries for Python 3 because it already has an enum implementation
• The 3rd party enum libraries are built to be compatible with Python 3's enum implementation, so switching between shouldn't cause an issue
• Filtering a package to only apply to a certain python version (the python2:... prefix) is stable
• String or UUID changes made by this patch:

Comment 35

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9155345 [details]
Bug 1621960: "enum34" and "enum" virtualenv packages are now only available to the python2 env r?rstewart

Reduces the risk of bustage when running mach commands across different branches. Approved for 78.1esr.

Comment 36

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9155345 [details]
Bug 1621960: "enum34" and "enum" virtualenv packages are now only available to the python2 env r?rstewart

Uplifted to ESR78. Removing the approval flag to avoid being on the needs-uplift radar.
https://hg.mozilla.org/releases/mozilla-esr78/rev/6f7d5ab59284