Closed Bug 1621960 Opened 7 months ago Closed 4 months ago

Run `mach python` with Python 3

Categories

(Firefox Build System :: General, task, P3)

task

Tracking

(firefox79 fixed)

RESOLVED FIXED
mozilla79
Tracking Status
firefox79 --- fixed

People

(Reporter: ahal, Assigned: mhentges)

References

(Blocks 1 open bug)

Details

Attachments

(9 files, 3 obsolete files)

47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review

Currently mach python still uses Python 2. If you want to run the Python 3 environment you have to run something like $ python3 mach python. We should switch this around as the Python 3 environment is slowly becoming the main one (even if it isn't there yet).

Currently making this switch results in some traceback in the mach's logging:

--- Logging error ---
Traceback (most recent call last):
  File "/home/ahal/.pyenv/versions/3.7.6/lib/python3.7/logging/__init__.py", line 1025, in emit
    msg = self.format(record)
  File "/home/ahal/.pyenv/versions/3.7.6/lib/python3.7/logging/__init__.py", line 869, in format
    return fmt.format(record)
  File "/home/ahal/dev/mozilla-central/python/mach/mach/logging.py", line 50, in format
    return json.dumps([record.created, action, params])
  File "/home/ahal/.pyenv/versions/3.7.6/lib/python3.7/json/__init__.py", line 231, in dumps
    return _default_encoder.encode(obj)
  File "/home/ahal/.pyenv/versions/3.7.6/lib/python3.7/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/home/ahal/.pyenv/versions/3.7.6/lib/python3.7/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
TypeError: keys must be str, int, float, bool or None, not bytes
Call stack:
  File "mach", line 264, in <module>
    main(sys.argv[1:])
  File "mach", line 260, in main
    sys.exit(mach.run(args))
  File "/home/ahal/dev/mozilla-central/python/mach/mach/main.py", line 360, in run
    return self._run(argv)
  File "/home/ahal/dev/mozilla-central/python/mach/mach/main.py", line 464, in _run
    **vars(args.command_args))
  File "/home/ahal/dev/mozilla-central/python/mach/mach/registrar.py", line 109, in _run_command_handler
    result = fn(**kwargs)
  File "/home/ahal/dev/mozilla-central/python/mach_commands.py", line 90, in python
    append_env=append_env)
  File "/home/ahal/dev/mozilla-central/python/mach/mach/mixin/process.py", line 104, in run_process
    self.log(logging.DEBUG, 'process', {'env': use_env}, 'Environment: {env}')
  File "/home/ahal/dev/mozilla-central/python/mach/mach/mixin/logging.py", line 54, in log
    extra={'action': action, 'params': params})
Message: 'Environment: {env}'
Arguments: ()
Python 3.7.6 (default, Feb 27 2020, 09:08:27) 
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

The console still works fine afterwards, but we'll need to figure out why it's throwing that.

From my matrix comments....

THIS is the issue for that traceback in py3 - https://searchfox.org/mozilla-central/source/python/mach_commands.py#59
(at least py3.6)
and of course https://searchfox.org/mozilla-central/source/python/mach_commands.py#64 as well
and we have a similar sort of shim at https://searchfox.org/mozilla-central/source/mach#206 (I couldn't find any others in my brief search).

Easy fix, it just needs to wrap the env with this function:
https://searchfox.org/mozilla-central/source/python/mozbuild/mozbuild/util.py#1491

Bugbug thinks this bug should belong to this component, but please revert this change in case of error.

Component: General → Mach Core
Component: Mach Core → General

Hey Andrew -- did you ever get around to throwing a patch together for this? If not I'll assign to someone else.

Flags: needinfo?(ahal)

Nope, I never started working on this (is unassigned atm).

Flags: needinfo?(ahal)

Can I work on this?

Hey Shubham, this issue will be tricky to do without having unintended consequences elsewhere in the codebase (there's some sneaky calls to ./mach python that use py3-incompatible code, and tracking these all down might be easier after having some familiarity with the codebase).

I'll probably grab this ticket, but there'll probably be some other more accessible py3-conversion tickets :)

This is a graft of py3-compat migrations occurring in-tree.

Note that this isn't a complete migration, it just adds compatibility for the code paths being worked on in bug 1621960.

Assignee: nobody → mhentges
Status: NEW → ASSIGNED
Attachment #9144918 - Attachment description: Bug 1621960 - Improves py3-compatiblity of taskgraph r?tomprince → Bug 1621960: Improve py3-compatiblity of taskgraph; r?tomprince

As |./mach| commands are migrated to python3, "taskgraph" needs to be compatible while still working with python2.

This patch migrates several iter*() calls and python2-specific imports to work with "six" instead.

Note that there's still python2-specific parts of taskgraph, I'm just modifying the pieces that are affecting
the code paths that I'm currently migrating (in this case, |./mach try|

Depends on D73099

|./mach try| subcommands are now compatible with both python 2 and 3.

Hand-tested with many combinations of subcommand and subcommand flags.

Depends on D73397

Comment on attachment 9144918 [details]
Bug 1621960: Improve py3-compatiblity of taskgraph; r?tomprince

Revision D73396 was moved to bug 1632429. Setting attachment 9144918 [details] to obsolete.

Attachment #9144918 - Attachment is obsolete: true

Comment on attachment 9144921 [details]
Bug 1621960: Improve py3-compatiblity of taskgraph; r?tomprince

Revision D73397 was moved to bug 1632429. Setting attachment 9144921 [details] to obsolete.

Attachment #9144921 - Attachment is obsolete: true

Comment on attachment 9144922 [details]
Bug 1621960: Migrate |./mach try| to python 3; r?rstewart

Revision D73398 was moved to bug 1632429. Setting attachment 9144922 [details] to obsolete.

Attachment #9144922 - Attachment is obsolete: true
Assignee: mhentges → nobody
Status: ASSIGNED → NEW

Wrong bug ID, sorry about that folks

When running |mach python| with python 3, it was failing when importing "enum", since it would get the python 2 backport. By making these packages only
available to python 2, we allow python 3 to import standard library packages properly.

Assignee: nobody → mhentges
Status: NEW → ASSIGNED

By explicitly specifying that subparsers.required = True, we encourage argparse to provide a useful message,
rather than "'NoneType' is not callable", when no arguments are provided.

Depends on D77966

Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/d7eb101e5df7
Improves mozhttpd compatibility with Python 3 r=egao
https://hg.mozilla.org/integration/autoland/rev/3e2cbe4dd777
Improves genpgocert compatibility with Python 3 r=rstewart
https://hg.mozilla.org/integration/autoland/rev/0e4661c9061d
Improves error when "profile" is called without any arguments in py3 r=ahal
https://hg.mozilla.org/integration/autoland/rev/f36a95234a5f
Change |mach python| default from Python 2 to Python 3 r=rstewart

There was a usage of ./mach python that I wasn't aware of that depends on a python2-specific import.
I'm going to re-land the other changes, then I'll investigate the python2-specific import.

Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/88d426f7019e
Improves mozhttpd compatibility with Python 3 r=egao
https://hg.mozilla.org/integration/autoland/rev/73349d7e362e
Improves genpgocert compatibility with Python 3 r=rstewart
https://hg.mozilla.org/integration/autoland/rev/f8456bcdfdb7
Improves error when "profile" is called without any arguments in py3 r=ahal
Status: ASSIGNED → RESOLVED
Closed: 4 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla79
Flags: needinfo?(mhentges)
Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/1177ebce20f8
upload_generated_sources uses py3-compatible import r=rstewart
Pushed by mhentges@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/3b1827c74e34
Change |mach python| default from Python 2 to Python 3 r=rstewart
Regressions: 1645393
Regressed by: 1645779
No longer regressed by: 1645779
Regressions: 1645779
Regressions: 1649266

Comment on attachment 9155345 [details]
Bug 1621960: "enum34" and "enum" virtualenv packages are now only available to the python2 env r?rstewart

ESR Uplift Approval Request

  • If this is not a sec:{high,crit} bug, please state case for ESR consideration: Some scripts in-tree now run with Python 3 by default, but the vendored enum libraries directly impede Python 3's enum implementation.
  • User impact if declined: This may cause issues when downstream is running different ./mach commands with ESR 78.
  • Fix Landed on Version: 79
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): * We know that we don't want the 3rd party enum libraries for Python 3 because it already has an enum implementation
  • The 3rd party enum libraries are built to be compatible with Python 3's enum implementation, so switching between shouldn't cause an issue
  • Filtering a package to only apply to a certain python version (the python2:... prefix) is stable
  • String or UUID changes made by this patch:
Attachment #9155345 - Flags: approval-mozilla-esr78?

Comment on attachment 9155345 [details]
Bug 1621960: "enum34" and "enum" virtualenv packages are now only available to the python2 env r?rstewart

Reduces the risk of bustage when running mach commands across different branches. Approved for 78.1esr.

Attachment #9155345 - Flags: approval-mozilla-esr78? → approval-mozilla-esr78+

Comment on attachment 9155345 [details]
Bug 1621960: "enum34" and "enum" virtualenv packages are now only available to the python2 env r?rstewart

Uplifted to ESR78. Removing the approval flag to avoid being on the needs-uplift radar.
https://hg.mozilla.org/releases/mozilla-esr78/rev/6f7d5ab59284

Attachment #9155345 - Flags: approval-mozilla-esr78+
You need to log in before you can comment on or make changes to this bug.