Closed Bug 1638990 Opened 6 months ago Closed 4 months ago

Convert 'mach taskgraph' to run with Python 3

Categories

(Firefox Build System :: Task Configuration, task)

task

Tracking

(firefox80 fixed)

RESOLVED FIXED
mozilla80
Tracking Status
firefox80 --- fixed

People

(Reporter: KrisWright, Assigned: catlee)

References

(Blocks 1 open bug)

Details

Attachments

(19 files, 1 obsolete file)

47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review

mach taskgraph still runs with python 2 [1]. We want to convert all mach commands to python 3.

To do this, any dependencies of the command may need to be converted as well as the command itself.

[1] https://searchfox.org/mozilla-central/rev/9193635dca8cfdcb68f114306194ffc860456044/askcluster/mach_commands.py

Duplicate of this bug: 1638989
Duplicate of this bug: 1638987
Duplicate of this bug: 1639316
Assignee: nobody → catlee
Status: NEW → ASSIGNED

In Py2, None compares as less than all strings and numbers.
In Py3, None isn't comparable with < to strings or numbers.

For unset variables, use "" as the value for the purposes of parsing manifest expressions.

Depends on D76133

In Py2 dictionaries are sort of orderable, but in Py3 they are not.
Here we explicitly sort on the test name.

Depends on D76134

With these patches applied, I do see some changes between tasks generated with py2 and py3. Most of the changes look like ordering differences in encoded json, which should be fine.

I do also see some changes to test_manifests for some tasks though, which is an actual problem.
e.g.
test-android-em-7.0-x86_64/debug-geckoview-web-platform-tests-e10s-10 loses "/imagebitmap-renderingcontext" and adds "/accelerometer".

(Drive-by comment because this comment ended up in my bugmail: don't know to what extent this will help you, but if you're not doing sort_keys=True when encoding json in Python, you might want to consider doing so. It'll both make it easier for you to make sure there are no diffs when converting from 2 to 3, and help keep you sane once you're all-in on Python 3 where (at least on older Python 3's) dict ordering is super unpredictable. If you're already doing so or if this doesn't matter for you please ignore :) )

(In reply to Ricky Stewart from comment #8)

(Drive-by comment because this comment ended up in my bugmail: don't know to what extent this will help you, but if you're not doing sort_keys=True when encoding json in Python, you might want to consider doing so. It'll both make it easier for you to make sure there are no diffs when converting from 2 to 3, and help keep you sane once you're all-in on Python 3 where (at least on older Python 3's) dict ordering is super unpredictable. If you're already doing so or if this doesn't matter for you please ignore :) )

Great reminder, thanks. There are a bunch of places where we're encoding json into environment variables that aren't using sort_keys=True. I'll fix those up, since it will make future comparisons easier.

Attachment #9150434 - Attachment is obsolete: true
Attachment #9150434 - Attachment is obsolete: false
Attachment #9150435 - Attachment is obsolete: true
Attachment #9152498 - Attachment description: Bug 1638990: Sort values to make comparing output easier → Bug 1638990: Sort values in taskgraph in order to make comparing output easier
Attachment #9150433 - Attachment description: Bug 1638990: Use six's urlopen for fetching parameters r=tomprince → Bug 1638990: Use requests for fetching parameters r=tomprince
Pushed by catlee@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/6b1882bbfcc7
Sort values in taskgraph in order to make comparing output easier r=rstewart
https://hg.mozilla.org/integration/autoland/rev/c1238581f4b0
Use requests for fetching parameters r=tomprince
https://hg.mozilla.org/integration/autoland/rev/3bc33af2669a
Allow manifest expression parsing to work with py3 r=egao
Status: ASSIGNED → RESOLVED
Closed: 6 months ago
Keywords: leave-open
Resolution: --- → FIXED

Why was this bug resolved? mach taskgraph is still running with Python 2 today.

Flags: needinfo?(catlee)
Status: RESOLVED → REOPENED
Resolution: FIXED → ---

In any case, time.time is preferable to time.clock when time.monotonic is
not available.

All the callers generate a fresh task id, so refactor that logic into the
function.

Attachment #9164540 - Attachment description: Bug 1638990: [taskgraph] Run `mach taskgraph with python 3; → Bug 1638990: [taskgraph] Run `mach taskgraph` with python 3;
Flags: needinfo?(catlee)
Keywords: leave-open
Pushed by mozilla@hocat.ca:
https://hg.mozilla.org/integration/autoland/rev/fc13b7163dad
[taskgraph] Make test_try_option_syntax pass on python 3; r=ahal
https://hg.mozilla.org/integration/autoland/rev/336a46acdb6c
[taskgraph] Make test_decision pass on python 3; r=ahal
https://hg.mozilla.org/integration/autoland/rev/2f5586dcfdf8
[taskgraph] Make test_generator pass on python 3; r=ahal
https://hg.mozilla.org/integration/autoland/rev/0d981c7058f6
[taskgraph] Make test_util_parameterization.py pass on python 3; r=ahal
https://hg.mozilla.org/integration/autoland/rev/8d0f64bb2238
[taskgraph] Make test_util_runnable_jobs pass on python 3; r=ahal
https://hg.mozilla.org/integration/autoland/rev/27faa044aad8
[taskgraph] Make test_taskcluster_yml pass on python 3; r=ahal
https://hg.mozilla.org/integration/autoland/rev/8ea5e146c19d
[taskgraph] Make test_create pass on python 3; r=ahal
https://hg.mozilla.org/integration/autoland/rev/44fbdf2391b7
[taskgraph] Make test_optimize pass on python 3; r=ahal
https://hg.mozilla.org/integration/autoland/rev/8474867c8ba5
[taskgraph] Make test_morph pass on python 3; r=ahal
https://hg.mozilla.org/integration/autoland/rev/379f4af46834
[taskgraph] Use `time.monotonic` for bugbug timing when avaiable; r=ahal
https://hg.mozilla.org/integration/autoland/rev/fb44bf643588
[taskgraph] Make `taskgraph.decision.write_artifact` work on python 3; r=ahal
https://hg.mozilla.org/integration/autoland/rev/6330322fb7f8
[taskgraph] Make `taskgraph.action.util.create_task_from_def` generate the task id; r=ahal
https://hg.mozilla.org/integration/autoland/rev/625d545ca409
[taskgraph] Store task ids as text, rather than bytes; r=ahal
https://hg.mozilla.org/integration/autoland/rev/7b1128e12c72
[taskgraph] Apply appropriate change from python-modernize; r=ahal
Pushed by mozilla@hocat.ca:
https://hg.mozilla.org/integration/autoland/rev/9cbebb8e39f3
[taskgraph] More python 3 fixes; r=ahal
https://hg.mozilla.org/integration/autoland/rev/5ce5c63b496c
[taskgraph] Run `mach taskgraph` with python 3; r=ahal
Keywords: leave-open
Status: REOPENED → RESOLVED
Closed: 6 months ago4 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla80
Regressions: 1655499
You need to log in before you can comment on or make changes to this bug.