1638990 - Convert 'mach taskgraph' to run with Python 3

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Kris Wright :KrisWright]

mach taskgraph still runs with python 2 [1]. We want to convert all mach commands to python 3.

To do this, any dependencies of the command may need to be converted as well as the command itself.

[1] https://searchfox.org/mozilla-central/rev/9193635dca8cfdcb68f114306194ffc860456044/askcluster/mach_commands.py

Comment 4

[Chris AtLee [:catlee]]

Attachment: Bug 1638990: Use requests for fetching parameters r=tomprince

Comment 5

[Chris AtLee [:catlee]]

Attachment: Bug 1638990: Allow manifest expression parsing to work with py3 r=egao

In Py2, None compares as less than all strings and numbers.
In Py3, None isn't comparable with < to strings or numbers.

For unset variables, use "" as the value for the purposes of parsing manifest expressions.

Depends on D76133

Comment 6

[Chris AtLee [:catlee]]

Attachment: Bug 1638990: Port test resolution to py3 r=ahal

In Py2 dictionaries are sort of orderable, but in Py3 they are not.
Here we explicitly sort on the test name.

Depends on D76134

Comment 7

[Chris AtLee [:catlee]]

With these patches applied, I do see some changes between tasks generated with py2 and py3. Most of the changes look like ordering differences in encoded json, which should be fine.

I do also see some changes to test_manifests for some tasks though, which is an actual problem.
e.g.
test-android-em-7.0-x86_64/debug-geckoview-web-platform-tests-e10s-10 loses "/imagebitmap-renderingcontext" and adds "/accelerometer".

Comment 8

[Ricky Stewart]

(Drive-by comment because this comment ended up in my bugmail: don't know to what extent this will help you, but if you're not doing sort_keys=True when encoding json in Python, you might want to consider doing so. It'll both make it easier for you to make sure there are no diffs when converting from 2 to 3, and help keep you sane once you're all-in on Python 3 where (at least on older Python 3's) dict ordering is super unpredictable. If you're already doing so or if this doesn't matter for you please ignore :) )

Comment 9

[Chris AtLee [:catlee]]

(In reply to Ricky Stewart from comment #8)

(Drive-by comment because this comment ended up in my bugmail: don't know to what extent this will help you, but if you're not doing sort_keys=True when encoding json in Python, you might want to consider doing so. It'll both make it easier for you to make sure there are no diffs when converting from 2 to 3, and help keep you sane once you're all-in on Python 3 where (at least on older Python 3's) dict ordering is super unpredictable. If you're already doing so or if this doesn't matter for you please ignore :) )

Great reminder, thanks. There are a bunch of places where we're encoding json into environment variables that aren't using sort_keys=True. I'll fix those up, since it will make future comparisons easier.

Comment 10

[Chris AtLee [:catlee]]

Attachment: Bug 1638990: Sort values in taskgraph in order to make comparing output easier

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6b1882bbfcc7
Sort values in taskgraph in order to make comparing output easier r=rstewart
https://hg.mozilla.org/integration/autoland/rev/c1238581f4b0
Use requests for fetching parameters r=tomprince
https://hg.mozilla.org/integration/autoland/rev/3bc33af2669a
Allow manifest expression parsing to work with py3 r=egao

Comment 12

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/6b1882bbfcc7
https://hg.mozilla.org/mozilla-central/rev/c1238581f4b0
https://hg.mozilla.org/mozilla-central/rev/3bc33af2669a

Comment 13

[Ricky Stewart]

Why was this bug resolved? mach taskgraph is still running with Python 2 today.

Comment 14

[Tom Prince [:tomprince]]

Attachment: Bug 1638990: [taskgraph] Make test_try_option_syntax pass on python 3;

Comment 15

[Tom Prince [:tomprince]]

Attachment: Bug 1638990: [taskgraph] Make test_decision pass on python 3;

Comment 16

[Tom Prince [:tomprince]]

Attachment: Bug 1638990: [taskgraph] Make test_generator pass on python 3;

Comment 17

[Tom Prince [:tomprince]]

Attachment: Bug 1638990: [taskgraph] Make test_util_parameterization.py pass on python 3;

Comment 18

[Tom Prince [:tomprince]]

Attachment: Bug 1638990: [taskgraph] Make test_util_runnable_jobs pass on python 3;

Comment 19

[Tom Prince [:tomprince]]

Attachment: Bug 1638990: [taskgraph] Make test_taskcluster_yml pass on python 3;

Comment 20

[Tom Prince [:tomprince]]

Attachment: Bug 1638990: [taskgraph] Make test_create pass on python 3;

Comment 21

[Tom Prince [:tomprince]]

Attachment: Bug 1638990: [taskgraph] Make test_optimize pass on python 3;

Comment 22

[Tom Prince [:tomprince]]

Attachment: Bug 1638990: [taskgraph] Make test_morph pass on python 3;

Comment 23

[Tom Prince [:tomprince]]

Attachment: Bug 1638990: [taskgraph] Use `time.monotonic` for bugbug timing when avaiable;

In any case, time.time is preferable to time.clock when time.monotonic is
not available.

Comment 24

[Tom Prince [:tomprince]]

Attachment: Bug 1638990: [taskgraph] Make `taskgraph.decision.write_artifact` work on python 3;

Comment 25

[Tom Prince [:tomprince]]

Attachment: Bug 1638990: [taskgraph] Make `taskgraph.action.util.create_task_from_def` generate the task id;

All the callers generate a fresh task id, so refactor that logic into the
function.

Comment 26

[Tom Prince [:tomprince]]

Attachment: Bug 1638990: [taskgraph] Store task ids as text, rather than bytes;

Comment 27

[Tom Prince [:tomprince]]

Attachment: Bug 1638990: [taskgraph] Apply appropriate change from python-modernize;

Comment 28

[Tom Prince [:tomprince]]

Attachment: Bug 1638990: [taskgraph] Run `mach taskgraph` with python 3;

Comment 29

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/fc13b7163dad
[taskgraph] Make test_try_option_syntax pass on python 3; r=ahal
https://hg.mozilla.org/integration/autoland/rev/336a46acdb6c
[taskgraph] Make test_decision pass on python 3; r=ahal
https://hg.mozilla.org/integration/autoland/rev/2f5586dcfdf8
[taskgraph] Make test_generator pass on python 3; r=ahal
https://hg.mozilla.org/integration/autoland/rev/0d981c7058f6
[taskgraph] Make test_util_parameterization.py pass on python 3; r=ahal
https://hg.mozilla.org/integration/autoland/rev/8d0f64bb2238
[taskgraph] Make test_util_runnable_jobs pass on python 3; r=ahal
https://hg.mozilla.org/integration/autoland/rev/27faa044aad8
[taskgraph] Make test_taskcluster_yml pass on python 3; r=ahal
https://hg.mozilla.org/integration/autoland/rev/8ea5e146c19d
[taskgraph] Make test_create pass on python 3; r=ahal
https://hg.mozilla.org/integration/autoland/rev/44fbdf2391b7
[taskgraph] Make test_optimize pass on python 3; r=ahal
https://hg.mozilla.org/integration/autoland/rev/8474867c8ba5
[taskgraph] Make test_morph pass on python 3; r=ahal
https://hg.mozilla.org/integration/autoland/rev/379f4af46834
[taskgraph] Use `time.monotonic` for bugbug timing when avaiable; r=ahal
https://hg.mozilla.org/integration/autoland/rev/fb44bf643588
[taskgraph] Make `taskgraph.decision.write_artifact` work on python 3; r=ahal
https://hg.mozilla.org/integration/autoland/rev/6330322fb7f8
[taskgraph] Make `taskgraph.action.util.create_task_from_def` generate the task id; r=ahal
https://hg.mozilla.org/integration/autoland/rev/625d545ca409
[taskgraph] Store task ids as text, rather than bytes; r=ahal
https://hg.mozilla.org/integration/autoland/rev/7b1128e12c72
[taskgraph] Apply appropriate change from python-modernize; r=ahal

Comment 30

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/fc13b7163dad
https://hg.mozilla.org/mozilla-central/rev/336a46acdb6c
https://hg.mozilla.org/mozilla-central/rev/2f5586dcfdf8
https://hg.mozilla.org/mozilla-central/rev/0d981c7058f6
https://hg.mozilla.org/mozilla-central/rev/8d0f64bb2238
https://hg.mozilla.org/mozilla-central/rev/27faa044aad8
https://hg.mozilla.org/mozilla-central/rev/8ea5e146c19d
https://hg.mozilla.org/mozilla-central/rev/44fbdf2391b7
https://hg.mozilla.org/mozilla-central/rev/8474867c8ba5
https://hg.mozilla.org/mozilla-central/rev/379f4af46834
https://hg.mozilla.org/mozilla-central/rev/fb44bf643588
https://hg.mozilla.org/mozilla-central/rev/6330322fb7f8
https://hg.mozilla.org/mozilla-central/rev/625d545ca409
https://hg.mozilla.org/mozilla-central/rev/7b1128e12c72

Comment 31

[Tom Prince [:tomprince]]

Attachment: Bug 1638990: [taskgraph] More python 3 fixes; r?ahal

Comment 32

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9cbebb8e39f3
[taskgraph] More python 3 fixes; r=ahal
https://hg.mozilla.org/integration/autoland/rev/5ce5c63b496c
[taskgraph] Run `mach taskgraph` with python 3; r=ahal

Comment 33

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/9cbebb8e39f3
https://hg.mozilla.org/mozilla-central/rev/5ce5c63b496c