1700716 - Port out-of-tree taskgraph to Python 3

Status: RESOLVED FIXED

(Release Engineering :: General, task)

Description

[Andrew Halberstadt [:ahal]]

The in-tree taskgraph code has been running with Python 3 for awhile, but the out-of-tree one is still running with Python 2. It hasn't been synced in awhile and is missing much of the py3 work that went into the in-tree one.

This is also the last known consumer of the taskcluster package with Python 2 and is therefore blocking them from dropping py2 support:
https://github.com/taskcluster/taskcluster/issues/4614

We also plan on merging the two taskgraphs at some point, and this is also blocking that in a way as we first need to sync up the two code bases (syncs haven't been happening for awhile).

Comment 1

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1700716 - Add a dependency on a backport of the 'typing' module in Python 2, r?aki!

We use 'basestring' in a few 'attrs' type hints. Since I'd like to maintain
backwards compatibility with Python 2 for now, I'd like to use the 'typing'
module's AnyStr feature.

Comment 2

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1700716 - Ensure all tests can at least be imported under Python 3, r?aki!

This removes all instances of 'basestring' and a few other necessary tweaks to get
all files importable under Python 3.

Depends on D115009

Comment 3

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1700716 - Fix remainder of flake8 errors under Python 3, r?aki!

Comment 4

[Andrew Halberstadt [:ahal]]

I originally wrote this in a phabricator comment, but it's worth copying to the bug:

My plan for the migration here is roughly:

1. Make standalone taskgraph support both Python 2 and Python 3 at the same time (via six).
2. Update all consumers to use latest version of taskgraph, but still use Python 2 (to make sure there are no issues)
3. Migrate consumers to Python 3 one by one
4. Drop Python 2 support (both in standalone and gecko taskgraph) by removing all references to six

Once step 4 is finished, the gecko and standalone taskgraphs will be in-sync again (at least as far as the Py2 -> Py3 compatibility layer goes).

The problem with using six.text_type like Gecko does, is that this will break compatibility with Python 2 (unless we can guarantee that all consumers are using u"foo" rather than "foo" everywhere). Which means if we need to make changes to taskgraph in the middle of the migration, we'll need to fork a new branch (one for Py2 consumers, and one for Py3 consumers). Having distinct Py2 / Py3 branches is certainly a valid approach to this problem, I just thought that trying to keep all consumers on the same code base would make things easier and smoother.

The downside to my approach is that getting to step 4 is a prerequisite to starting on the gecko <-> standalone sync up. But I think that's a reasonable thing to block on.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/taskgraph/rev/7f313995bf2c
Add a dependency on a backport of the 'typing' module in Python 2, r=aki
https://hg.mozilla.org/ci/taskgraph/rev/79c94db86ac1
Ensure all tests can at least be imported under Python 3, r=aki
https://hg.mozilla.org/ci/taskgraph/rev/cf58568bf6a1
Fix remainder of flake8 errors under Python 3, r=aki

Comment 6

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1700716 - Fix use of iteritems in parameterization.py, r?#taskgraph-reviewers!

Comment 7

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1700716 - Fix error in util.parametrization:_recurse, r?#taskgraph-reviewers!

This uses 'set' rather than 'list' to stay more in-sync with gecko taskgraph
(which also uses set).

Depends on D115218

Comment 8

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1700716 - Vendor in latest copy of util.path:match from Gecko, r?#taskgraph-reviewers!

This fixes a compatibility issue in 're.sub' after Python 3.7

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/taskgraph/rev/2566be1c1079
Fix use of iteritems in parameterization.py, r=taskgraph-reviewers,bhearsum
https://hg.mozilla.org/ci/taskgraph/rev/e80e6d361796
Fix error in util.parametrization:_recurse, r=taskgraph-reviewers,bhearsum

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/taskgraph/rev/301c56c0bd7c
Vendor in latest copy of util.path:match from Gecko, r=taskgraph-reviewers,bhearsum

Comment 11

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1700716 - Fix test_generator.py under Python 3, r?#taskgraph-reviewers!

This also fixed a few of the 'test_optimize.py' tests as a side effect.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/taskgraph/rev/c22358cf1b0f
Fix test_generator.py under Python 3, r=taskgraph-reviewers,aki,bhearsum

Comment 13

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1700716 - Fix remaining tests in test_parameters.py and test_util_vcs.py under Python 3, r?#taskgraph-reviewers!

These tests were failing for the same reasons.

Comment 14

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1700716 - Fix remaining tests in test_create.py under Python 3, r?#taskgraph-reviewers!

Depends on D115401

Comment 15

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1700716 - Fix remaining tests in test_util_attributes.py under Python 3, r?#taskgraph-reviewers!

Depends on D115409

Comment 16

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1700716 - Sync latest copy of taskgraph.util.keyed_by:evaluate_keyed_by from Gecko, r?#taskgraph-reviewers!

I was looking at Gecko to see how we solved the compat issues there and noticed
they functions were nearly the same, so decided to just copy it wholesale.

This adds the 'enforce_single_match' feature which likely isn't needed in
standalone taskgraph (yet), but I added it anyway in the interest of keeping
things synced.

Depends on D115410

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/taskgraph/rev/4b145514c467
Fix remaining tests in test_parameters.py and test_util_vcs.py under Python 3, r=taskgraph-reviewers,bhearsum
https://hg.mozilla.org/ci/taskgraph/rev/16ab15e5ab4a
Fix remaining tests in test_create.py under Python 3, r=taskgraph-reviewers,bhearsum
https://hg.mozilla.org/ci/taskgraph/rev/be9996087215
Fix remaining tests in test_util_attributes.py under Python 3, r=taskgraph-reviewers,bhearsum
https://hg.mozilla.org/ci/taskgraph/rev/51d8be72ad83
Sync latest copy of taskgraph.util.keyed_by:evaluate_keyed_by from Gecko, r=taskgraph-reviewers,bhearsum

Comment 18

[Justin Wood (:Callek)]

(In reply to Andrew Halberstadt [:ahal] from comment #4)

4. Drop Python 2 support (both in standalone and gecko taskgraph) by removing all references to six

https://github.com/asottile/pyupgrade should help with Step 4 and remove a bunch of busy-work.

Comment 19

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1700716 - Fix remaining tests in test_optimize.py under Python 3, r?#taskgraph-reviewers!

Comment 20

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1700716 - Fix remaining tests in test_util_docker.py under Python 3, r?#taskgraph-reviewers!

Depends on D115505

Comment 21

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1700716 - Mark tests in 'test_util_docker.py' xfail when running with Python 3.8+, r?#taskgraph-reviewers!

I noticed these were failing for me locally when using 3.8+ as the hashes are
different than the expectations. More than that, the hashes with 3.9 are also
different from the hashes in 3.8.

Interestingly the version of the test in Gecko passes with 3.6 -> 3.8, but
fails with different hashes under 3.9. There are quite a few differences in the
tar generation logic between Gecko -> standalone, but even syncing everything
wouldn't fix the issue entirely.

I guess the root problem is that I don't know whether the tests are just overly
brittle, or if there is an actual problem here. Either way, marking these xfail
for now.

Depends on D115506

Comment 22

[Andrew Halberstadt [:ahal]]

(In reply to Justin Wood (:Callek) from comment #18)

(In reply to Andrew Halberstadt [:ahal] from comment #4)

4. Drop Python 2 support (both in standalone and gecko taskgraph) by removing all references to six

https://github.com/asottile/pyupgrade should help with Step 4 and remove a bunch of busy-work.

Thanks! I recall you pointing that library out in the gecko version of this bug, but forgot the name. IIRC it even handles removal of six code, so we can run it on both Gecko and standalone taskgraphs to get back into sync.

Comment 23

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1700716 - Set 'HGPLAIN=1' in mercurial 'test_util_vcs.py' tests, r?#taskgraph-reviewers!

This avoids errors when running locally with custom extensions / defaults in
the user ~/.hgrc file.

Depends on D115507

Comment 24

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/taskgraph/rev/3c6281396c24
Fix remaining tests in test_optimize.py under Python 3, r=taskgraph-reviewers,jmaher
https://hg.mozilla.org/ci/taskgraph/rev/d7719965db6a
Fix remaining tests in test_util_docker.py under Python 3, r=taskgraph-reviewers,jmaher
https://hg.mozilla.org/ci/taskgraph/rev/177358aad274
Mark tests in 'test_util_docker.py' xfail when running with Python 3.8+, r=taskgraph-reviewers,jmaher
https://hg.mozilla.org/ci/taskgraph/rev/867252213a1d
Set 'HGPLAIN=1' in mercurial 'test_util_vcs.py' tests, r=taskgraph-reviewers,jmaher

Comment 25

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1700716 - Vendor in latest copy of 'tooltool.py', r?#taskgraph-reviewers!

Comment 26

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1700716 - Vendor in latest copy of 'robustcheckout.py', r?#taskgraph-reviewers!

Depends on D115626

Comment 27

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1700716 - Fix some leftover Python 3 incompatibilities not covered by tests, r?#taskgraph-reviewers!

Depends on D115633

Comment 28

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/taskgraph/rev/209fb5655b69
Vendor in latest copy of 'tooltool.py', r=taskgraph-reviewers,bhearsum
https://hg.mozilla.org/ci/taskgraph/rev/93e9c35b7b6e
Vendor in latest copy of 'robustcheckout.py', r=taskgraph-reviewers,aki
https://hg.mozilla.org/ci/taskgraph/rev/37d105c37ed9
Fix some leftover Python 3 incompatibilities not covered by tests, r=taskgraph-reviewers,aki

Comment 29

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1700716 - Use latest Decision task docker image, r?#taskgraph-reviewers!

This version of the image contains the 'python3-pip' package (as well as few
others I wasn't entirely sure were necessary).

Comment 30

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1700716 - Run decision task with Python 3, r?#taskgraph-reviewers!

Depends on D115926

Comment 31

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1700716 - Use pinned dependencies for Python 3 in the Decision image, r?#taskgraph-reviewers!

While the decision task passes even without this, it's grabbing dependencies
directly from 'setup.py' (which uses the 'requirements.in' file rather than
'requirements.txt'). This will ensure the actual dependencies used in the lock
file are also used in CI.

Comment 32

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/taskgraph/rev/3f6431fae6c8
Use pinned dependencies for Python 3 in the Decision image, r=taskgraph-reviewers,jmaher

Comment 33

[Andrew Halberstadt [:ahal]]

With the latest two patches, taskgraph's own CI is running with Python 3!

So I think I'm going to call this bug done for now, and I'll file follow-ups to track migrating taskgraph's various consumers to Python 3.

Comment 34

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/ci/taskgraph/rev/dbef569aa0f5
Use latest Decision task docker image, r=taskgraph-reviewers,bhearsum
https://hg.mozilla.org/ci/taskgraph/rev/47f9b5c3faf4
Run decision task with Python 3, r=taskgraph-reviewers,jmaher