1460470 - Port run-task to Python 3

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Gregory Szorc [:gps]]

I'd like to get run-task running on Python 3 so we are future compatible.

Ideally, we'd make run-task Python 3 only. Depending on results of a Try push, I may or may not do that in this bug.

Comment 1

[Gregory Szorc [:gps]]

Attachment: Bug 1460470 - Make run-task compile on Python 3;

The file failed to compile due to octal syntax and missing imports.
After this change, we get a run-time error, which is strictly better.

Review commit: https://reviewboard.mozilla.org/r/243252/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/243252/

Comment 2

[Gregory Szorc [:gps]]

Attachment: Bug 1460470 - Make run-task somewhat usable on Python 3;

This required a lot of attention to bytes versus strings.

The hacks around handling process output are somewhat gross. Apparently
readline() doesn't work on bytes streams in Python 3?! So we install a
custom stream decoder so we can have nice things.

There are still some failures in run-task on Python 3. But we're a big
step closer.

Review commit: https://reviewboard.mozilla.org/r/243254/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/243254/

Comment 3

[Gregory Szorc [:gps]]

Attachment: Bug 1460470 - More run-task Python 3 porting;

Mostly normalization of str and bytes. Python 3 is annoying for
systems level code where most things are bytes.

Review commit: https://reviewboard.mozilla.org/r/243256/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/243256/

Comment 4

[Gregory Szorc [:gps]]

Attachment: Bug 1460470 - Change run-task to use Python 3 by default;

Python 3 is the future.

We'll keep support for Python 2 for now. But hopefully we can make the
script Python 3 only in the near future, since we are getting pretty
close to having Python 3.5+ available everywhere.

Review commit: https://reviewboard.mozilla.org/r/243258/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/243258/

Comment 5

[Gregory Szorc [:gps]]

I'm not 100% certain this passes Try. And we're blocked on landing this at present due to gmplib.org being offline. Try pushes in the past ~24h are using the patch in bug 1460777 to work around the problem.

Comment 6

[Gregory Szorc [:gps]]

This also requires bug 1460475 to get Python 3 on image_builder (or a change to use /usr/bin/python2.7 with run-task on image_builder until Python 3 is available).

Comment 7

[Gregory Szorc [:gps]]

Comment on attachment 8974867 [details]
Bug 1460470 - Make run-task compile on Python 3;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/243252/diff/1-2/

Comment 8

[Gregory Szorc [:gps]]

Comment on attachment 8974868 [details]
Bug 1460470 - Make run-task somewhat usable on Python 3;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/243254/diff/1-2/

Comment 9

[Gregory Szorc [:gps]]

Comment on attachment 8974869 [details]
Bug 1460470 - More run-task Python 3 porting;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/243256/diff/1-2/

Comment 10

[Gregory Szorc [:gps]]

Comment on attachment 8974870 [details]
Bug 1460470 - Change run-task to use Python 3 by default;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/243258/diff/1-2/

Comment 11

[Gregory Szorc [:gps]]

Attachment: Bug 1460470 - Make run-task Python 3.5+ only;

A try push converting run-task to Python 3 seemed to complete without
error.

Since it is annoying writing code that needs to work on both Python
2 and 3, let's require Python 3 and remove code for supporting Python 2.

We implement a version check enforcing Python 3.5+. This is because
we're supposed to be standardizing on 3.5+ everywhere. I want to
prevent accidental usage of older Python 3 versions.

Review commit: https://reviewboard.mozilla.org/r/243436/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/243436/

Comment 12

[Gregory Szorc [:gps]]

Comment on attachment 8974867 [details]
Bug 1460470 - Make run-task compile on Python 3;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/243252/diff/2-3/

Comment 13

[Gregory Szorc [:gps]]

Comment on attachment 8974868 [details]
Bug 1460470 - Make run-task somewhat usable on Python 3;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/243254/diff/2-3/

Comment 14

[Gregory Szorc [:gps]]

Comment on attachment 8974869 [details]
Bug 1460470 - More run-task Python 3 porting;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/243256/diff/2-3/

Comment 15

[Gregory Szorc [:gps]]

Comment on attachment 8974870 [details]
Bug 1460470 - Change run-task to use Python 3 by default;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/243258/diff/2-3/

Comment 16

[Gregory Szorc [:gps]]

Comment on attachment 8975094 [details]
Bug 1460470 - Make run-task Python 3.5+ only;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/243436/diff/1-2/

Comment 17

[Michael Shal [:mshal]]

Comment on attachment 8974867 [details]
Bug 1460470 - Make run-task compile on Python 3;

https://reviewboard.mozilla.org/r/243252/#review250048

Comment 18

[Michael Shal [:mshal]]

Comment on attachment 8974868 [details]
Bug 1460470 - Make run-task somewhat usable on Python 3;

https://reviewboard.mozilla.org/r/243254/#review250058

::: taskcluster/scripts/run-task:729
(Diff revision 3)
>      return run_and_prefix_output(b'task', task_args)
>  
>  
>  if __name__ == '__main__':
>      # Unbuffer stdio.
> +    if not PY3

nit: Missing colon. You fix this in a future patch - maybe a mixed up rebase?

Comment 19

[Michael Shal [:mshal]]

Comment on attachment 8974869 [details]
Bug 1460470 - More run-task Python 3 porting;

https://reviewboard.mozilla.org/r/243256/#review250060

Comment 20

[Michael Shal [:mshal]]

Comment on attachment 8974870 [details]
Bug 1460470 - Change run-task to use Python 3 by default;

https://reviewboard.mozilla.org/r/243258/#review250064

Comment 21

[Michael Shal [:mshal]]

Comment on attachment 8975094 [details]
Bug 1460470 - Make run-task Python 3.5+ only;

https://reviewboard.mozilla.org/r/243436/#review250066

Comment 22

[Gregory Szorc [:gps]]

Comment on attachment 8974868 [details]
Bug 1460470 - Make run-task somewhat usable on Python 3;

https://reviewboard.mozilla.org/r/243254/#review250058

> nit: Missing colon. You fix this in a future patch - maybe a mixed up rebase?

Yup. I'll submit a new series with this fixed up.

Comment 23

[Gregory Szorc [:gps]]

Comment on attachment 8974867 [details]
Bug 1460470 - Make run-task compile on Python 3;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/243252/diff/3-4/

Comment 24

[Gregory Szorc [:gps]]

Comment on attachment 8974868 [details]
Bug 1460470 - Make run-task somewhat usable on Python 3;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/243254/diff/3-4/

Comment 25

[Gregory Szorc [:gps]]

Comment on attachment 8974869 [details]
Bug 1460470 - More run-task Python 3 porting;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/243256/diff/3-4/

Comment 26

[Gregory Szorc [:gps]]

Comment on attachment 8974870 [details]
Bug 1460470 - Change run-task to use Python 3 by default;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/243258/diff/3-4/

Comment 27

[Gregory Szorc [:gps]]

Comment on attachment 8975094 [details]
Bug 1460470 - Make run-task Python 3.5+ only;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/243436/diff/2-3/

Comment 28

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ebd569c9d870
Make run-task compile on Python 3; r=mshal
https://hg.mozilla.org/integration/autoland/rev/16d15b4b97fa
Make run-task somewhat usable on Python 3; r=mshal
https://hg.mozilla.org/integration/autoland/rev/88324086394e
More run-task Python 3 porting; r=mshal
https://hg.mozilla.org/integration/autoland/rev/94c3b68ccc48
Change run-task to use Python 3 by default; r=mshal
https://hg.mozilla.org/integration/autoland/rev/07a3e76abc8c
Make run-task Python 3.5+ only; r=mshal

Comment 29

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/0648f20903e7
Backed out 5 changesets for gps: image build failures a=backout CLOSED TREE

Comment 30

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ef477a048b57
Make run-task compile on Python 3; r=mshal
https://hg.mozilla.org/integration/autoland/rev/19fe5702cf6d
Make run-task somewhat usable on Python 3; r=mshal
https://hg.mozilla.org/integration/autoland/rev/4902cab3ce5d
More run-task Python 3 porting; r=mshal
https://hg.mozilla.org/integration/autoland/rev/33fe8423f88c
Change run-task to use Python 3 by default; r=mshal
https://hg.mozilla.org/integration/autoland/rev/94a9641c5a01
Make run-task Python 3.5+ only; r=mshal

Comment 31

[Gregory Szorc [:gps]]

8a820882e927 (bug 1460411) introduced octal syntax to run-task between my last try run and the initial landing. Yay bit rot. Relanded with an "0666" changed to "0o666".

Comment 32

[Gregory Szorc [:gps]]

Comment on attachment 8974867 [details]
Bug 1460470 - Make run-task compile on Python 3;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/243252/diff/4-5/

Comment 33

[Gregory Szorc [:gps]]

Comment on attachment 8974868 [details]
Bug 1460470 - Make run-task somewhat usable on Python 3;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/243254/diff/4-5/

Comment 34

[Gregory Szorc [:gps]]

Comment on attachment 8974869 [details]
Bug 1460470 - More run-task Python 3 porting;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/243256/diff/4-5/

Comment 35

[Gregory Szorc [:gps]]

Comment on attachment 8974870 [details]
Bug 1460470 - Change run-task to use Python 3 by default;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/243258/diff/4-5/

Comment 36

[Gregory Szorc [:gps]]

Comment on attachment 8975094 [details]
Bug 1460470 - Make run-task Python 3.5+ only;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/243436/diff/3-4/

Comment 37

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/278ac3ea0ce5
Backed out 5 changesets for toolchains bustages a=backout CLOSED TREE

Comment 38

[Gregory Szorc [:gps]]

GCC tasks failed to build because www.multiprecision.org is currently offline. We'll be able to reland once that service comes back online.

Comment 39

[Andrei Ciure[:aciure]]

Backed out 5 changesets (bug 1460470) for toolchains bustages a=backout CLOSED TREE

push that caused the backout: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=94a9641c5a018cfe729ebe748e75a7c4373e4322&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-classifiedState=unclassified&filter-searchStr=277212d3a533ae470142152dc0c6c9573610e574

log: https://treeherder.mozilla.org/logviewer.html#?job_id=178858690&repo=autoland

backout: https://hg.mozilla.org/integration/autoland/rev/278ac3ea0ce59ba78e3b582641f85fdc67315ff1

Comment 40

[Mike Hommey [:glandium]]

With the taskcluster-images rebuilt with this change, this can actually reland without triggering rebuilds of gcc.

Comment 41

[Mozilla Autoland]

hg error in cmd: hg rebase -s ef477a048b575958be74287a2273830813b385f1 -d 51cd668c0e87: abort: unknown revision '51cd668c0e87'!

Comment 42

[Mozilla Autoland]

hg error in cmd: hg rebase -s ef477a048b575958be74287a2273830813b385f1 -d fb6bfcb8a12b: abort: unknown revision 'fb6bfcb8a12b'!

Comment 43

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3332fd1664fd
Make run-task compile on Python 3; r=mshal
https://hg.mozilla.org/integration/autoland/rev/7926b62aaed8
Make run-task somewhat usable on Python 3; r=mshal
https://hg.mozilla.org/integration/autoland/rev/27253061b12b
More run-task Python 3 porting; r=mshal
https://hg.mozilla.org/integration/autoland/rev/d12067020656
Change run-task to use Python 3 by default; r=mshal
https://hg.mozilla.org/integration/autoland/rev/26af09fb8be9
Make run-task Python 3.5+ only; r=mshal

Comment 44

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/3332fd1664fd
https://hg.mozilla.org/mozilla-central/rev/7926b62aaed8
https://hg.mozilla.org/mozilla-central/rev/27253061b12b
https://hg.mozilla.org/mozilla-central/rev/d12067020656
https://hg.mozilla.org/mozilla-central/rev/26af09fb8be9