1460475 - Updates to image_builder

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Gregory Szorc [:gps]]

Let's make some updates to the image_builder Docker image so it is more modern.

Comment 1

[Gregory Szorc [:gps]]

Attachment: Bug 1460475 - Install Python 3 on image_builder;

We want Python 3 available everywhere because it is 2018.

Review commit: https://reviewboard.mozilla.org/r/242948/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/242948/

Comment 2

[Gregory Szorc [:gps]]

Attachment: Bug 1460475 - Upgrade python-zstandard in image_builder;

Version 0.9.0 bundles a newer version of the zstandard library, which
is a little faster and has a few minor bug fixes (none that we were
likely hitting, however).

Review commit: https://reviewboard.mozilla.org/r/242950/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/242950/

Comment 3

[Gregory Szorc [:gps]]

Attachment: Bug 1460475 - Port download-and-compress to Python 3;

download-and-compress isn't very complicated and should work on Python 3
with minimal effort. So let's switch it to use Python 3.

Review commit: https://reviewboard.mozilla.org/r/242952/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/242952/

Comment 4

[Gregory Szorc [:gps]]

Attachment: Bug 1460475 - Install zstandard 0.9.0 in mach;

The latest python-zstandard uses a newer zstandard that is faster.
It also has wheels available, which means installation doesn't require
Python development headers, etc.

Review commit: https://reviewboard.mozilla.org/r/242972/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/242972/

Comment 5

[Gregory Szorc [:gps]]

Comment on attachment 8974594 [details]
Bug 1460475 - Install Python 3 on image_builder;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/242948/diff/1-2/

Comment 6

[Gregory Szorc [:gps]]

Comment on attachment 8974595 [details]
Bug 1460475 - Upgrade python-zstandard in image_builder;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/242950/diff/1-2/

Comment 7

[Gregory Szorc [:gps]]

Comment on attachment 8974596 [details]
Bug 1460475 - Port download-and-compress to Python 3;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/242952/diff/1-2/

Comment 8

[Gregory Szorc [:gps]]

Attachment: Bug 1460475 - Use stream_reader API;

python-zstandard 0.9 has an API that exposes a file object interface
for compression and decompression. This means we can remove our
stream wrapper in order to consume a zstandard compressed tar file.

Review commit: https://reviewboard.mozilla.org/r/243430/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/243430/

Comment 9

[Gregory Szorc [:gps]]

Comment on attachment 8974594 [details]
Bug 1460475 - Install Python 3 on image_builder;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/242948/diff/2-3/

Comment 10

[Gregory Szorc [:gps]]

Comment on attachment 8974595 [details]
Bug 1460475 - Upgrade python-zstandard in image_builder;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/242950/diff/2-3/

Comment 11

[Gregory Szorc [:gps]]

Comment on attachment 8974605 [details]
Bug 1460475 - Install zstandard 0.9.0 in mach;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/242972/diff/1-2/

Comment 12

[Gregory Szorc [:gps]]

Comment on attachment 8974596 [details]
Bug 1460475 - Port download-and-compress to Python 3;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/242952/diff/2-3/

Comment 13

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8974596 [details]
Bug 1460475 - Port download-and-compress to Python 3;

https://reviewboard.mozilla.org/r/242952/#review249964

::: taskcluster/docker/image_builder/setup.sh:52
(Diff revision 3)
>  ]
>  EOF
>  )
>  
> +# We need to install for both Python 2 and 3 because `mach taskcluster-load-image`
> +# uses Python 2 and `download-and-compress` uses Python 3.

..but python-requests{,-unixsocket} is no longer installed..

Comment 14

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8975088 [details]
Bug 1460475 - Use stream_reader API;

https://reviewboard.mozilla.org/r/243430/#review249960

Same question here -- upgrading to 0.9.0 but not applying this cset would cause runtime failures, no?

Comment 15

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8974605 [details]
Bug 1460475 - Install zstandard 0.9.0 in mach;

https://reviewboard.mozilla.org/r/242972/#review249958

Shouldn't this be squashed with the previous commit?

Comment 16

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8974595 [details]
Bug 1460475 - Upgrade python-zstandard in image_builder;

https://reviewboard.mozilla.org/r/242950/#review249954

Comment 17

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8974594 [details]
Bug 1460475 - Install Python 3 on image_builder;

https://reviewboard.mozilla.org/r/242948/#review249956

Comment 18

[Gregory Szorc [:gps]]

Comment on attachment 8975088 [details]
Bug 1460475 - Use stream_reader API;

https://reviewboard.mozilla.org/r/243430/#review249960

I don't see a problem here: 0.9 is backwards compatible and the code deleted in this patch will still work. However, this patch does depend on using zstandard 0.9. But since this patch is after the 0.9 patches, there shouldn't be an issue.

Comment 19

[Gregory Szorc [:gps]]

Comment on attachment 8974596 [details]
Bug 1460475 - Port download-and-compress to Python 3;

https://reviewboard.mozilla.org/r/242952/#review249964

> ..but python-requests{,-unixsocket} is no longer installed..

I don't think we use these packages as part of `mach` and in CI? But I can certainly re-add them to the Docker image: that's not a problem. Worst case is it is a cargo cult.

Comment 20

[Gregory Szorc [:gps]]

Comment on attachment 8974594 [details]
Bug 1460475 - Install Python 3 on image_builder;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/242948/diff/3-4/

Comment 21

[Gregory Szorc [:gps]]

Comment on attachment 8974595 [details]
Bug 1460475 - Upgrade python-zstandard in image_builder;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/242950/diff/3-4/

Comment 22

[Gregory Szorc [:gps]]

Comment on attachment 8974605 [details]
Bug 1460475 - Install zstandard 0.9.0 in mach;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/242972/diff/2-3/

Comment 23

[Gregory Szorc [:gps]]

Comment on attachment 8975088 [details]
Bug 1460475 - Use stream_reader API;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/243430/diff/1-2/

Comment 24

[Gregory Szorc [:gps]]

Comment on attachment 8974596 [details]
Bug 1460475 - Port download-and-compress to Python 3;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/242952/diff/3-4/

Comment 25

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f093c57bcc02
Install Python 3 on image_builder; r=dustin
https://hg.mozilla.org/integration/autoland/rev/358d587a6bd9
Upgrade python-zstandard in image_builder; r=dustin
https://hg.mozilla.org/integration/autoland/rev/dce97dea67c5
Install zstandard 0.9.0 in mach; r=dustin
https://hg.mozilla.org/integration/autoland/rev/359dd8da6ef8
Use stream_reader API; r=dustin
https://hg.mozilla.org/integration/autoland/rev/64ef3fab796d
Port download-and-compress to Python 3; r=dustin

Comment 26

[Eliza Balazs [:ebalazs_]]

https://hg.mozilla.org/mozilla-central/rev/f093c57bcc02
https://hg.mozilla.org/mozilla-central/rev/358d587a6bd9
https://hg.mozilla.org/mozilla-central/rev/dce97dea67c5
https://hg.mozilla.org/mozilla-central/rev/359dd8da6ef8
https://hg.mozilla.org/mozilla-central/rev/64ef3fab796d