1433033 - Various cleanups in the Debian-based docker images

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Comment 1

[Mike Hommey [:glandium]]

Attachment: Bug 1433033 - Remove a few packages from the toolchain-build image.

python-dev was required to build mercurial, but the need for that was
removed in bug 1429669.
The others were required for mingw32 toolchains, but they are using a
different docker image and will switch to another different docker image.

Review commit: https://reviewboard.mozilla.org/r/215596/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/215596/

Comment 2

[Mike Hommey [:glandium]]

Attachment: Bug 1433033 - Set Acquire::Check-Valid-Until in apt config file rather than pass it on the command line.

Also set it in the android-build image, which is Debian-snapshot-based too.

Review commit: https://reviewboard.mozilla.org/r/215598/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/215598/

Comment 3

[Mike Hommey [:glandium]]

Attachment: Bug 1433033 - Set apt defaults to avoid passing -q, -y and --no-install-recommends to it.

Review commit: https://reviewboard.mozilla.org/r/215600/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/215600/

Comment 4

[Mike Hommey [:glandium]]

Attachment: Bug 1433033 - Don't `apt-get clean` from Debian-based docker images.

The base images from docker hub actually contain a
/etc/apt/apt.conf.d/docker-clean that does the equivalent of an apt-get
clean after installing packages.

Review commit: https://reviewboard.mozilla.org/r/215602/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/215602/

Comment 5

[Mike Hommey [:glandium]]

Attachment: Bug 1433033 - Create a base Debian docker image.

That image is used to derive all the debian7-* images, and its
definition is parametrized, which will allow to create other images
based on other versions of Debian, from the same definition.

XZ_OPT is kept in each of those because we don't want to automatically
set it in all further derived images.

Review commit: https://reviewboard.mozilla.org/r/215604/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/215604/

Comment 6

[Mike Hommey [:glandium]]

Attachment: Bug 1433033 - Create a mercurial package for Debian stretch.

Review commit: https://reviewboard.mozilla.org/r/215606/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/215606/

Comment 7

[Mike Hommey [:glandium]]

Attachment: Bug 1433033 - Workarounds for Debian stretch.

The apt in Debian stretch doesn't allow repositories with a Release file
not being GPG signed. Setting up GPG signatures on the
taskcluster-artifact-based repositories is a tricky process, and not
strictly necessary. It turns out not creating a Release file at all
works just as well, and works across all current Debian versions. The
packages priorities remain the same, such that packages from those
repositories are still prefered over the ones from the main Debian
repository (as long as versions are higher).

See comment in cloud-mirror-workaround.sh for that part.

Review commit: https://reviewboard.mozilla.org/r/215608/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/215608/

Comment 8

[Mike Hommey [:glandium]]

Attachment: Bug 1433033 - Derive the android-build image from a base Debian stretch image.

Review commit: https://reviewboard.mozilla.org/r/215610/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/215610/

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8945429 [details]
Bug 1433033 - Derive the android-build image from a base Debian stretch image.

https://reviewboard.mozilla.org/r/215610/#review221408

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8945428 [details]
Bug 1433033 - Workarounds for Debian stretch.

https://reviewboard.mozilla.org/r/215608/#review221410

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8945427 [details]
Bug 1433033 - Create a mercurial package for Debian stretch.

https://reviewboard.mozilla.org/r/215606/#review221406

::: taskcluster/ci/packages/kind.yml:99
(Diff revision 1)
> +        sha256: dc2f72caccd6b760226753d48c2f4e8889fe176a6b23ef50775caac55ce28b85
> +      pre-build-command: >-
> +        cp -r contrib/debian debian &&
> +        sed -i -e "s/__VERSION__/$(awk -F\" '$2 {print $2}' mercurial/__version__.py)-1.deb9moz1/" \
> +               -e "s/__DATE__/$(date --rfc-2822)/" \
> +               -e "s/__CODENAME__/wheezy/" debian/changelog

stretch?

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8945426 [details]
Bug 1433033 - Create a base Debian docker image.

https://reviewboard.mozilla.org/r/215604/#review221404

This is great!

Comment 13

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8945425 [details]
Bug 1433033 - Don't `apt-get clean` from Debian-based docker images.

https://reviewboard.mozilla.org/r/215602/#review221402

Comment 14

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8945424 [details]
Bug 1433033 - Set apt defaults to avoid passing -q, -y and --no-install-recommends to it.

https://reviewboard.mozilla.org/r/215600/#review221400

Comment 15

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8945423 [details]
Bug 1433033 - Set Acquire::Check-Valid-Until in apt config file rather than pass it on the command line.

https://reviewboard.mozilla.org/r/215598/#review221398

Comment 16

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8945422 [details]
Bug 1433033 - Remove a few packages from the toolchain-build image.

https://reviewboard.mozilla.org/r/215596/#review221396

Comment 17

[Mike Hommey [:glandium]]

Comment on attachment 8945428 [details]
Bug 1433033 - Workarounds for Debian stretch.

https://reviewboard.mozilla.org/r/215608/#review221480

::: taskcluster/docker/debian-base/cloud-mirror-workaround.sh:7
(Diff revision 1)
> +
> +# The apt in Debian stretch mishandles the HTTP redirects from queue.taskcluster.net to
> +# cloud-mirror.taskcluster.net, and unescapes the url. This apt method wrapper
> +# re-escapes the url.
> +
> +sed -u '/^URI: https:\/\/cloud-mirror.taskcluster.net\/v1\/redirect\//{s,/,%2F,8g;s,:,%3A,2g}' | /usr/lib/apt/methods/https

There are still problems with this hack :(

Comment 18

[Mike Hommey [:glandium]]

Comment on attachment 8945422 [details]
Bug 1433033 - Remove a few packages from the toolchain-build image.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/215596/diff/1-2/

Comment 19

[Mike Hommey [:glandium]]

Comment on attachment 8945423 [details]
Bug 1433033 - Set Acquire::Check-Valid-Until in apt config file rather than pass it on the command line.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/215598/diff/1-2/

Comment 20

[Mike Hommey [:glandium]]

Comment on attachment 8945424 [details]
Bug 1433033 - Set apt defaults to avoid passing -q, -y and --no-install-recommends to it.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/215600/diff/1-2/

Comment 21

[Mike Hommey [:glandium]]

Comment on attachment 8945425 [details]
Bug 1433033 - Don't `apt-get clean` from Debian-based docker images.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/215602/diff/1-2/

Comment 22

[Mike Hommey [:glandium]]

Comment on attachment 8945426 [details]
Bug 1433033 - Create a base Debian docker image.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/215604/diff/1-2/

Comment 23

[Mike Hommey [:glandium]]

Comment on attachment 8945427 [details]
Bug 1433033 - Create a mercurial package for Debian stretch.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/215606/diff/1-2/

Comment 24

[Mike Hommey [:glandium]]

Comment on attachment 8945428 [details]
Bug 1433033 - Workarounds for Debian stretch.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/215608/diff/1-2/

Comment 25

[Mike Hommey [:glandium]]

Comment on attachment 8945429 [details]
Bug 1433033 - Derive the android-build image from a base Debian stretch image.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/215610/diff/1-2/

Comment 26

[Mike Hommey [:glandium]]

Comment on attachment 8945428 [details]
Bug 1433033 - Workarounds for Debian stretch.

(In reply to Mike Hommey [:glandium] from comment #17)
> There are still problems with this hack :(

Okay, this new version works, but it does so quite differently, so I'd rather get another review for it.

Please note that the sed is effectively a no-op on Debian wheezy (because there, the https method does return the escaped url unchanged), so this only would make stretch-based docker images hit the us-west-2 region.

This would also effeectively stop doing anything when https://github.com/taskcluster/cloud-mirror/issues/39 is fixed.

Comment 27

[Mike Hommey [:glandium]]

Attachment: Bug 1433033 - Make APT retry when it fails to download files.

We've observed apt failures multiple times where it apparently fails to
get a file in full from snapshot.debian.org. Making it retry aumatically
rather than retriggering tasks seems better.

Review commit: https://reviewboard.mozilla.org/r/215816/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/215816/

Comment 28

[Mike Hommey [:glandium]]

Comment on attachment 8945682 [details]
Bug 1433033 - Make APT retry when it fails to download files.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/215816/diff/1-2/

Comment 29

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8945428 [details]
Bug 1433033 - Workarounds for Debian stretch.

https://reviewboard.mozilla.org/r/215608/#review221758

::: taskcluster/docker/debian-base/cloud-mirror-workaround.sh:8
(Diff revision 2)
> +# The apt in Debian stretch mishandles the HTTP redirects from queue.taskcluster.net to
> +# cloud-mirror.taskcluster.net, and unescapes the url. This apt method wrapper
> +# strips redirections and sends directly to the S3 bucket. This has the downside of
> +# always hitting us-west-2 independently of the zone this runs on, but this is only
> +# used when creating docker images, so shouldn't generate huge cross-AWS-zone S3
> +# transfers.

++ good fix.

Comment 30

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8945682 [details]
Bug 1433033 - Make APT retry when it fails to download files.

https://reviewboard.mozilla.org/r/215816/#review221760

Comment 31

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7ec508c2094f
Remove a few packages from the toolchain-build image. r=dustin
https://hg.mozilla.org/integration/autoland/rev/18bbec0d7d5e
Set Acquire::Check-Valid-Until in apt config file rather than pass it on the command line. r=dustin
https://hg.mozilla.org/integration/autoland/rev/06acfb703485
Set apt defaults to avoid passing -q, -y and --no-install-recommends to it. r=dustin
https://hg.mozilla.org/integration/autoland/rev/f3b938ae517b
Don't `apt-get clean` from Debian-based docker images. r=dustin
https://hg.mozilla.org/integration/autoland/rev/ff9ce55c9291
Create a base Debian docker image. r=dustin
https://hg.mozilla.org/integration/autoland/rev/e1e8d1028b17
Create a mercurial package for Debian stretch. r=dustin
https://hg.mozilla.org/integration/autoland/rev/3bebaac29a15
Workarounds for Debian stretch. r=dustin
https://hg.mozilla.org/integration/autoland/rev/90650d0682f1
Derive the android-build image from a base Debian stretch image. r=dustin
https://hg.mozilla.org/integration/autoland/rev/b57d3491c10c
Make APT retry when it fails to download files. r=dustin

Comment 32

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/7ec508c2094f
https://hg.mozilla.org/mozilla-central/rev/18bbec0d7d5e
https://hg.mozilla.org/mozilla-central/rev/06acfb703485
https://hg.mozilla.org/mozilla-central/rev/f3b938ae517b
https://hg.mozilla.org/mozilla-central/rev/ff9ce55c9291
https://hg.mozilla.org/mozilla-central/rev/e1e8d1028b17
https://hg.mozilla.org/mozilla-central/rev/3bebaac29a15
https://hg.mozilla.org/mozilla-central/rev/90650d0682f1
https://hg.mozilla.org/mozilla-central/rev/b57d3491c10c

Comment 33

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/caa15de76ecf
Port Bug 1433033: Create a base Debian docker image; r=me