Various cleanups in the Debian-based docker images

RESOLVED FIXED in mozilla60

Status

Taskcluster
Task Configuration
RESOLVED FIXED
29 days ago
16 days ago

People

(Reporter: glandium, Assigned: glandium)

Tracking

unspecified
mozilla60
Dependency tree / graph

Details

MozReview Requests

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(9 attachments)

59 bytes, text/x-review-board-request
dustin
: review+
Details | Review
59 bytes, text/x-review-board-request
dustin
: review+
Details | Review
59 bytes, text/x-review-board-request
dustin
: review+
Details | Review
59 bytes, text/x-review-board-request
dustin
: review+
Details | Review
59 bytes, text/x-review-board-request
dustin
: review+
Details | Review
59 bytes, text/x-review-board-request
dustin
: review+
Details | Review
59 bytes, text/x-review-board-request
dustin
: review+
Details | Review
59 bytes, text/x-review-board-request
dustin
: review+
Details | Review
59 bytes, text/x-review-board-request
dustin
: review+
Details | Review
Comment hidden (empty)
(Assignee)

Updated

29 days ago
Assignee: nobody → mh+mozilla
Depends on: 1399679
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)

Comment 9

29 days ago
mozreview-review
Comment on attachment 8945429 [details]
Bug 1433033 - Derive the android-build image from a base Debian stretch image.

https://reviewboard.mozilla.org/r/215610/#review221408
Attachment #8945429 - Flags: review?(dustin) → review+

Comment 10

29 days ago
mozreview-review
Comment on attachment 8945428 [details]
Bug 1433033 - Workarounds for Debian stretch.

https://reviewboard.mozilla.org/r/215608/#review221410
Attachment #8945428 - Flags: review?(dustin) → review+
Attachment #8945427 - Flags: review?(dustin) → review+

Comment 11

29 days ago
mozreview-review
Comment on attachment 8945427 [details]
Bug 1433033 - Create a mercurial package for Debian stretch.

https://reviewboard.mozilla.org/r/215606/#review221406

::: taskcluster/ci/packages/kind.yml:99
(Diff revision 1)
> +        sha256: dc2f72caccd6b760226753d48c2f4e8889fe176a6b23ef50775caac55ce28b85
> +      pre-build-command: >-
> +        cp -r contrib/debian debian &&
> +        sed -i -e "s/__VERSION__/$(awk -F\" '$2 {print $2}' mercurial/__version__.py)-1.deb9moz1/" \
> +               -e "s/__DATE__/$(date --rfc-2822)/" \
> +               -e "s/__CODENAME__/wheezy/" debian/changelog

stretch?

Comment 12

29 days ago
mozreview-review
Comment on attachment 8945426 [details]
Bug 1433033 - Create a base Debian docker image.

https://reviewboard.mozilla.org/r/215604/#review221404

This is great!
Attachment #8945426 - Flags: review?(dustin) → review+

Comment 13

29 days ago
mozreview-review
Comment on attachment 8945425 [details]
Bug 1433033 - Don't `apt-get clean` from Debian-based docker images.

https://reviewboard.mozilla.org/r/215602/#review221402
Attachment #8945425 - Flags: review?(dustin) → review+

Comment 14

29 days ago
mozreview-review
Comment on attachment 8945424 [details]
Bug 1433033 - Set apt defaults to avoid passing -q, -y and --no-install-recommends to it.

https://reviewboard.mozilla.org/r/215600/#review221400
Attachment #8945424 - Flags: review?(dustin) → review+

Comment 15

29 days ago
mozreview-review
Comment on attachment 8945423 [details]
Bug 1433033 - Set Acquire::Check-Valid-Until in apt config file rather than pass it on the command line.

https://reviewboard.mozilla.org/r/215598/#review221398
Attachment #8945423 - Flags: review?(dustin) → review+

Comment 16

29 days ago
mozreview-review
Comment on attachment 8945422 [details]
Bug 1433033 - Remove a few packages from the toolchain-build image.

https://reviewboard.mozilla.org/r/215596/#review221396
Attachment #8945422 - Flags: review?(dustin) → review+
(Assignee)

Comment 17

29 days ago
mozreview-review
Comment on attachment 8945428 [details]
Bug 1433033 - Workarounds for Debian stretch.

https://reviewboard.mozilla.org/r/215608/#review221480

::: taskcluster/docker/debian-base/cloud-mirror-workaround.sh:7
(Diff revision 1)
> +
> +# The apt in Debian stretch mishandles the HTTP redirects from queue.taskcluster.net to
> +# cloud-mirror.taskcluster.net, and unescapes the url. This apt method wrapper
> +# re-escapes the url.
> +
> +sed -u '/^URI: https:\/\/cloud-mirror.taskcluster.net\/v1\/redirect\//{s,/,%2F,8g;s,:,%3A,2g}' | /usr/lib/apt/methods/https

There are still problems with this hack :(
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
(Assignee)

Comment 26

28 days ago
Comment on attachment 8945428 [details]
Bug 1433033 - Workarounds for Debian stretch.

(In reply to Mike Hommey [:glandium] from comment #17)
> There are still problems with this hack :(

Okay, this new version works, but it does so quite differently, so I'd rather get another review for it.

Please note that the sed is effectively a no-op on Debian wheezy (because there, the https method does return the escaped url unchanged), so this only would make stretch-based docker images hit the us-west-2 region.

This would also effeectively stop doing anything when https://github.com/taskcluster/cloud-mirror/issues/39 is fixed.
Attachment #8945428 - Flags: review+ → review?(dustin)
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)

Comment 29

28 days ago
mozreview-review
Comment on attachment 8945428 [details]
Bug 1433033 - Workarounds for Debian stretch.

https://reviewboard.mozilla.org/r/215608/#review221758

::: taskcluster/docker/debian-base/cloud-mirror-workaround.sh:8
(Diff revision 2)
> +# The apt in Debian stretch mishandles the HTTP redirects from queue.taskcluster.net to
> +# cloud-mirror.taskcluster.net, and unescapes the url. This apt method wrapper
> +# strips redirections and sends directly to the S3 bucket. This has the downside of
> +# always hitting us-west-2 independently of the zone this runs on, but this is only
> +# used when creating docker images, so shouldn't generate huge cross-AWS-zone S3
> +# transfers.

++ good fix.
Attachment #8945428 - Flags: review?(dustin) → review+

Comment 30

28 days ago
mozreview-review
Comment on attachment 8945682 [details]
Bug 1433033 - Make APT retry when it fails to download files.

https://reviewboard.mozilla.org/r/215816/#review221760
Attachment #8945682 - Flags: review?(dustin) → review+

Comment 31

28 days ago
Pushed by mh@glandium.org:
https://hg.mozilla.org/integration/autoland/rev/7ec508c2094f
Remove a few packages from the toolchain-build image. r=dustin
https://hg.mozilla.org/integration/autoland/rev/18bbec0d7d5e
Set Acquire::Check-Valid-Until in apt config file rather than pass it on the command line. r=dustin
https://hg.mozilla.org/integration/autoland/rev/06acfb703485
Set apt defaults to avoid passing -q, -y and --no-install-recommends to it. r=dustin
https://hg.mozilla.org/integration/autoland/rev/f3b938ae517b
Don't `apt-get clean` from Debian-based docker images. r=dustin
https://hg.mozilla.org/integration/autoland/rev/ff9ce55c9291
Create a base Debian docker image. r=dustin
https://hg.mozilla.org/integration/autoland/rev/e1e8d1028b17
Create a mercurial package for Debian stretch. r=dustin
https://hg.mozilla.org/integration/autoland/rev/3bebaac29a15
Workarounds for Debian stretch. r=dustin
https://hg.mozilla.org/integration/autoland/rev/90650d0682f1
Derive the android-build image from a base Debian stretch image. r=dustin
https://hg.mozilla.org/integration/autoland/rev/b57d3491c10c
Make APT retry when it fails to download files. r=dustin

Comment 33

25 days ago
Pushed by mozilla@hocat.ca:
https://hg.mozilla.org/comm-central/rev/caa15de76ecf
Port Bug 1433033: Create a base Debian docker image; r=me
(Assignee)

Updated

16 days ago
Blocks: 1436283
You need to log in before you can comment on or make changes to this bug.