Closed Bug 1212411 Opened 10 years ago Closed 10 years ago

Clean up base-test and tester images

Categories

(Taskcluster Graveyard :: Docker Images, defect)

Product:
Taskcluster Graveyard
Component:
Docker Images
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: dustin, Assigned: dustin)

Details

Attachments

(1 file)

MozReview Request: Bug 1212411: fork docker images for tests; r?ahal
40 bytes, text/x-review-board-request
ahal
: review+
Details
These are - pointing to quay.io instead of docker hub - installing from puppetagain.pub.b.m.o - creating too many intermediate images (should use system-setup.sh)
Summary: Clean up bas-test and tester images → Clean up base-test and tester images
Dropping the use of the puppetagain repos brings about this difference in installed packages: > --- old.installed 2015-10-08 11:24:28.000000000 -0400 > +++ new.installed 2015-10-08 11:24:36.000000000 -0400 > @@ -564,15 +564,15 @@ > ii libxau6 1:1.0.6-4 X11 authorisation library > ii libxau6:i386 1:1.0.6-4 X11 authorisation library > ii libxaw7 2:1.0.9-3ubuntu1 X11 Athena Widget library > -ii libxcb-glx0 1.8.1-2ubuntu2.1mozilla1 X C Binding, glx extension > -ii libxcb-glx0:i386 1.8.1-2ubuntu2.1mozilla1 X C Binding, glx extension > -ii libxcb-render0 1.8.1-2ubuntu2.1mozilla1 X C Binding, render extension > -ii libxcb-render0-dev 1.8.1-2ubuntu2.1mozilla1 X C Binding, render extension, development files > -ii libxcb-shm0 1.8.1-2ubuntu2.1mozilla1 X C Binding, shm extension > -ii libxcb-shm0-dev 1.8.1-2ubuntu2.1mozilla1 X C Binding, shm extension, development files > -ii libxcb1 1.8.1-2ubuntu2.1mozilla1 X C Binding > -ii libxcb1:i386 1.8.1-2ubuntu2.1mozilla1 X C Binding > -ii libxcb1-dev 1.8.1-2ubuntu2.1mozilla1 X C Binding, development files > +ii libxcb-glx0 1.8.1-1ubuntu0.2 X C Binding, glx extension > +ii libxcb-glx0:i386 1.8.1-1ubuntu0.2 X C Binding, glx extension > +ii libxcb-render0 1.8.1-1ubuntu0.2 X C Binding, render extension > +ii libxcb-render0-dev 1.8.1-1ubuntu0.2 X C Binding, render extension, development files > +ii libxcb-shm0 1.8.1-1ubuntu0.2 X C Binding, shm extension > +ii libxcb-shm0-dev 1.8.1-1ubuntu0.2 X C Binding, shm extension, development files > +ii libxcb1 1.8.1-1ubuntu0.2 X C Binding > +ii libxcb1:i386 1.8.1-1ubuntu0.2 X C Binding > +ii libxcb1-dev 1.8.1-1ubuntu0.2 X C Binding, development files > ii libxcomposite-dev 1:0.4.3-2build1 X11 Composite extension library (development headers) > ii libxcomposite1 1:0.4.3-2build1 X11 Composite extension library > ii libxcursor-dev 1:1.1.12-1ubuntu0.1 X cursor management library (development files) > @@ -637,8 +637,8 @@ > ii manpages 3.35-0.1ubuntu1 Manual pages about using a GNU/Linux system > ii manpages-dev 3.35-0.1ubuntu1 Manual pages about using GNU/Linux for development > ii mawk 1.3.3-17 a pattern scanning and text processing language > -ii mercurial 2.5.4-0mozilla1 easy-to-use, scalable distributed version control system > -ii mercurial-common 2.5.4-0mozilla1 easy-to-use, scalable distributed version control system (common files) > +ii mercurial 2.0.2-1ubuntu1.2 easy-to-use, scalable distributed version control system > +ii mercurial-common 2.0.2-1ubuntu1.2 easy-to-use, scalable distributed version control system (common files) > ii mesa-common-dev 8.0.4-0ubuntu0.7 Developer documentation for Mesa > ii mime-support 3.51-1ubuntu1.1 MIME files 'mime.types' & 'mailcap', and support programs > ii module-init-tools 3.16-1ubuntu2 tools for managing Linux kernel modules For mercurial, we can use `peep`, similar to how we do for builders. libxcb was from bug 975216 (which was fun!)
OK, I put the libxcb stuff into a custom apt repo in a tarball on tooltool. That gets downloaded, unpacked, installed from, and deleted. I've arranged to install mercurial with `peep`, so we get hash validation of its contents, but can still install the latest version. So the remaining package diff is --- tmp/old.installed 2015-10-08 11:24:28.000000000 -0400 +++ tmp/new.installed 2015-10-08 17:23:47.000000000 -0400 @@ -637,8 +637,6 @@ ii manpages 3.35-0.1ubuntu1 Manual pages about using a GNU/Linux system ii manpages-dev 3.35-0.1ubuntu1 Manual pages about using GNU/Linux for development ii mawk 1.3.3-17 a pattern scanning and text processing language -ii mercurial 2.5.4-0mozilla1 easy-to-use, scalable distributed version control system -ii mercurial-common 2.5.4-0mozilla1 easy-to-use, scalable distributed version control system (common files) ii mesa-common-dev 8.0.4-0ubuntu0.7 Developer documentation for Mesa ii mime-support 3.51-1ubuntu1.1 MIME files 'mime.types' & 'mailcap', and support programs ii module-init-tools 3.16-1ubuntu2 tools for managing Linux kernel modules as expected.
Oh, as importantly: I "forked" the B2G images (base-test and tester) so as not to risk messing up B2G. So the image I created just now is `ubuntu1204-test`, and I've be generated `ubuntu1204-test-upd` and `desktop-test:0.4.0` based on it.
I rebased Andrew's changes from https://treeherder.mozilla.org/#/jobs?repo=try&revision=844213339596 and combined them with my changes; the result is running now at https://treeherder.mozilla.org/#/jobs?repo=try&revision=458f25f83e84
Aww, it doesn't seem to have actually scheduled the test job. It's a little late for me to puzzle that out today.
Also, the (unchanged!) desktop-test image had lots of "errors" like this: gyp WARN EACCES user "worker" does not have permission to access the dev dir "/home/worker/.node-gyp/0.10.36" gyp WARN EACCES attempting to reinstall using temporary dev dir "/usr/local/lib/node_modules/taskcluster-vcs/node_modules/taskcluster-client/node_modules/sockjs-client-node/node_modules/jsdom/node_modules/contextify/.node-gyp" which didn't actually stop the docker build. I'm not sure why it's running install stuff as 'worker' and not root??
(In reply to Dustin J. Mitchell [:dustin] from comment #5) > Aww, it doesn't seem to have actually scheduled the test job. It's a little > late for me to puzzle that out today. How can you tell? The build hasn't started yet. But I've refactored the testing configs a fair amount since that try run, so don't spend too much time figuring out why it isn't working.
Oh, I was assuming the test jobs would appear light gray to start. You're right - it's in the graph. And still not started! Anyway, as long as it gets to the pactl error I'll be happy and put this up for review. Comment 6 has me worried, though.
Grr, and that build failed :(
Bug 1212411: fork docker images for tests; r?ahal This also refactors the image generation to not depend on puppetagain, and to install the latest and greatest mercurial in a verified, repeatable way. As with centos6-build-upd, the ubuntu1204-test-upd image exists to run 'apt-get upgrade' periodically without regenerating the huge parent image.
Attachment #8671938 - Flags: review?(ahalberstadt)
Attachment #8671938 - Flags: review?(ahalberstadt) → review+
Comment on attachment 8671938 [details] MozReview Request: Bug 1212411: fork docker images for tests; r?ahal https://reviewboard.mozilla.org/r/21653/#review19477 Thanks, this looks much more organized than before! Disclaimer: this kind of stuff is not my area of expertise.. so if you're looking for more than a nominal r+, I'd recommend also asking someone else. ::: testing/docker/desktop-test/VERSION:1 (Diff revision 1) > +0.4.0 I think this image should be called something like "linux-tester" instead of "desktop-test". B2G and android emulator tests will likely be using the same image, so 'desktop' could be misleading. ::: testing/docker/desktop-test/dot-config/user-dirs.dirs:7 (Diff revision 1) > +# nit: extra comment/whitespace ::: testing/docker/desktop-test/dot-pulse/default.pa:1 (Diff revision 1) > +#!/usr/bin/pulseaudio -nF Does this file fix the pactl issue? ::: testing/docker/ubuntu1204-test/Dockerfile:1 (Diff revision 1) > +FROM quay.io/mozilla/ubuntu:12.04 Why do we use mozilla/ubuntu as opposed to the official ones?
Good questions! I used 'desktop-test' to parallel 'desktop-build', both of which are intended to distinguish them from b2g. Maybe "gecko" would be a better name. Anyway, for the moment let's leave them matching one another. The dot-config and dot-pulse directories were copied over from the existing `tester` image. I'll clean up the whitespace, but I have no idea about the pactl issue -- maybe! Finally, you're right about starting with mozilla/ubuntu:12.04. We should start with the official upstream image. I'll fix that, regenerate and push images, push the patch. Note that, since your work to run tests hasn't landed yet, nothing's actually *using* these images even after I push this patch. If you'd be so kind as to update your task definitions to use {{#docker-image}}desktop-test{{/docker-image}} instead of tester, that'd be great.
I don't see any substantive differences between ubuntu:12.04 and quay.io/mozilla/ubuntu:12.04, by the way. Just zoneinfo and some apt caches and logs.
https://hg.mozilla.org/mozilla-central/rev/d9ea3b00f061
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Product: Taskcluster → Taskcluster Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: