switch as much of firefox-ci to 18.04 docker-worker base images
Categories
(Release Engineering :: Firefox-CI Administration, task)
Tracking
(Not tracked)
People
(Reporter: miles, Unassigned)
References
Details
Attachments
(1 file)
It is believed that there are several categories of tasks that fail on Ubuntu 18.04.
These include:
- tasks involving Valgrind
- ASAN builds
We should confirm that these tasks fail on Ubuntu 18.04 worker images intended for production use and evaluate whether it makes more sense to fix root causes on 18.04 or create a new set of images for 16.04.
|
||
Comment 1•6 years ago
|
||
in the last 3 months we have upgraded our linux test images from 16.04 to 18.04. I am not sure if the base image that docker runs on top of matters, but upgrading that would be interesting.
Could someone clarify what image we are talking about and what we run and where it is defined at?
|
||
Comment 2•6 years ago
|
||
(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #1)
Could someone clarify what image we are talking about and what we run and where it is defined at?
Good point of clarification.
Miles: for 18.04, are we talking about the base OS that runs docker, or the docker image itself?
|
Reporter | |
Comment 3•6 years ago
|
||
This bug is referring to the base OS image - my understanding is that the issues were with the base image.
The current production docker-worker images in AWS are built on Ubuntu 14.04, which is deprecated. The overarching goal is to move to a supported LTS Ubuntu release, and 18.04 is current. Current 14.04 images are built with a process here [0]. We have a tool that we've been using to bake the 18.04 images that we'd like to use in various clouds here [1].
I'm not sure when the testing that resulted in our understanding that Valgrind and ASAN builds were broken on 18.04 happened, but it might be worth re-doing so that we have up-to-date state of what is broken.
[0] https://github.com/taskcluster/docker-worker/tree/master/deploy
[1] https://github.com/taskcluster/monopacker/
|
|
||
Comment 4•6 years ago
|
||
if there is needs to help with testing how this affects our builds/tests, typically it is best to stand up a small pool and push to try. This isn't a hard problem to test, and I am happy to help with what to run and interpret the results, etc.
|
||
Comment 5•6 years ago
|
||
Looking at a recent autoland push, the count of worker pools was
1 gecko-3/t-linux-xlarge
973 gecko-t/t-linux-large
108 gecko-t/t-linux-metal
1328 gecko-t/t-linux-xlarge
81 gecko-t/t-linux-xlarge-source
of those, only metal is using an 18.04 image.
So I think the thing to do is what Joel suggested: stand up a pool using the (existing) monopacker image and run a try job against it. What passes can stay on that worker pool, an what doesn't can remain behind and get triaged by the folks who know more about those test cases.
|
|
||
Comment 6•6 years ago
|
||
This was a little premature since the worker pools don't exist (bug 1636284), but should at least be easy to mach try again with:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=4e4255cc8d1b7b55678301ef6fd10eae015d2a15
|
|
||
Comment 7•6 years ago
|
||
The t-linux-* workers are in the process of transitioning to images based on
Ubuntu 18.04 in https://bugzilla.mozilla.org/show_bug.cgi?id=1580575.
t-linux-metal is already running 18.04; for the rest this change maps
<workertype> to a <workertype>-1804 pool and <workertype>-1404 to the existing
<workertype> pool. Once testing is complete, we can rename the underlying
pools with a change to these aliases.
|
|
||
Comment 8•6 years ago
|
||
|
|
||
Comment 9•6 years ago
|
||
|
|
||
Comment 10•6 years ago
|
||
(In reply to Dustin J. Mitchell [:dustin] (he/him) from comment #6)
This was a little premature since the worker pools don't exist (bug 1636284), but should at least be easy to
mach try againwith:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=4e4255cc8d1b7b55678301ef6fd10eae015d2a15
I've tried another push here: https://treeherder.mozilla.org/#/jobs?repo=try&revision=3a7229a3cbecbdd4b56877926a553aa4dc5e0786
|
|
||
Comment 11•6 years ago
|
||
(In reply to Chris Cooper [:coop] pronoun: he from comment #10)
I've tried another push here: https://treeherder.mozilla.org/#/jobs?repo=try&revision=3a7229a3cbecbdd4b56877926a553aa4dc5e0786
Here's the list of failures that remain after stripping out intermittents:
- Linux 18.04 x64 debug M-fis(bc1)
- Linux 18.04 x64 debug M-fis[tier-2](dt6)
- Linux 18.04 x64 asan opt M(gl2c)
The marionette tests also seem prone to timing failures, although they do occasionally succeed: https://treeherder.mozilla.org/#/jobs?repo=try&selectedTaskRun=QH9SN6jLRRioXkSrToDYFQ-0&revision=3a7229a3cbecbdd4b56877926a553aa4dc5e0786&searchStr=marionette
I thought Valgrind was working, but it turns out that worker still needs to be updated to use 18.04.
|
|
||
Comment 12•6 years ago
|
||
you can use push health to help out- typically you can confirm with some retriggers that intermittents are real (I did a few more to help out):
https://treeherder.mozilla.org/pushhealth.html?repo=try&revision=3a7229a3cbecbdd4b56877926a553aa4dc5e0786
|
|
||
Comment 13•6 years ago
|
||
I updated the phab patch to use builders too.
|
|
||
Comment 14•6 years ago
|
||
(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #12)
you can use push health to help out- typically you can confirm with some retriggers that intermittents are real (I did a few more to help out):
https://treeherder.mozilla.org/pushhealth.html?repo=try&revision=3a7229a3cbecbdd4b56877926a553aa4dc5e0786
Thanks, Joel. That's really useful!
|
|
||
Comment 15•6 years ago
|
||
I don't see any -perf workers here, so I think the patch works as-is for the update to bug 1636284.
|
|
||
Comment 16•6 years ago
|
||
(In reply to Chris Cooper [:coop] pronoun: he from comment #11)
I thought Valgrind was working, but it turns out that worker still needs to be updated to use 18.04.
I have a new push running that uses 18.04 builders: https://treeherder.mozilla.org/#/jobs?repo=try&revision=4f8cfe1788fa7fa0a96b00461e6aaa76306fd36a
|
|
||
Updated•5 years ago
|
|
||
Comment 17•3 years ago
|
||
As far as I can tell, everything is on Ubuntu 18.04 or higher, or Debian at this point.
Description
•