1508016 - Mass test failures because of docker images jobs being added

Status: RESOLVED FIXED

(Taskcluster :: Workers, enhancement)

Description

[Cosmin Sabou [:CosminS]]

There is a mass test failure because of issues with docker images.
The landing of Bug 1501796 here https://hg.mozilla.org/integration/autoland/rev/c62c378d4f58e2d82a52b329e312cf9f0758cbc1 caused the deb7 job to end up in exception 

https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=pending%2Crunning%2Csuccess%2Ctestfailed%2Cbusted%2Cexception&searchStr=docker&fromchange=c62c378d4f58e2d82a52b329e312cf9f0758cbc1&tochange=b3412b91afe2f6fabaf3a9b13c65e8ee45d9c1b4&selectedJob=212314803

This was backed out by https://hg.mozilla.org/integration/autoland/rev/ed2f2725f3206c94fe1bd27dfc7d138a3c249dcd

My mistake was that I added the docker jobs on that push to see if it was fixed, this screwed up the docker images. 
After Bug 1501796 landed here https://hg.mozilla.org/integration/autoland/rev/b3412b91afe2f6fabaf3a9b13c65e8ee45d9c1b4 it triggered new docker jobs and after that, we had mass test failures on autoland and inbound. 

Did a backout on the latest bug landing but the tests kept on failing: https://hg.mozilla.org/integration/autoland/rev/e3f7bc3a94a158807643cdc0b3253e469e529f0f 

:glandium tried to help and change https://tools.taskcluster.net/index/gecko.cache.level-3.docker-images.v1.desktop1604-test.hash/e31481d8ea76de77fe045766d851f7eb2113ff6cf3b59ce810bdb3b5ebb9bbc3 to point to K-qJkvIqQQ-EFK8-4zg_nA but was unsuccesful. Trees are still closed because of this.

Failures on autoland: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception

Comment 1

[Mike Hommey [:glandium]]

I reset gecko.cache.level-3.docker-images.v1.desktop1604-test.hash.e31481d8ea76de77fe045766d851f7eb2113ff6cf3b59ce810bdb3b5ebb9bbc3 to K-qJkvIqQQ-EFK8-4zg_nA (which was the ubuntu image before the forced retrigger) with https://tools.taskcluster.net/groups/MBQNl35TSreTqQrrxJMWOg/tasks/MBQNl35TSreTqQrrxJMWOg/details