1367491 - esr52: docker image creation is unreliable

Status: RESOLVED WONTFIX

(Release Engineering :: Release Automation: Other, enhancement, P3)

Description

[Aki Sasaki (not active)]

We're failing and retrying a lot in our docker image creation for releasetasks.
Sometimes 5 taskcluster retries is enough; occasionally not.  Part of the reason is clamav db updates are flaky and we run those updates at docker image creation time.

We can possibly move the db update to a periodic process, and the docker image creation tasks could rely on the latest successful artifacts.  If we do so, we should probably have a check somewhere for freshness, so silently failed clamav db update tasks don't result in old virus definitions.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Recently we've been hitting lots of hg clone errors like 
[vcs 2017-07-13T22:34:34.078707Z] updating [=======================================>          ] 155500/194126 30s
[vcs 2017-07-13T22:34:34.464456Z] updating [=======================================>          ] 156700/194126 29s
[vcs 2017-07-13T22:34:34.464521Z]                                                                                 
[vcs 2017-07-13T22:34:34.489589Z] abort: No such file or directory: '/home/worker/checkouts/gecko/addon-sdk/source/test/addons/toolkit-require-reload/main.js'

gps pointed me to https://github.com/mozilla/normandy/issues/757#issuecomment-303526887 for how to avoid AUFS+docker flakiness. It's not immediately obvious what needs to be done there, but maybe imagebuilder didn't get some fixes that compile tasks have ?

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Strongly suspect https://hg.mozilla.org/mozilla-central/rev/84fd52d2832a#l4.14 fixes the problem here. See also bug 1394657 for new retry handling.

Comment 3

[Sylvestre Ledru [:Sylvestre]]

Aki, do we care about that?

Comment 4

[Aki Sasaki (not active)]

WONTFIX for esr52 :)