Closed Bug 1294108 Opened 8 years ago Closed 8 years ago

Docker Image task "desktop1604-test" fails in clean-up step

Categories

(Taskcluster Graveyard :: Docker Images, defect)

Product:
Taskcluster Graveyard
Component:
Docker Images
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED DUPLICATE of bug 1286279

People

(Reporter: whimboo, Unassigned)

References

Details

I have seen this failure today twice in a row: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=ed7a4daa831ac3c547959854e44585dd62fad508&selectedJob=4619417 https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=b83235ed0ea5a46e58ee9e6f402171bf908ac25f&selectedJob=4619307 It looks like that we somehow fail to remove the current script at the very end. # clean up cd / rm -rf /setup ~/.ccache ~/.cache ~/.npm apt-get clean apt-get autoclean Reading package lists... Building dependency tree... Reading state information... rm -f $0 FATA[1269] unexpected EOF [taskcluster 2016-08-10 14:13:07.127Z] === Task Finished ===
It also looks like that this task doesn't upload logs which can be parsed by Treeherder. So bug suggestions do not work.
I've seen this frequently as well. Pretty sure it's bug (possibly a timeout) in the Docker image generation system. I've noticed this tends to occur when there are a lot of images being generated at the same time. The taskcluster-images worker currently has a capacity of 4. Attempting to create, save, and upload multiple 3+ GB images simultaneously will likely overwhelm a machine. I've lowered the capacity of taskcluster-images to 2 to help mitigate this failure.
I wonder if we could also reduce the size of the new Ubuntu 16.04 docker image. It looks like we have a complete desktop system with all the applications (like LibreOffice) installed we never use. Could we do that as well to ensure have faster task execution times, and especially reduce the amount of MB to be downloaded for each dependent task.
Bug 1294264 tracks reducing the image size. Lowering the worker capacity to 2 does seem to have helped increase the success rate of Docker image generation.
The "FATA" error is a docker client error, not an rm error. I suspect that this is bug 1286279 or at least shares a root cause.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → DUPLICATE
Product: Taskcluster → Taskcluster Graveyard
You need to log in before you can comment on or make changes to this bug.