Closed
Bug 1397503
Opened 7 years ago
Closed 7 years ago
error: volume /builds/worker/checkouts is not empty on docker-image tasks
Categories
(Firefox Build System :: Task Configuration, task)
Firefox Build System
Task Configuration
Tracking
(Not tracked)
RESOLVED
FIXED
mozilla57
People
(Reporter: gps, Assigned: gps)
References
Details
Attachments
(2 files)
docker-image tasks can fail with something like: [taskcluster 2017-09-06 22:13:19.652Z] Task ID: Huoxt2uWSACT1VzE0wuhyA [taskcluster 2017-09-06 22:13:19.652Z] Worker ID: i-0266709d74502174c [taskcluster 2017-09-06 22:13:19.652Z] Worker Group: us-east-1 [taskcluster 2017-09-06 22:13:19.652Z] Worker Node Type: c4.2xlarge [taskcluster 2017-09-06 22:13:19.652Z] Worker Type: gecko-1-images [taskcluster 2017-09-06 22:13:19.652Z] Public IP: 34.201.137.98 [taskcluster 2017-09-06 22:13:19.652Z] using cache "level-1-imagebuilder-v1" -> /builds/worker/checkouts [taskcluster 2017-09-06 22:13:22.694Z] === Task Starting === #!/bin/bash -vex # Set bash options to exit immediately if a pipeline exists non-zero, expand # print a trace of commands, and make output verbose (print shell input as it's # read) # See https://www.gnu.org/software/bash/manual/html_node/The-Set-Builtin.html set -x -e -v -o pipefail + set -x -e -v -o pipefail # Prefix errors with taskcluster error prefix so that they are parsed by Treeherder raise_error() { echo echo "[taskcluster-image-build:error] $1" exit 1 } # Ensure that the PROJECT is specified so the image can be indexed test -n "$PROJECT" || raise_error "PROJECT must be provided." + test -n try test -n "$HASH" || raise_error "Context HASH must be provided." + test -n 8995d4cfbf4610d74ed6c1bd3ba485184d6dc0cd0df1ec5e667780045b4a6836 test -n "$IMAGE_NAME" || raise_error "IMAGE_NAME must be provided." + test -n system-package-debian-stretch # Construct a CONTEXT_FILE CONTEXT_FILE=/builds/worker/workspace/context.tar + CONTEXT_FILE=/builds/worker/workspace/context.tar # Run ./mach taskcluster-build-image with --context-only to build context run-task \ --vcs-checkout "/builds/worker/checkouts/gecko" \ -- \ /builds/worker/checkouts/gecko/mach taskcluster-build-image \ --context-only "$CONTEXT_FILE" \ "$IMAGE_NAME" + run-task --vcs-checkout /builds/worker/checkouts/gecko -- /builds/worker/checkouts/gecko/mach taskcluster-build-image --context-only /builds/worker/workspace/context.tar system-package-debian-stretch [setup 2017-09-06T22:13:23.212863Z] run-task started error: volume /builds/worker/checkouts is not empty Our Docker image policy requires volumes to be empty. The volume was likely populated as part of building the Docker image. Change the Dockerfile and anything run from it to not create files in any VOLUME. A lesser possibility is that you stumbled upon a TaskCluster platform bug where it fails to use new volumes for tasks. entries in root directory: gecko hg-store --- Essentially, the cache name for these tasks should use the naming scheme we deployed in bug 1391476. The cache name was never changed. So when a modern run-task hits a cache on a worker from an old run-task, it barfs. The failure in run-task is expected and intentional: we want CI to uncover bugs like this. We just need to fix the cache names.
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment hidden (mozreview-request) |
Comment 10•7 years ago
|
||
mozreview-review |
Comment on attachment 8905302 [details] Bug 1397503 - Vary cache name when using out-of-tree Docker images; https://reviewboard.mozilla.org/r/177096/#review182342 ::: commit-message-84d3c:16 (Diff revision 4) > +Taskgraph. > + > +So, for out-of-tree Docker images we now vary the cache name by > +the Docker image name. This means that out-of-tree run-task tasks > +will get separate caches for each a) Docker image b) Docker image > +version. In fact, it varies on the image sha256 too, from what I can tell.
Attachment #8905302 -
Flags: review?(dustin) → review+
Comment 11•7 years ago
|
||
mozreview-review |
Comment on attachment 8905290 [details] Bug 1397503 - Perform cache sanitization for image_builder tasks; https://reviewboard.mozilla.org/r/177090/#review182340 ::: commit-message-c9593:6 (Diff revision 5) > +Bug 1397503 - Perform cache sanitization for image_builder tasks; r?dustin > + > +The image_builder Docker image doesn't set a "command" in its task > +definition. The image instead relies on a RUN in its Dockerfile to > +control the started command. This command is a shell script which > +eventually runs run-task. I tripped over this while hacking on the /home -> /builds migration. I tried to rewrite it to use run-task explicitly, but it got complicated quickly so I backed off.
Attachment #8905290 -
Flags: review?(dustin) → review+
Comment hidden (mozreview-request) |
Comment 13•7 years ago
|
||
Pushed by gszorc@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/fbaaf63c4dd6 Perform cache sanitization for image_builder tasks; r=dustin https://hg.mozilla.org/integration/autoland/rev/34529d61b6a8 Vary cache name when using out-of-tree Docker images; r=dustin
Comment 14•7 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/fbaaf63c4dd6 https://hg.mozilla.org/mozilla-central/rev/34529d61b6a8
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla57
Updated•6 years ago
|
Product: TaskCluster → Firefox Build System
You need to log in
before you can comment on or make changes to this bug.
Description
•