Closed Bug 1397503 Opened 2 years ago Closed 2 years ago

error: volume /builds/worker/checkouts is not empty on docker-image tasks

Categories

(Firefox Build System :: Task Configuration, task)

task
Not set

Tracking

(Not tracked)

RESOLVED FIXED
mozilla57

People

(Reporter: gps, Assigned: gps)

References

Details

Attachments

(2 files)

docker-image tasks can fail with something like:

[taskcluster 2017-09-06 22:13:19.652Z] Task ID: Huoxt2uWSACT1VzE0wuhyA
[taskcluster 2017-09-06 22:13:19.652Z] Worker ID: i-0266709d74502174c
[taskcluster 2017-09-06 22:13:19.652Z] Worker Group: us-east-1
[taskcluster 2017-09-06 22:13:19.652Z] Worker Node Type: c4.2xlarge
[taskcluster 2017-09-06 22:13:19.652Z] Worker Type: gecko-1-images
[taskcluster 2017-09-06 22:13:19.652Z] Public IP: 34.201.137.98
[taskcluster 2017-09-06 22:13:19.652Z] using cache "level-1-imagebuilder-v1" -> /builds/worker/checkouts

[taskcluster 2017-09-06 22:13:22.694Z] === Task Starting ===
#!/bin/bash -vex

# Set bash options to exit immediately if a pipeline exists non-zero, expand
# print a trace of commands, and make output verbose (print shell input as it's
# read)
# See https://www.gnu.org/software/bash/manual/html_node/The-Set-Builtin.html
set -x -e -v -o pipefail
+ set -x -e -v -o pipefail

# Prefix errors with taskcluster error prefix so that they are parsed by Treeherder
raise_error() {
  echo
  echo "[taskcluster-image-build:error] $1"
  exit 1
}

# Ensure that the PROJECT is specified so the image can be indexed
test -n "$PROJECT"    || raise_error "PROJECT must be provided."
+ test -n try
test -n "$HASH"       || raise_error "Context HASH must be provided."
+ test -n 8995d4cfbf4610d74ed6c1bd3ba485184d6dc0cd0df1ec5e667780045b4a6836
test -n "$IMAGE_NAME" || raise_error "IMAGE_NAME must be provided."
+ test -n system-package-debian-stretch

# Construct a CONTEXT_FILE
CONTEXT_FILE=/builds/worker/workspace/context.tar
+ CONTEXT_FILE=/builds/worker/workspace/context.tar

# Run ./mach taskcluster-build-image with --context-only to build context
run-task \
  --vcs-checkout "/builds/worker/checkouts/gecko" \
  -- \
  /builds/worker/checkouts/gecko/mach taskcluster-build-image \
  --context-only "$CONTEXT_FILE" \
  "$IMAGE_NAME"
+ run-task --vcs-checkout /builds/worker/checkouts/gecko -- /builds/worker/checkouts/gecko/mach taskcluster-build-image --context-only /builds/worker/workspace/context.tar system-package-debian-stretch
[setup 2017-09-06T22:13:23.212863Z] run-task started

error: volume /builds/worker/checkouts is not empty

Our Docker image policy requires volumes to be empty.

The volume was likely populated as part of building the Docker image.
Change the Dockerfile and anything run from it to not create files in
any VOLUME.

A lesser possibility is that you stumbled upon a TaskCluster platform bug
where it fails to use new volumes for tasks.

entries in root directory: gecko hg-store


---

Essentially, the cache name for these tasks should use the naming scheme we deployed in bug 1391476. The cache name was never changed. So when a modern run-task hits a cache on a worker from an old run-task, it barfs.

The failure in run-task is expected and intentional: we want CI to uncover bugs like this. We just need to fix the cache names.
Comment on attachment 8905302 [details]
Bug 1397503 - Vary cache name when using out-of-tree Docker images;

https://reviewboard.mozilla.org/r/177096/#review182342

::: commit-message-84d3c:16
(Diff revision 4)
> +Taskgraph.
> +
> +So, for out-of-tree Docker images we now vary the cache name by
> +the Docker image name. This means that out-of-tree run-task tasks
> +will get separate caches for each a) Docker image b) Docker image
> +version.

In fact, it varies on the image sha256 too, from what I can tell.
Attachment #8905302 - Flags: review?(dustin) → review+
Comment on attachment 8905290 [details]
Bug 1397503 - Perform cache sanitization for image_builder tasks;

https://reviewboard.mozilla.org/r/177090/#review182340

::: commit-message-c9593:6
(Diff revision 5)
> +Bug 1397503 - Perform cache sanitization for image_builder tasks; r?dustin
> +
> +The image_builder Docker image doesn't set a "command" in its task
> +definition. The image instead relies on a RUN in its Dockerfile to
> +control the started command. This command is a shell script which
> +eventually runs run-task.

I tripped over this while hacking on the /home -> /builds migration.  I tried to rewrite it to use run-task explicitly, but it got complicated quickly so I backed off.
Attachment #8905290 - Flags: review?(dustin) → review+
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/fbaaf63c4dd6
Perform cache sanitization for image_builder tasks; r=dustin
https://hg.mozilla.org/integration/autoland/rev/34529d61b6a8
Vary cache name when using out-of-tree Docker images; r=dustin
https://hg.mozilla.org/mozilla-central/rev/fbaaf63c4dd6
https://hg.mozilla.org/mozilla-central/rev/34529d61b6a8
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla57
Product: TaskCluster → Firefox Build System
You need to log in before you can comment on or make changes to this bug.