1524637 - Workers do not clean up workspaces between runs

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Chris AtLee [:catlee]]

In this task:
https://tools.taskcluster.net/groups/CZAHDAJ4Sn6mVwhEP1AAcA/tasks/O-_hEYcmSra_uG5lyfDCsw/runs/0/artifacts

Several logs are uploaded:

• https://taskcluster-artifacts.net/O-_hEYcmSra_uG5lyfDCsw/0/public/logs/live_backing.log is the log from that task
• https://taskcluster-artifacts.net/O-_hEYcmSra_uG5lyfDCsw/0/public/logs//log_info.log is also uploaded, but shouldn't have been produced in the first place

Looking more closely, the timestamps don't match, and the build they're trying to fetch is different:

installer-url=https://queue.taskcluster.net/v1/task/Y626dlWcSq2W94v8a0hOrA/artifacts/public/build/target.tar.bz2
installer-url=https://queue.taskcluster.net/v1/task/Oq4FVRmeTHeIjG3c1HPuiw/artifacts/public/build/target.tar.bz2

This is a pretty serious problem because it means that builds/tests could be uploading artifacts from previous runs on the machine.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

{
  "artifacts": {
    "public/logs/": {
      "expires": "2019-02-15T17:43:41.329Z",
      "path": "/builds/worker/workspace/logs/",
      "type": "directory"
    },
    ...
    "public/test_info/": {
      "expires": "2019-02-15T17:43:41.329Z",
      "path": "/builds/worker/workspace/build/blobber_upload_dir/",
      "type": "directory"
    }
  },
  "cache": {
    ...
    "level-1-try-test-workspace-v3-b1a0a2cc55b8f4544db1": "/builds/worker/workspace"
  },
}

so basically this is doing what the task says: caching the workspace, and uploading files in that workspace. Either those logs should not be on a cache, or the task should be more careful to clean the cache at startup..

Comment 2

[Chris AtLee [:catlee]]

Attachment: Bug 1524637: Clean up workspace directories before running the test r=tomprince

Comment 3

[Chris AtLee [:catlee]]

Attachment: Bug 1524637: Disable workerspace cache for mozharness test jobs r=tomprince

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/88f45113318e
Disable workerspace cache for mozharness test jobs r=tomprince

Comment 5

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/88f45113318e