Open Bug 1841640 Opened 2 years ago Updated 2 years ago

The application-services mac workers keep data around between builds

Categories

(Infrastructure & Operations :: RelOps: Posix OS, task)

Tracking

(Not tracked)

People

(Reporter: bdk, Unassigned)

References

Details

(Whiteboard: [relops-mac])

Our mac worker seems to keep around directories between builds, which causes taskcluster errors like this one. The task worked once, but the second time around the artifacts directory was still there from the last run which made it fail.

Is there a way to get a fresh build directory each time? Alternatively, is there something we could be doing differently in our taskcluster code?

Whiteboard: [relops-mac]

This looks like a l3 worker, so it uses generic-worker multiuser (which creates a new user per task) so we shouldn't be seeing task interference.

Looking at the output, it seems like the task is working inside it's task directory (so even without multiuser tasks shouldn't interfere with each other).

Checking out the worker, I see that /var/opt/generic-worker/tasks/task_168661858129484/ is being used for multiple tasks... not sure why.

:masterwayz or :pmoore, any ideas?

Flags: needinfo?(pmoore)
Flags: needinfo?(mgoossens)

It could be caused by something like this:
https://github.com/mozilla-platform-ops/ronin_puppet/pull/477/files

If that isn't it, I can take a closer look.

Flags: needinfo?(pmoore)

I can't find worker logs in papertrail - do they go somewhere else?
To be clear, numberOfTasksToRun should be 0, not 1.

I think that was a troubleshooting step for something unrelated, I'll fix that in a PR and quick rollout to just those workers.
No idea what could cause that folder directory issue.

Flags: needinfo?(mgoossens)
You need to log in before you can comment on or make changes to this bug.