Closed Bug 1402324 Opened 9 years ago Closed 8 years ago

Task dir including timestamp may break caching feature

Categories

(Taskcluster :: Workers, enhancement)

enhancement
Not set
normal

Tracking

(Not tracked)

RESOLVED DUPLICATE of bug 1499063

People

(Reporter: gerard-majax, Unassigned)

Details

Context: OSX DeepSpeech. We have a few dependencies installed over brew, and I want to keep as much as possible of brew within the task. Doing > brew install xxx Consumes quite some time, thus wasting resources. I've setup the use of cache as available in generic-worker, and when taking a look at one cache, I see things that will be problematic in the future when restoring that cache: > /Users/build-user/TaskCluster/LightTasks/2/caches/SWwmggGvRCGuKkCCwCUqCw//lib/node_modules/npm/npmrc:prefix = /Users/build-user/TaskCluster/LightTasks/2/tasks/task_1506083883/homebrew The varying timestamp in "task_TIMESTAMP" is likely to break a lot of the packages installed. Current workaround would be to install Homebrew above "task_TIMESTAMP", but that's far from being a good idea in general ...
Symlinks can also be affected: > ds-tc-worker:~ tc-gh-ds$ ls -hal /Users/build-user/TaskCluster/LightTasks/2/tasks/task_1506087223/homebrew/bin/npm lrwxr-xr-x 1 build-user staff 109B 22 sep 15:16 /Users/build-user/TaskCluster/LightTasks/2/tasks/task_1506087223/homebrew/bin/npm -> /Users/build-user/TaskCluster/LightTasks/2/tasks/task_1506083883/homebrew/lib/node_modules/npm/bin/npm-cli.js
Thanks for raising this. Although there are of course workarounds to this, I see this is less than ideal. Changing the worker to always use the same directory is possible, but quite a large change that could potentially affect a lot of existing jobs. Probably it is worth it though - it will require a lot of testing before rolling out to all of our worker types though. As you have workarounds in place now, I'll consider this as a candidate for a generic-worker 11 release, which may have to get moved to a later release if other critical non-backwardly-compatible changes necessitate a more immediate release.
Pete, maybe a half-way solution could be a configuration flag to give a stable name to "task_TIMESTAMP" ?
QA Contact: pmoore
(In reply to Alexandre LISSY :gerard-majax from comment #3) > Pete, maybe a half-way solution could be a configuration flag to give a > stable name to "task_TIMESTAMP" ? Indeed, that could be an option.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → DUPLICATE
Component: Generic-Worker → Workers
You need to log in before you can comment on or make changes to this bug.