1948133 - Enable pip cache for Windows test / perf tasks

Status: NEW

(Firefox Build System :: Task Configuration, task)

Description

[Andrew Halberstadt [:ahal]]

In bug 1937801 I enabled pip / uv caches for test tasks. However the perf ones had permission denied failures on Windows, here's an example log:
https://treeherder.mozilla.org/logviewer?job_id=494253132&repo=autoland&lineNumber=833

[task 2025-02-10T19:48:08.062Z] 19:48:08     INFO -  Collecting aiohttp==3.7.4.post0 (from -r C:\task_173921162463007\build\tests\config\mozbase_requirements.txt (line 27))
[task 2025-02-10T19:48:08.062Z] 19:48:08     INFO -    Using cached aiohttp-3.7.4.post0-py3-none-any.whl
[task 2025-02-10T19:48:08.078Z] 19:48:08     INFO -  ERROR: Could not install packages due to an OSError: [Errno 13] Permission denied: 'C:\\task_173921162463007\\.task-cache\\pip\\wheels\\01\\52\\ef\\8085eea725d72c04bcb908b26b905377225b380a96b993639a\\aiohttp-3.7.4.post0-py3-none-any.whl'
[task 2025-02-10T19:48:08.078Z] 19:48:08     INFO -  Check the permissions.
[task 2025-02-10T19:48:08.156Z] 19:48:08     INFO - Return code: 1
[task 2025-02-10T19:48:08.156Z] 19:48:08    FATAL - Could not install python package: failed all attempts.
[task 2025-02-10T19:48:08.156Z] 19:48:08    FATAL - Running post_fatal callback...
[task 2025-02-10T19:48:08.156Z] 19:48:08    FATAL - Exiting -1

Rather than resolve the issue, I disabled the caches so I could get the rest of the stack landed (as I didn't have time to look into it). This bug is the follow-up.

It occurs to me that if this is a problem with g-w multiuser on Windows, it's possible that the caches are not working for regular test tasks either. More investigation is required.

Comment 1

[Andrew Halberstadt [:ahal]]

Bastien did some digging and indeed it looks like this is a bug in generic-worker:
https://github.com/taskcluster/taskcluster/issues/7532

And this is affecting all test tasks (I think it just so happens that the cache isn't even getting used on Windows tests). So we need to:

1. Fix the g-w issue above and roll it out to test pools

2. Re-enable caches for Windows perf / hw tasks , see:
   https://hg.mozilla.org/integration/autoland/rev/8a0e41f86a99
   https://hg.mozilla.org/mozilla-central/rev/083fcba009fc

3. Figure out why caches aren't being used for other test tasks and fix it

Comment 2

[:jmoss]

(In reply to Andrew Halberstadt [:ahal] from comment #1)

Bastien did some digging and indeed it looks like this is a bug in generic-worker:
https://github.com/taskcluster/taskcluster/issues/7532

And this is affecting all test tasks (I think it just so happens that the cache isn't even getting used on Windows tests). So we need to:

1. Fix the g-w issue above and roll it out to test pools

2. Re-enable caches for Windows perf / hw tasks , see:
   https://hg.mozilla.org/integration/autoland/rev/8a0e41f86a99
   https://hg.mozilla.org/mozilla-central/rev/083fcba009fc

3. Figure out why caches aren't being used for other test tasks and fix it

I'll roll it out to gecko-t/win11-64-24h2-hw-alpha where this primarily happens once taskcluster folks create a release.

Comment 3

[:jmoss]

gecko-t/win11-64-24h2-hw-alpha is now running generic worker 81.0.3, which includes the fix.

Comment 4

[Andrew Halberstadt [:ahal]]

Awesome, thanks! It might be a little bit before I have a chance to revisit this. I'll leave the needinfo as a reminder.