1748781 - Perma [tier 2] PermissionError: [WinError 5] Access is denied: 'z:/build/build/src/obj-x86_64-pc-mingw32\\_virtualenvs\\common\\Scripts\\python.exe'

Status: RESOLVED FIXED

(Firefox Build System :: Mach Core, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: malexandru [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=363198694&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/ZsHfvyzvSHyeMtR2xJHhbA/runs/0/artifacts/public/logs/live_backing.log

---

[task 2022-01-06T04:23:35.339Z] 04:23:35     INFO - Return code: 0
[task 2022-01-06T04:23:38.730Z] Error running mach:
[task 2022-01-06T04:23:38.730Z] 
[task 2022-01-06T04:23:38.730Z]     ['--log-no-times', 'static-analysis', 'autotest', '--intree-tool']
[task 2022-01-06T04:23:38.730Z] 
[task 2022-01-06T04:23:38.730Z] The error occurred in code that was called by the mach command. This is either
[task 2022-01-06T04:23:38.730Z] a bug in the called code itself or in the way that mach is calling it.
[task 2022-01-06T04:23:38.730Z] You can invoke |./mach busted| to check if this issue is already on file. If it
[task 2022-01-06T04:23:38.730Z] isn't, please use |./mach busted file static-analysis| to report it. If |./mach busted| is
[task 2022-01-06T04:23:38.730Z] misbehaving, you can also inspect the dependencies of bug 1543241.
[task 2022-01-06T04:23:38.730Z] 
[task 2022-01-06T04:23:38.730Z] If filing a bug, please include the full output of mach, including this error
[task 2022-01-06T04:23:38.730Z] message.
[task 2022-01-06T04:23:38.730Z] 
[task 2022-01-06T04:23:38.730Z] The details of the failure are as follows:
[task 2022-01-06T04:23:38.730Z] 
[task 2022-01-06T04:23:38.730Z] PermissionError: [WinError 5] Access is denied: 'z:/build/build/src/obj-x86_64-pc-mingw32\\_virtualenvs\\common\\Scripts\\python.exe'
[task 2022-01-06T04:23:38.730Z] 
[task 2022-01-06T04:23:38.730Z]   File "z:\build\build\src\python/mozbuild/mozbuild/code_analysis/mach_commands.py", line 616, in autotest
[task 2022-01-06T04:23:38.730Z]     command_context.activate_virtualenv()
[task 2022-01-06T04:23:38.730Z]   File "z:\build\build\src\python\mozbuild\mozbuild\base.py", line 878, in activate_virtualenv
[task 2022-01-06T04:23:38.730Z]     self.virtualenv_manager.activate()
[task 2022-01-06T04:23:38.730Z]   File "z:\build\build\src\python\mach\mach\site.py", line 531, in activate
[task 2022-01-06T04:23:38.730Z]     self.ensure()
[task 2022-01-06T04:23:38.733Z]   File "z:\build\build\src\python\mach\mach\site.py", line 520, in ensure
[task 2022-01-06T04:23:38.733Z]     self._metadata,
[task 2022-01-06T04:23:38.733Z]   File "z:\build\build\src\python\mach\mach\site.py", line 1045, in _create_venv_with_pthfile
[task 2022-01-06T04:23:38.733Z]     shutil.rmtree(virtualenv_root)
[task 2022-01-06T04:23:38.733Z]   File "c:\mozilla-build\python3\lib\shutil.py", line 494, in rmtree
[task 2022-01-06T04:23:38.733Z]     return _rmtree_unsafe(path, onerror)
[task 2022-01-06T04:23:38.733Z]   File "c:\mozilla-build\python3\lib\shutil.py", line 384, in _rmtree_unsafe
[task 2022-01-06T04:23:38.733Z]     _rmtree_unsafe(fullname, onerror)
[task 2022-01-06T04:23:38.733Z]   File "c:\mozilla-build\python3\lib\shutil.py", line 389, in _rmtree_unsafe
[task 2022-01-06T04:23:38.733Z]     onerror(os.unlink, fullname, sys.exc_info())
[task 2022-01-06T04:23:38.733Z]   File "c:\mozilla-build\python3\lib\shutil.py", line 387, in _rmtree_unsafe
[task 2022-01-06T04:23:38.733Z]     os.unlink(fullname)
[task 2022-01-06T04:23:38.761Z] 04:23:38  WARNING - setting return code to 2
[task 2022-01-06T04:23:38.761Z] 04:23:38    FATAL - 'mach static-analysis autotest --intree-tool' did not run successfully. Please check log for errors.
[task 2022-01-06T04:23:38.761Z] 04:23:38    FATAL - Running post_fatal callback...
[task 2022-01-06T04:23:38.777Z] 04:23:38    FATAL - Exiting -1
[task 2022-01-06T04:23:38.777Z] 04:23:38     INFO - [mozharness: 2022-01-06 04:23:38.777314Z] Finished static-analysis-autotest step (failed)
[task 2022-01-06T04:23:38.777Z] 04:23:38     INFO - Running post-run listener: _parse_build_tests_ccov
[task 2022-01-06T04:23:38.777Z] 04:23:38     INFO - Running command: ['z:\\build\\build\\src\\obj-x86_64-pc-mingw32\\_virtualenvs\\common\\Scripts\\python.exe', 'mach', 'python', 'testing\\parse_build_tests_ccov.py'] in z:/build/build/src
[task 2022-01-06T04:23:38.777Z] 04:23:38     INFO - Copy/paste: z:\build\build\src\obj-x86_64-pc-mingw32\_virtualenvs\common\Scripts\python.exe mach python testing\parse_build_tests_ccov.py
<...>
[task 2022-01-06T04:23:38.797Z] 04:23:38     INFO -  Fatal Python error: Py_Initialize: unable to load the file system codec
[task 2022-01-06T04:23:38.797Z] 04:23:38     INFO -  ModuleNotFoundError: No module named 'encodings'
[task 2022-01-06T04:23:38.797Z] 04:23:38     INFO -  Current thread 0x00000460 (most recent call first):
[task 2022-01-06T04:23:39.339Z] 04:23:39    ERROR - Return code: 3221226505
[task 2022-01-06T04:23:39.339Z] 04:23:39    ERROR - 3221226505 not in success codes: [0]
[task 2022-01-06T04:23:39.339Z] 04:23:39    FATAL - Halting on failure while running ['z:\\build\\build\\src\\obj-x86_64-pc-mingw32\\_virtualenvs\\common\\Scripts\\python.exe', 'mach', 'python', 'testing\\parse_build_tests_ccov.py']
[task 2022-01-06T04:23:39.354Z] 04:23:39    FATAL - Running post_fatal callback...
[task 2022-01-06T04:23:39.354Z] 04:23:39    FATAL - Exiting 2
[task 2022-01-06T04:23:39.371Z] Creating default state directory: C:\Users\task_164144173370065/.mozbuild
[taskcluster 2022-01-06T04:23:39.445Z]    Exit Code: 2
[taskcluster 2022-01-06T04:23:39.445Z]    User Time: 0s
[taskcluster 2022-01-06T04:23:39.445Z]  Kernel Time: 31.25ms
[taskcluster 2022-01-06T04:23:39.445Z]    Wall Time: 17m52.0985883s
[taskcluster 2022-01-06T04:23:39.445Z]       Result: FAILED
[taskcluster 2022-01-06T04:23:39.445Z] === Task Finished ===
[taskcluster 2022-01-06T04:23:39.445Z] Task Duration: 17m52.3015913s
[taskcluster 2022-01-06T04:23:39.502Z] Uploading artifact public/logs/localconfig.json from file logs\localconfig.json with content encoding "gzip", mime type "application/json" and expiry 2023-01-06T03:50:05.679Z
[taskcluster 2022-01-06T04:23:39.697Z] [mounts] Preserving cache: Moving "Z:\\task_164144173370065\\build" to "Z:\\caches\\HhMOqj5LTZKVjUKce37H7A"
[taskcluster 2022-01-06T04:23:40.308Z] [mounts] Denying task_164144173370065 access to 'Z:\caches\HhMOqj5LTZKVjUKce37H7A'
[taskcluster 2022-01-06T04:23:55.405Z] Uploading link artifact public/logs/live.log to artifact public/logs/live_backing.log with expiry 2023-01-06T03:50:05.679Z
[taskcluster:error] exit status 2

Comment 1

[Alexandru Michis [:malexandru]]

Mitchell, could this be caused by one of your changes in Bug 1725895 or Bug 1740123?

Comment 2

[Mitchell Hentges [:mhentges] 🦀]

I don't think it's caused by either of those two but it's definitely related to the consistent import scope changes. I'll dig into this tomorrow.

Comment 3

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1748781: Don't re-include site's prefix in import scope

When the redundant-site-package-dir pruning was originally implemented,
it assumed that only <venv>/lib/<site-packages-dir> was automatically
added to the sys.path. However, it looks like some platforms
(including Windows) also add the prefix of the venv itself to the
sys.path.

We can work around this by removing both from the sys.path carry-over,
and the deprioritization logic.

Comment 4

[Intermittent Failures Robot]

19 failures in 2801 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 19

Platform and build breakdown:

• windows2012-64: 19
  • debug: 19

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1748781&startday=2022-01-03&endday=2022-01-09&tree=all

Comment 5

[Intermittent Failures Robot]

23 failures in 3060 pushes (0.008 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 23

Platform and build breakdown:

• windows2012-64: 23
  • debug: 23

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1748781&startday=2022-01-10&endday=2022-01-16&tree=all

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2f9f43882af6
Don't re-include site's prefix in import scope r=ahal

Comment 7

[Cristina Cozmuta (:CrissCozmuta)]

https://hg.mozilla.org/mozilla-central/rev/2f9f43882af6

Comment 8

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1730712

Comment 9

[BugBot [:suhaib / :marco/ :calixte]]

The patch landed in nightly and beta is affected.
:mhentges, is this bug important enough to require an uplift?
If not please set status_beta to wontfix.

For more information, please visit auto_nag documentation.

Comment 10

[Intermittent Failures Robot]

7 failures in 3444 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 7

Platform and build breakdown:

• windows2012-64: 7
  • debug: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1748781&startday=2022-01-17&endday=2022-01-23&tree=all