1901564 - Frequent mach commands failure when auto-formatting is enabled, a toolchain update is available and clangd.exe is in use

Status: ASSIGNED

(Firefox Build System :: General, defect, P3)

Description

[Paul Adenot (:padenot)]

This happens in this common case:

• clangd.exe is being used (e.g., vscode is opened)
• A ./mach command is issued, that uses a clang tool
• An update is available for clang tools
• The developer is running Windows

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke ``./mach busted`` to check if this issue is already on file. If it
isn't, please use ``./mach busted file artifact`` to report it. If ``./mach busted`` is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

PermissionError: [WinError 5] Access is denied: 'C:\\Users\\paden\\.mozbuild\\clang-tools\\clang-tidy\\bin\\clangd.exe'

  File "C:\Users\paden\mozilla-unified\python\mozbuild\mozbuild\artifact_commands.py", line 590, in artifact_toolchain
    unpack_file(local)
  File "c:\users\paden\mozilla-unified\python\mozbuild\mozbuild\action\tooltool.py", line 991, in unpack_file
    clean_path(base_file)
  File "c:\users\paden\mozilla-unified\python\mozbuild\mozbuild\action\tooltool.py", line 942, in clean_path
    shutil.rmtree(dirname)
  File "C:\mozilla-build\python3\Lib\shutil.py", line 759, in rmtree
    return _rmtree_unsafe(path, onerror)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\mozilla-build\python3\Lib\shutil.py", line 617, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\mozilla-build\python3\Lib\shutil.py", line 622, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\mozilla-build\python3\Lib\shutil.py", line 620, in _rmtree_unsafe
    os.unlink(fullname)

Sentry event ID: 32f0d0a650354e3d8687226c67fafeb3
Sentry is attempting to send 0 pending error messages
Waiting up to 2 seconds
Press Ctrl-Break to quit

It's generally not OK to delete the file of a running program on Windows, so the correct thing to do is to close e.g. vscode and restart the command, but that isn't obvious.

Comment 1

[Paul Adenot (:padenot)]

Another variant of this is sccache.exe:

 1:43.37 Error running mach:
 1:43.38     mach --log-no-times artifact toolchain --from-task X8NXjYLVRzGWBXtZaJZPQQ:public/build/sccache.tar.zst
 1:43.38 The error occurred in code that was called by the mach command. This is either
 1:43.38 a bug in the called code itself or in the way that mach is calling it.
 1:43.38 You can invoke ``./mach busted`` to check if this issue is already on file. If it
 1:43.38 isn't, please use ``./mach busted file artifact`` to report it. If ``./mach busted`` is
 1:43.38 misbehaving, you can also inspect the dependencies of bug 1543241.
 1:43.38 If filing a bug, please include the full output of mach, including this error
 1:43.38 message.
 1:43.38 The details of the failure are as follows:
 1:43.38 PermissionError: [WinError 5] Access is denied: 'C:\\Users\\paden\\.mozbuild\\sccache\\sccache.exe'
 1:43.38   File "C:\Users\paden\mozilla-unified\python\mozbuild\mozbuild\artifact_commands.py", line 590, in artifact_toolchain
 1:43.38     unpack_file(local)
 1:43.39   File "c:\users\paden\mozilla-unified\python\mozbuild\mozbuild\action\tooltool.py", line 991, in unpack_file
 1:43.39     clean_path(base_file)
 1:43.39   File "c:\users\paden\mozilla-unified\python\mozbuild\mozbuild\action\tooltool.py", line 942, in clean_path
 1:43.39     shutil.rmtree(dirname)
 1:43.39   File "C:\mozilla-build\python3\Lib\shutil.py", line 759, in rmtree
 1:43.39     return _rmtree_unsafe(path, onerror)
 1:43.39            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 1:43.39   File "C:\mozilla-build\python3\Lib\shutil.py", line 622, in _rmtree_unsafe
 1:43.39     onerror(os.unlink, fullname, sys.exc_info())
 1:43.39   File "C:\mozilla-build\python3\Lib\shutil.py", line 620, in _rmtree_unsafe
 1:43.39     os.unlink(fullname)
 1:43.39 Sentry event ID: dacc8d4f3de34175bc52d179eff7637e
 1:43.48 Sentry is attempting to send 0 pending error messages
 1:43.48 Waiting up to 2 seconds
 1:43.48 Press Ctrl-Break to quit
 1:43.83 ERROR: Command '['C:\\Users\\paden\\.mozbuild\\srcdirs\\mozilla-unified-4141a8dbda9b\\_virtualenvs\\build\\Scripts\\python.exe', 'C:/Users/paden/mozilla-unified/mach', '--log-no-times', 'artifact', 'toolchain', '--from-task', 'X8NXjYLVRzGWBXtZaJZPQQ:public/build/sccache.tar.zst']' returned non-zero exit status 1.
 1:43.83 ERROR: If you can't fix the above, retry with --disable-bootstrap.
*** Fix above errors and then restart with "./mach build"

paden@padenot-win ~/mozilla-unified

Comment 2

[Alex Hochheiden [:ahochheiden]]

We could catch the access is denied error on Windows and add some text stating that it's likely a running process may be using the file?

The error as-is seems pretty straight forward to me, but maybe that additional hint would help.

Do you have any suggestion for contents of the extra 'hint'?

Comment 3

[Paul Adenot (:padenot)]

We might want to kill sccache and clangd, or ask to have them closed and retry, like is it usual to see on Windows.

Otherwise we can print a message that says this is expected and that isn't an error in mach, in contrast to what is said .

Comment 4

[Paul Adenot (:padenot)]

Attachment: Bug 1901564 - Kill sccache and clangd if an exception happen when getting new versions.