1408212 - generic worker permission errors downloading new toolchains

Status: RESOLVED INVALID

(Taskcluster :: Workers, defect)

Description

[Ralph Giles (:rillian)]

When I try to bump the rust toolchain versions, windows builds fail with a permission error downloading the new artifacts:

20:52:26     INFO -  Exception in thread Thread-1:
20:52:26     INFO -  Traceback (most recent call last):
20:52:26     INFO -    File "c:\mozilla-build\python\lib\threading.py", line 801, in __bootstrap_inner
20:52:26     INFO -      self.run()
20:52:26     INFO -    File "c:\mozilla-build\python\lib\threading.py", line 754, in run
20:52:26     INFO -      self.__target(*self.__args, **self.__kwargs)
20:52:26     INFO -    File "z:\build\build\src\third_party/python/dlmanager\dlmanager\manager.py", line 189, in _download
20:52:26     INFO -      finished_callback(self)
20:52:26     INFO -    File "z:\build\build\src\third_party/python/dlmanager\dlmanager\manager.py", line 323, in _download_finished
20:52:26     INFO -      self.persist_limit.remove_old_files()
20:52:26     INFO -    File "z:\build\build\src\python/mozbuild\mozbuild\artifacts.py", line 747, in remove_old_files
20:52:26     INFO -      fs.remove(f.path)
20:52:26     INFO -    File "z:\build\build\src\third_party/python/dlmanager\dlmanager\fs.py", line 94, in remove
20:52:26     INFO -      _update_permissions(path)
20:52:26     INFO -    File "z:\build\build\src\third_party/python/dlmanager\dlmanager\fs.py", line 87, in _update_permissions
20:52:26     INFO -      _call_with_windows_retry(os.chmod, (path, mode))
20:52:26     INFO -    File "z:\build\build\src\third_party/python/dlmanager\dlmanager\fs.py", line 64, in _call_with_windows_retry
20:52:26     INFO -      _call_windows_retry(*args, **kwargs)
20:52:26     INFO -    File "z:\build\build\src\third_party/python/dlmanager\dlmanager\fs.py", line 24, in _call_windows_retry
20:52:26     INFO -      func(*args)
20:52:26     INFO -  WindowsError: [Error 5] Access is denied: u'c:/builds/tooltool_cache\\4128a395f6908375-clang.tar.bz2'
20:52:26     INFO -   0:11.23 Downloaded artifact to c:\builds\tooltool_cache\31f8f99d8242c571-clang.tar.bz2

Comment 1

[Ralph Giles (:rillian)]

I've not been able to figure out what's going on here. Maybe something with '/' vs '\' in the target directory path, or multiple hashes colliding with the same unpack location?

Comment 2

[Ralph Giles (:rillian)]

See https://treeherder.mozilla.org/#/jobs?repo=try&revision=42a9e082a2ddc284beab0f265db9bd45e1061bc5 for detailed logs. The macOS build failure is bug 1381043.

Comment 3

[Mike Hommey [:glandium]]

(In reply to Ralph Giles (:rillian) | needinfo me from comment #0)
> When I try to bump the rust toolchain versions, windows builds fail with a
> permission error downloading the new artifacts:
> 20:52:26     INFO -  WindowsError: [Error 5] Access is denied:
> u'c:/builds/tooltool_cache\\4128a395f6908375-clang.tar.bz2'
> 20:52:26     INFO -   0:11.23 Downloaded artifact to
> c:\builds\tooltool_cache\31f8f99d8242c571-clang.tar.bz2

Those are a red herring. The real error is later in the log:

20:49:44     INFO -  DEBUG: Executing: `z:/build/build/src/sccache2/sccache.exe z:/build/build/src/clang/bin/clang-cl.exe -E 'c:\users\task_1507662921\appdata\local\temp\conftest.v1y4dz.c'`
20:49:44     INFO -  DEBUG: The command returned non-zero exit status 2.
20:49:44     INFO -  DEBUG: Its error output was:
20:49:44     INFO -  DEBUG: | error: failed to execute compile
20:49:44     INFO -  DEBUG: | caused by: Failed to send data to or receive data from server
20:49:44     INFO -  DEBUG: | caused by: Failed to read response header
20:49:44     INFO -  DEBUG: | caused by: An existing connection was forcibly closed by the remote host. (os error 10054)
20:49:44     INFO -  ERROR: Command `z:/build/build/src/sccache2/sccache.exe z:/build/build/src/clang/bin/clang-cl.exe -E 'c:\users\task_1507662921\appdata\local\temp\conftest.v1y4dz.c'` failed with exit status 2.
20:49:44     INFO -  *** Fix above errors and then restart with\
20:49:44     INFO -                 "z:/build/build/src/mozmake.EXE -f client.mk build"
20:49:44     INFO -  client.mk:384: recipe for target 'configure' failed
20:49:44     INFO -  mozmake.EXE: *** [configure] Error 1

That is, sccache built with rust 1.20 is busted.

Comment 4

[Ralph Giles (:rillian)]

Thanks! I saw that too, but normal builds don't fail with the permission denied error so I thought they were related.

Comment 5

[Pete Moore [:pmoore][:pete]]

I suspect there is/was a permissions issue with c:\builds\tooltool_cache

I'd recommend using a Writable Directory Cache for tooltool, rather than a system folder, search for "Writable Directory Cache" in https://docs.taskcluster.net/reference/workers/generic-worker/docs/payload

This will cause the worker to manage the ACLs of the cache, and guard it with a scope such as generic-worker:cache:tooltool or generic-worker:cache:rust/tooltool etc.

Ralph, can you try this out, or let me know if you are still seeing this issue?

Comment 6

[Chris Cooper [:coop] (he/him)]

Pete: can this be resolved, INCOMPLETE or otehrwise?

Comment 7

[Pete Moore [:pmoore][:pete]]

I think we can resolve as invalid, since a writable directory cache was not used, and this is the supported way to utilise caches.