1787993 - mach bootstrap failing with git-cinnabar "PermissionError: [WinError 5] Access is denied:"

Status: RESOLVED FIXED

(Firefox Build System :: Bootstrap Configuration, defect, P3)

Description

[Mark Straver]

As a follow-up to bug 1386368 and bug 1787370, the following issue popped up after 6 hours of processing data:

mcwer@Win11TP-VM /c/source
$ python3 bootstrap.py --vcs=git
Cloning into mozilla-unified using Git...
Destination directory for clone (leave empty to use default destination of mozilla-unified):
Attempting to add exclusion path to Microsoft Defender Antivirus for: C:\source\mozilla-unified
Attempting to add exclusion path to Microsoft Defender Antivirus for: C:\mozilla-build\
Attempting to add exclusion path to Microsoft Defender Antivirus for: C:\Users\mcwer\.mozbuild
Cloning Firefox Git repository to mozilla-unified
Cloning into 'C:\Users\mcwer\AppData\Local\Temp\tmpvbyg0wsl\git-cinnabar-master'...
remote: Enumerating objects: 106, done.
remote: Counting objects: 100% (106/106), done.
remote: Compressing objects: 100% (100/100), done.
Receiving objects:  96% (102/106)ed 22 (delta 1), pack-reused 0Receiving objects:  95% (101/106)
Receiving objects: 100% (106/106), 198.23 KiB | 4.50 MiB/s, done.
Resolving deltas: 100% (2/2), done.
Downloading from https://github.com/glandium/git-cinnabar/releases/download/0.5.10/git-cinnabar.windows.x86_64.zip...
 100%
Extracting git-cinnabar-helper.exe...
 100%
WARNING! git-cinnabar is required for Firefox development  with git. After the clone is complete, the bootstrapper will ask if you would like to configure git; answer yes, and be sure to add git-cinnabar to your PATH according to the bootstrapper output.
Cloning into 'mozilla-unified'...
Fetching cinnabar metadata from https://community-tc.services.mozilla.com/api/index/v1/task/project.git-cinnabar.bundle.mozilla-unified/artifacts/public/bundle.git
ERROR [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)
WARNING Falling back to normal clone.
Getting clone bundle from https://hg.cdn.mozilla.net/mozilla-unified/6105060ec9613901bbad5c2406d24e33e111bc54.zstd-max.hg
Reading 702044 changesets
Reading and importing 700952 manifests
Reading and importing 4423227 revisions of 673115 files
WARNING [bundle2] ignoring bundle2 part: b'hgtagsfnodes'
WARNING [bundle2] ignoring bundle2 part: b'cache:rev-branch-cache'
Importing 702044 changesets
Reading 85 changesets
Reading and importing 85 manifests
Reading and importing 300 revisions of 169 files
Importing 85 changesets
Checking 253824 imported file root and head revisions
It is recommended that you set "remote.origin.prune" or "fetch.prune" to "true".
  git config remote.origin.prune true
or
  git config fetch.prune true

Run the following command to update tags:
  git fetch --tags hg::tags: tag "*"
Updating files: 100% (316475/316475), done.
Configuring watchman
Copying mozilla-unified\.git\hooks\fsmonitor-watchman.sample to mozilla-unified\.git\hooks\query-watchman
Failed to install git-cinnabar. Try performing a manual installation: https://github.com/glandium/git-cinnabar/wiki/Mozilla:-A-git-workflow-for-Gecko-development
Could not bootstrap Firefox! Consider filing a bug.
Traceback (most recent call last):
  File "C:\source\bootstrap.py", line 420, in <module>
    sys.exit(main(sys.argv))
  File "C:\source\bootstrap.py", line 387, in main
    srcdir = clone(options)
  File "C:\source\bootstrap.py", line 333, in clone
    return git_clone_firefox(binary, dest, watchman)
  File "C:\source\bootstrap.py", line 243, in git_clone_firefox
    shutil.rmtree(str(tempdir))
  File "C:\mozilla-build\python3\lib\shutil.py", line 749, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\mozilla-build\python3\lib\shutil.py", line 622, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\mozilla-build\python3\lib\shutil.py", line 622, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\mozilla-build\python3\lib\shutil.py", line 622, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  [Previous line repeated 1 more time]
  File "C:\mozilla-build\python3\lib\shutil.py", line 627, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\mozilla-build\python3\lib\shutil.py", line 625, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 5] Access is denied: 'C:\\Users\\mcwer\\AppData\\Local\\Temp\\tmpvbyg0wsl\\git-cinnabar-master\\.git\\objects\\pack\\pack-06fbba4134765c0f8b4753d94c9d3ea137e9fc48.idx'

mcwer@Win11TP-VM /c/source
$

Environment is a fresh install of Windows 11 Pro, following the instructions outlined on the sourcedoc website.

• Windows 11 Pro
• Visual Studio 2022 CE
• Windows 11 SDK
• Mozillabuild-latest
• Bootstrap.py modified as per :glandium's instructions to make cloning work https://bugzilla.mozilla.org/show_bug.cgi?id=1386368#c7

Comment 1

[Mike Hommey [:glandium]]

So, there are two different problems in there:

• The removal of the temporary git-cinnabar clone fails on Windows, probably because of the permissions that git sets to pack files.
• If git-cinnabar was not found, an error is printed that installing git-cinnabar failed, even when it hasn't failed.

The good news for you, though, is that you do do have a clone now, and you can run ./mach bootstrap from there.

Comment 2

[Mike Hommey [:glandium]]

Alex, do you want to look into comment 1?

Comment 3

[Alex Hochheiden [:ahochheiden] [he/him]]

Sure, I'll take a look.

Comment 5

[Mathew Hodson]

I also see CERTIFICATE_VERIFY_FAILED in the log, which is bug 1386368.

Comment 6

[Mike Hommey [:glandium]]

Attachment: Bug 1787993 - Update how we install git-cinnabar.

The download.py script can now run without a clone, which solves e.g.
problems stemming from the temporary git repository on windows.
It also now pulls from github by default, which doesn't suffer from
certificate validation issues like the taskcluster artifacts urls do.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7222fa484c0a
Update how we install git-cinnabar. r=firefox-build-system-reviewers,andi

Comment 8

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/7222fa484c0a