1828771 - Cleanup after installing msvs from bootstrap fails - OSError: [WinError 145] The directory is not empty

Status: RESOLVED FIXED

(Firefox Build System :: Bootstrap Configuration, defect)

Description

[:Gijs (he/him)]

Traceback (most recent call last):
  File "path\to\srcdir\taskcluster\scripts\misc\get_vs.py", line 95, in <module>
    fh.write(yaml.dump(overlay))
  File "C:\mozilla-build\python3\lib\tempfile.py", line 830, in __exit__
    self.cleanup()
  File "C:\mozilla-build\python3\lib\tempfile.py", line 834, in cleanup
    self._rmtree(self.name)
  File "C:\mozilla-build\python3\lib\tempfile.py", line 816, in _rmtree
    _shutil.rmtree(name, onerror=onerror)
  File "C:\mozilla-build\python3\lib\shutil.py", line 749, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\mozilla-build\python3\lib\shutil.py", line 622, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\mozilla-build\python3\lib\shutil.py", line 622, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  File "C:\mozilla-build\python3\lib\shutil.py", line 622, in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
  [Previous line repeated 5 more times]
  File "C:\mozilla-build\python3\lib\shutil.py", line 631, in _rmtree_unsafe
    onerror(os.rmdir, path, sys.exc_info())
  File "C:\mozilla-build\python3\lib\shutil.py", line 629, in _rmtree_unsafe
    os.rmdir(path)
OSError: [WinError 145] The directory is not empty: '.\\get_vse4ekvtyh\\unpack\\Windows Kits\\10\\References\\10.0.19041.0\\Windows.ApplicationModel.CommunicationBlocking.CommunicationBlockingContract\\2.0.0.0\\zh-hans'

Comment 1

[:Gijs (he/him)]

Re-running mach bootstrap after this re-downloads all the things. :-\

Comment 2

[:Gijs (he/him)]

(In reply to :Gijs (he/him) from comment #1)

Re-running mach bootstrap after this re-downloads all the things. :-\

... and then fails in exactly the same way.

Edit: in case it's helpful, the output immediately preceding this is:

Adding DIA SDK\Samples\DIA2Dump\Callback.h
Adding DIA SDK\Samples\DIA2Dump\DIA2Dump.cpp
Adding DIA SDK\Samples\DIA2Dump\DIA2Dump.h
Adding DIA SDK\Samples\DIA2Dump\DIA2Dump.sln
Adding DIA SDK\Samples\DIA2Dump\DIA2Dump.vcxproj
Adding DIA SDK\Samples\DIA2Dump\DIA2Dump.vcxproj.filters
Adding DIA SDK\Samples\DIA2Dump\makefile
Adding DIA SDK\Samples\DIA2Dump\PrintSymbol.cpp
Adding DIA SDK\Samples\DIA2Dump\PrintSymbol.h
Adding DIA SDK\Samples\DIA2Dump\Readme.txt
Adding DIA SDK\Samples\DIA2Dump\regs.cpp
Adding DIA SDK\Samples\DIA2Dump\regs.h
Adding DIA SDK\Samples\DIA2Dump\stdafx.cpp
Adding DIA SDK\Samples\DIA2Dump\stdafx.h

Comment 3

[Mike Hommey [:glandium]]

OSError: [WinError 145] The directory is not empty: '.\\get_vse4ekvtyh\\unpack\\Windows Kits\\10\\References\\10.0.19041.0\\Windows.ApplicationModel.CommunicationBlocking.CommunicationBlockingContract\\2.0.0.0\\zh-hans'

Is there anything left in that directory?

Comment 4

[:Gijs (he/him)]

(In reply to Mike Hommey [:glandium] from comment #3)

OSError: [WinError 145] The directory is not empty: '.\\get_vse4ekvtyh\\unpack\\Windows Kits\\10\\References\\10.0.19041.0\\Windows.ApplicationModel.CommunicationBlocking.CommunicationBlockingContract\\2.0.0.0\\zh-hans'

Is there anything left in that directory?

Yes, Windows.ApplicationModel.CommunicationBlocking.CommunicationBlockingContract.xml

Comment 5

[Mike Hommey [:glandium]]

And there's no error message about failing to remove that file?

Comment 6

[:Gijs (he/him)]

(In reply to Mike Hommey [:glandium] from comment #5)

And there's no error message about failing to remove that file?

Not that I can see. I also don't see any indication (before this error) that there's any removing of files in progress at all. The first I see is the error in comment 0, immediately after the log in comment 2.

Comment 7

[Mike Hommey [:glandium]]

The irony here is that it fails right at the end when it has extracted everything you need, but also right before what is needed to avoid re-bootstrapping is recorded. So you can unblock yourself by doing echo 9d2fecd973a75e92dd733a728c66edd1c1d03afe2b0930a8be70989955462697 > ~/.mozbuild/indices/vs. That doesn't say anything about why this is happening though... Like... why in the hell would windows fail to remove the file?

Comment 8

[Mike Hommey [:glandium]]

I suppose this is not a VM?

Comment 9

[:Gijs (he/him)]

(In reply to Mike Hommey [:glandium] from comment #8)

I suppose this is not a VM?

Nope, regular Win10 on one of the threadripper machines we got set up for a number of Mozillians a while back (I think maybe 2 years now but I'm not sure.)

Comment 10

[:Gijs (he/him)]

(In reply to Mike Hommey [:glandium] from comment #7)

Like... why in the hell would windows fail to remove the file?

I feel like I've heard this story before but I can't remember exactly where from. Maybe Windows Defender or something else holding on to the file?

Comment 11

[:Gijs (he/him)]

The comment in https://searchfox.org/mozilla-central/rev/31f5847a4494b3646edabbdd7ea39cb88509afe2/testing/mozbase/mozfile/mozfile/mozfile.py#196 seems potentially relevant.

Comment 12

[:Gijs (he/him)]

> `C:\\Users\\Gijs\\.mozbuild\\get_vse4ekvtyh\\unpack\\Windows Kits\\10\\References\\10.0.19041.0\\Windows.ApplicationModel.CommunicationBlocking.CommunicationBlockingContract\\2.0.0.0\\zh-hans\\Windows.ApplicationModel.CommunicationBlocking.CommunicationBlockingContract.xml`.length
> 259 

I bet we hit max_path - the code in mozfile works around this here

Comment 13

[Mike Hommey [:glandium]]

Attachment: Bug 1828771 - Exclude MSI files with the longest extracted paths.

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e674b2f9d147
Exclude MSI files with the longest extracted paths. r=firefox-build-system-reviewers,andi

Comment 17

[BugBot [:suhaib / :marco/ :calixte]]

The following field has been copied from a duplicate bug:

Field	Value	Source
Regressed by	bug 1123821	bug 1828877

For more information, please visit auto_nag documentation.

Comment 18

[:Gijs (he/him)]

Is it not possible to run msiexec with the hackaround from https://searchfox.org/mozilla-central/rev/31f5847a4494b3646edabbdd7ea39cb88509afe2/testing/mozbase/mozfile/mozfile/mozfile.py#284-290:

    if (
        sys.platform in ("win32", "cygwin")
        and len(path) >= 3
        and path[1] == ":"
        and path[2] == "\\"
    ):
        path = "\\\\?\\%s" % path

to avoid the max_path limitation?

As it is, the bootstrap code will break if/when there's an msi package that we do need that has long paths/filenames...

Comment 19

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1123821

Comment 20

[Mike Hommey [:glandium]]

I tried plenty of things, including UNC paths and virtual drives, and they don't work with msiexec. As mentioned in the commit message, the longest path is now 126 characters, down from 213. We have room. Eventually, we'll switch to not use msiexec, though.

Comment 22

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/e674b2f9d147