859178 - errors doing automatic clobber not handled.

Status: NEW

(Firefox Build System :: General, defect)

Description

[Mark Hammond [:markh] [:mhammond]]

I just did 'mach build' and got:

 0:01.81 C:/mozilla-build/msys/bin/sh.exe -c o:/src/mm/mozilla-hg/mc-socialapi-landing/build/pymake/make.py -f client.mk -s
 0:03.45 Adding client.mk options from o:/src/mm/mozilla-hg/mc-socialapi-landing/.mozconfig-release:
...
 0:06.29 Automatically clobbering o:\src\mm\mozilla-hg\mc-socialapi-landing\obj-release
 0:06.30 Traceback (most recent call last):
...
 0:06.30   File "c:\mozilla-build\python\lib\shutil.py", line 247, in rmtree
 0:06.30     os.remove(fullname)
 0:06.30 WindowsError: [Error 5] Access is denied: 'o:\\src\\mm\\mozilla-hg\\mc-socialapi-landing\\obj-release\\dist\\include\\dbm\\cdefs.h'
...
etc - so the clobber failed as it could not remove one of the files in objdir.

So after fixing the problem, I tried again:

$ ./mach build
 0:00.30 C:/mozilla-build/msys/bin/sh.exe -c o:/src/mm/mozilla-hg/mc-socialapi-landing/build/pymake/make.py -f client.mk -s
 0:01.51 Adding client.mk options from o:/src/mm/mozilla-hg/mc-socialapi-landing/.mozconfig-release:
 0:01.51     MOZ_OBJDIR=$(TOPSRCDIR)/obj-release
 0:01.51     MOZ_MAKE_FLAGS=-j12
 0:02.05 Clobber not needed.
 
Note how mach no longer thinks a clobber is necessary, but the previous clobber didn't complete.

Comment 1

[Jeff Hammel]

Noting for mozbase (OT for the bug itself): we should curate this information of shutil.rmtree failure on windows, as the asserted advantage of mozfile.rmtree is that it purports/intends to correct this inadequacy.  BUT...we don't actually state what it is.  We should.  Then we should get it into python's stdlib as opportunity dictates

Comment 2

[Mark Hammond [:markh] [:mhammond]]

(In reply to Jeff Hammel [:jhammel] from comment #1)
> Noting for mozbase (OT for the bug itself): we should curate this
> information of shutil.rmtree failure on windows, as the asserted advantage
> of mozfile.rmtree is that it purports/intends to correct this inadequacy. 
> BUT...we don't actually state what it is.  We should.  Then we should get it
> into python's stdlib as opportunity dictates

I'm not 100% sure what you are saying, but the root problem here isn't a lack of permissions, just that the file was "in use" by something else, which is almost impossible to work around on Windows (other than obviously stopping that "something else" from using it).

The simplest option here might just be for mach to record the modified time of the top-level dir, then restore it should the rmtree fail - presumably that will cause the next 'build' to also attempt a clobber.

Comment 3

[Phil Ringnalda (:philor)]

Was it in use? This looks like a straight up dupe of bug 858752, where the NSS build system is getting the blame for creating its exports without write permission.

Comment 4

[Mark Hammond [:markh] [:mhammond]]

(In reply to Phil Ringnalda (:philor) from comment #3)
> Was it in use? This looks like a straight up dupe of bug 858752, where the
> NSS build system is getting the blame for creating its exports without write
> permission.

Nice spot.  So yeah, this report is a direct dupe of that :)  However, I have seen the same problem before when the file *was* in use (eg, another shell having the cwd inside the obj-dir, or when Visual Studio was open.)

Comment 5

[Jeff Hammel]

(In reply to Mark Hammond (:markh) from comment #2)
> (In reply to Jeff Hammel [:jhammel] from comment #1)
> > Noting for mozbase (OT for the bug itself): we should curate this
> > information of shutil.rmtree failure on windows, as the asserted advantage
> > of mozfile.rmtree is that it purports/intends to correct this inadequacy. 
> > BUT...we don't actually state what it is.  We should.  Then we should get it
> > into python's stdlib as opportunity dictates
> 
> I'm not 100% sure what you are saying, but the root problem here isn't a
> lack of permissions, just that the file was "in use" by something else,
> which is almost impossible to work around on Windows (other than obviously
> stopping that "something else" from using it).
> 
> The simplest option here might just be for mach to record the modified time
> of the top-level dir, then restore it should the rmtree fail - presumably
> that will cause the next 'build' to also attempt a clobber.

As said, OT for this bug.  Filed separately: https://bugzilla.mozilla.org/show_bug.cgi?id=859422