1298210 - Windows build bustage due to clobber failure on last_log.json

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Masatoshi Kimura [:emk]]

+++ This bug was initially created as a clone of Bug #1290201 +++

$ mach build
 0:04.92 c:\mozilla-build\mozmake\mozmake.EXE -f client.mk -s
 0:55.14 Automatically clobbering f:\m\mozilla-unified\obj-x86_64-pc-mingw32
 0:55.14 rmtree() failed for "('f:\\m\\mozilla-unified\\obj-x86_64-pc-mingw32',)". Reason: プロセスはファイルにアクセスできません。別のプロセスが使用中です。 (13). Retrying...
 0:55.14 rmtree() failed for "('f:\\m\\mozilla-unified\\obj-x86_64-pc-mingw32',)". Reason: プロセスはファイルにアクセスできません。別のプロセスが使用中です。 (13). Retrying...
 0:55.14 rmtree() failed for "('f:\\m\\mozilla-unified\\obj-x86_64-pc-mingw32',)". Reason: プロセスはファイルにアクセスできません。別のプロセスが使用中です。 (13). Retrying...
 0:55.14 rmtree() failed for "('f:\\m\\mozilla-unified\\obj-x86_64-pc-mingw32',)". Reason: プロセスはファイルにアクセスできません。別のプロセスが使用中です。 (13). Retrying...
 0:55.14 rmtree() failed for "('f:\\m\\mozilla-unified\\obj-x86_64-pc-mingw32',)". Reason: プロセスはファイルにアクセスできません。別のプロセスが使用中です。 (13). Retrying...
 0:55.14 Traceback (most recent call last):
 0:55.14   File "f:/m/mozilla-unified/config/pythonpath.py", line 56, in <module>
 0:55.14     main(sys.argv[1:])
 0:55.14   File "f:/m/mozilla-unified/config/pythonpath.py", line 48, in main
 0:55.14     execfile(script, frozenglobals)
 0:55.14   File "f:/m/mozilla-unified/python/mozbuild/mozbuild/controller/clobber.py", line 196, in <module>
 0:55.14     sys.exit(main(sys.argv[1:], os.environ, os.getcwd(), sys.stdout))
 0:55.14   File "f:/m/mozilla-unified/python/mozbuild/mozbuild/controller/clobber.py", line 184, in main
 0:55.14     required, performed, message = clobber.maybe_do_clobber(cwd, auto, fh)
 0:55.14   File "f:/m/mozilla-unified/python/mozbuild/mozbuild/controller/clobber.py", line 153, in maybe_do_clobber
 0:55.14     rmtree(self.topobjdir)
 0:55.14   File "f:\m\mozilla-unified\testing\mozbase\mozfile\mozfile\mozfile.py", line 137, in rmtree
 0:55.14     return remove(dir)
 0:55.14   File "f:\m\mozilla-unified\testing\mozbase\mozfile\mozfile\mozfile.py", line 234, in remove
 0:55.14     _call_with_windows_retry(shutil.rmtree, (path,))
 0:55.14   File "f:\m\mozilla-unified\testing\mozbase\mozfile\mozfile\mozfile.py", line 193, in _call_with_windows_retry
 0:55.14     _call_windows_retry(*args, **kwargs)
 0:55.15   File "f:\m\mozilla-unified\testing\mozbase\mozfile\mozfile\mozfile.py", line 151, in _call_windows_retry
 0:55.15     func(*args)
 0:55.15   File "c:\mozilla-build\python\lib\shutil.py", line 247, in rmtree
 0:55.15     rmtree(fullname, ignore_errors, onerror)
 0:55.15   File "c:\mozilla-build\python\lib\shutil.py", line 252, in rmtree
 0:55.15     onerror(os.remove, fullname, sys.exc_info())
 0:55.15   File "c:\mozilla-build\python\lib\shutil.py", line 250, in rmtree
 0:55.15     os.remove(fullname)
 0:55.15 WindowsError: [Error 32] プロセスはファイルにアクセスできません。別のプロセスが使用中です。: 'f:\\m\\mozilla-unified\\obj-x86_64-pc-mingw32\\.mozbuild\\last_log.json'
 0:55.15 client.mk:344: recipe for target 'f:/m/mozilla-unified/obj-x86_64-pc-mingw32/CLOBBER' failed
 0:55.15 mozmake.EXE: *** [f:/m/mozilla-unified/obj-x86_64-pc-mingw32/CLOBBER] Error 1
 0:55.19 235 compiler warnings present.
2

Comment 1

[David Anderson [:dvander] - inactive, e-mail if emergency]

Can confirm, I got this error again when I tried to update and build this morning.

Comment 2

[Xidorn Quan [:xidorn] UTC+11]

Hmmm, it seems autoclobber goes a completely different way than |mach clobber|.

Autoclobber works via makefile [1], which calls "clobber.py", rather than the Python function directly.

I actually think we should reopen the old bug rather than opening a new one, as the patch in the old bug doesn't have any effect at all (|mach clobber| itself wasn't affected by this issue from the beginning.)


[1] https://dxr.mozilla.org/mozilla-central/rev/7963ebdd52b93f96b812eff2eab8d94097147b9c/client.mk#343-345

Comment 3

[Xidorn Quan [:xidorn] UTC+11]

I think the solution is to move the remove_objdir code into "clobber.py", and make |mach clobber| invoke "clobber.py" as well.

Comment 4

[Masatoshi Kimura [:emk]]

(In reply to Xidorn Quan [:xidorn] (UTC+10) from comment #2)
> I actually think we should reopen the old bug rather than opening a new one,
> as the patch in the old bug doesn't have any effect at all (|mach clobber|
> itself wasn't affected by this issue from the beginning.)

Yes, I filed a new bug as requested in bug 1290201 comment #17 just because I would not like to waste time battling bureaucracy. If :gps eventually dupes this bug to bug 1290201 and reopens it, that's fine.

Comment 5

[Xidorn Quan [:xidorn] UTC+11]

I've been tired seeing this every day...

Comment 6

[Xidorn Quan [:xidorn] UTC+11]

Attachment: Bug 1298210 - Move remove objdir code into clobber.py so that autoclobber and clobber command share the same code.

Review commit: https://reviewboard.mozilla.org/r/74834/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/74834/

Comment 7

[Gregory Szorc [:gps]]

Comment on attachment 8785748 [details]
Bug 1298210 - Move remove objdir code into clobber.py so that autoclobber and clobber command share the same code.

https://reviewboard.mozilla.org/r/74834/#review72970

I trust you can fix the import issue in flight before landing.

::: python/mozbuild/mozbuild/mach_commands.py:38
(Diff revision 1)
>      MozbuildObject,
>      MozconfigFindException,
>      MozconfigLoadException,
>      ObjdirMismatchException,
>  )
> +from mozbuild.controller.clobber import Clobberer

This import should be deferred to the function that needs it. (We try to delay imports in mach_commands.py to avoid import bloat at process start time.)

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/7e2409da6e06
Move remove objdir code into clobber.py so that autoclobber and clobber command share the same code. r=gps

Comment 9

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/7e2409da6e06