859349 - "mach build" fails with "No such file or directory: .mozbuild/warnings.json" after a clobber fails

Status: RESOLVED FIXED

(Firefox Build System :: Mach Core, enhancement)

Description

[Matt Brubeck (:mbrubeck)]

This happened after clobber.py failed with an "Access is denied" error.  It got far enough to clobber the .mozbuild directory before it aborted, so save_to_warnings then failed too, making the real error harder to see:

mbrub_000@X1:~/src/elm$ mach build
 0:00.38 C:/mozilla-build/msys/bin/sh.exe -c c:/Users/mbrub_000/src/elm/build/pymake/make.py -f client.mk -s
 0:01.78 Adding client.mk options from c:/Users/mbrub_000/src/elm/.mozconfig:
 0:01.78     MOZ_OBJDIR=$(TOPSRCDIR)/obj-metro
 0:02.04 TEST-PASS | check-sync-dirs.py | c:\Users\mbrub_000\src\elm\js\src\build <= c:\Users\mbrub_000\src\elm\build
 0:02.05 Generating c:/Users/mbrub_000/src/elm/configure using autoconf
 0:02.84 TEST-PASS | check-sync-dirs.py | c:\Users\mbrub_000\src\elm\js\src\build <= c:\Users\mbrub_000\src\elm\build
 0:02.85 Generating c:/Users/mbrub_000/src/elm/js/src/configure using autoconf
 0:06.67 Automatically clobbering c:\Users\mbrub_000\src\elm\obj-metro
 0:06.67 Traceback (most recent call last):
 0:06.67   File "c:/Users/mbrub_000/src/elm/python/mozbuild/mozbuild/controller/clobber.py", line 193, in <module>
 0:06.67     sys.exit(main(sys.argv[1:], os.environ, os.getcwd(), sys.stdout))
 0:06.67   File "c:/Users/mbrub_000/src/elm/python/mozbuild/mozbuild/controller/clobber.py", line 183, in main
 0:06.67     required, performed, message = clobber.maybe_do_clobber(cwd, auto, fh)
 0:06.68   File "c:/Users/mbrub_000/src/elm/python/mozbuild/mozbuild/controller/clobber.py", line 152, in maybe_do_clobber
 0:06.68     rmtree(self.topobjdir)
 0:06.68   File "c:\mozilla-build\python\lib\shutil.py", line 244, in rmtree
 0:06.68     rmtree(fullname, ignore_errors, onerror)
 0:06.68   File "c:\mozilla-build\python\lib\shutil.py", line 244, in rmtree
 0:06.68     rmtree(fullname, ignore_errors, onerror)
 0:06.68   File "c:\mozilla-build\python\lib\shutil.py", line 244, in rmtree
 0:06.68     rmtree(fullname, ignore_errors, onerror)
 0:06.68   File "c:\mozilla-build\python\lib\shutil.py", line 249, in rmtree
 0:06.68     onerror(os.remove, fullname, sys.exc_info())
 0:06.68   File "c:\mozilla-build\python\lib\shutil.py", line 247, in rmtree
 0:06.68     os.remove(fullname)
 0:06.68 WindowsError: [Error 5] Access is denied: 'c:\\Users\\mbrub_000\\src\\elm\\obj-metro\\dist\\include\\dbm\\cdefs.h'
 0:06.68 c:\Users\mbrub_000\src\elm\client.mk:309:0: command 'python c:/Users/mbrub_000/src/elm/python/mozbuild/mozbuild/controller/clobber.py c:/Users/mbrub_000/src/elm c:/Users/mbrub_000/src/elm/obj-metro' failed, return code 1
 0:06.68 c:\Users\mbrub_000\src\elm\client.mk:339:0: command 'c:/mozillabuild/python/python.exe c:/Users/mbrub_000/src/elm/build/pymake/pymake/../make.py -f c:/Users/mbrub_000/src/elm/client.mk configure' failed, return code
 2 0:06.68 c:\Users\mbrub_000\src\elm\client.mk:163:0: command 'c:/mozilla-build/python/python.exe c:/Users/mbrub_000/src/elm/build/pymake/pymake/../make.py -f c:/Users/mbrub_000/src/elm/client.mk realbuild' failed, return code 2
 0:06.75 1962 compiler warnings present.
Error running mach:

    ['build']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.

You should consider filing a bug for this issue.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

IOError: [Errno 2] No such file or directory: u'c:\\Users\\mbrub_000\\src\\elm/obj-metro\\.mozbuild\\warnings.json'

  File "c:\Users\mbrub_000\src\elm\python/mozbuild/mozbuild/mach_commands.py", line 115, in build
    warnings_database.save_to_file(warnings_path)
  File "c:\Users\mbrub_000\src\elm\python/mozbuild\mozbuild\compilation\warnings.py", line 268, in save_to_file
    with open(filename, 'wb') as fh:

Comment 1

[Matt Brubeck (:mbrubeck)]

Attachment: patch

This patch contains two changes:
* warnings.save_to_file will create the parent directory if it doesn't exist.
* Don't bother writing the warnings file if the build failed.

Either change on its own would fix this bug, but each seemed worthwhile for other reasons, so I kept both.  It would be reasonable to accept just the second change, if you think the first one adds too much implicit behavior.

Comment 2

[Gregory Szorc [:gps]]

Comment on attachment 734664 [details] [diff] [review]
patch

Review of attachment 734664 [details] [diff] [review]:
-----------------------------------------------------------------

::: python/mozbuild/mozbuild/compilation/warnings.py
@@ +266,5 @@
>      def save_to_file(self, filename):
>          """Save the database to a file."""
> +        dirname = os.path.dirname(filename)
> +        if not os.path.exists(dirname):
> +            os.makedirs(dirname)

As I relearned recently, this isn't good enough. We need to do:

try:
    os.makedirs(path)
except OSError as e:
    if e.errno != errno.EEXIST:
        raise

::: python/mozbuild/mozbuild/mach_commands.py
@@ +112,5 @@
>                  '{count} compiler warnings present.')
>  
> +        if status == 0:
> +            warnings_database.prune()
> +            warnings_database.save_to_file(warnings_path)

We want to save the warnings database for failed builds. Otherwise warnings from before the failure point will be lost until the next recompile and we will under report total warnings.

Comment 3

[Matt Brubeck (:mbrubeck)]

Attachment: patch v2

Comment 4

[Matt Brubeck (:mbrubeck)]

Attachment: patch v3

Added a missing import.

Comment 5

[Matt Brubeck (:mbrubeck)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/3250333a5e6c

Comment 6

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/3250333a5e6c