Closed Bug 1293784 Opened 3 years ago Closed 3 years ago

No such file or directory: './xpidllex.py' (or './xpidlyacc.py') in header.py during incremental builds

Categories

(Firefox Build System :: General, defect)

defect
Not set

Tracking

(firefox51 fixed)

RESOLVED FIXED
mozilla51
Tracking Status
firefox51 --- fixed

People

(Reporter: mshal, Assigned: mshal)

Details

Attachments

(1 file)

We end up recursing into xpcom/idl-parser/xpidl twice, once from moz.build and once from xpcom/xpidl/Makefile.in:

1) https://dxr.mozilla.org/mozilla-central/rev/720b5d2c84d5b253d4dfde4897e13384dc97a46a/xpcom/moz.build#11
2) https://dxr.mozilla.org/mozilla-central/rev/720b5d2c84d5b253d4dfde4897e13384dc97a46a/xpcom/xpidl/Makefile.in#6

During a clobber build this happens to work fine - either we only get one invocation, or the second doesn't conflict with the first because there are no outputs to remove. And since we generally don't touch header.py, incremental builds work fine since both invocations of make see that xpidllex.py is up-to-date and don't re-run header.py.

However, if header.py is touched (as in bug 1293387), and we're doing an incremental build, there is an xpidllex.py and xpidlyacc.py left over from the previous build. In this case, it is possible that header.py is invoked twice simultaneously, and both see that xpidllex.py & xpidlyacc.py exist (along with .pyc), and try to remove them:

https://dxr.mozilla.org/mozilla-central/rev/720b5d2c84d5b253d4dfde4897e13384dc97a46a/xpcom/idl-parser/xpidl/header.py#537

The first one to remove succeeds, and the second one fails. This race condition is pretty difficult to hit, but seems to happen a lot on OSX for some reason. It is much easier to reproduce by adding a sleep:

+    print("Running header.py")
     for fileglobs in [os.path.join(cachedir, f) for f in ["xpidllex.py*", "xpidlyacc.py*"]]:
         for filename in glob.glob(fileglobs):
+            print("Sleeping....")
+            time.sleep(5)
+            print("Done sleeping...")
             os.remove(filename)

STR:
$ ./mach configure
$ ./mach build export
$ (make a meaningless change to header.py)
$ ./mach build export
# This fails with the 'No such file or directory' message

I'm currently testing out a patch.
Comment on attachment 8779536 [details]
Bug 1293784 - Only recurse into xpcom/idl-parser/xpidl once;

https://reviewboard.mozilla.org/r/70508/#review67844

I think I've written a similar patch myself!

This patch should fix things. One downside is we can no longer evaluate the `export` target in xpcom/xpidl. But I don't think we care.
Attachment #8779536 - Flags: review?(gps) → review+
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/31c4f11b0731
Only recurse into xpcom/idl-parser/xpidl once; r=gps
https://hg.mozilla.org/mozilla-central/rev/31c4f11b0731
Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla51
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.