925236 - Race condition with buildlist.py

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Mike Hommey [:glandium]]

This is a build failure i got on the birch branch with pseudo derecurse enabled:
https://tbpl.mozilla.org/php/getParsedLog.php?id=28904665&tree=Birch

Looking at e.g. httpd.manifest that is only on one hand and not the other, the log does contain both buildlist.py commands for it, so somehow, this must be overwritten by another buildlist.py command...

My guess is that comes from some of the parallel dirs during the libs tier. It probably didn't happen before because libs was doing so much. With pseudo derecurse, libs doesn't do compilation, so there's more chances of existing race conditions being triggered.

Comment 1

[Mike Hommey [:glandium]]

Attachment: Don't parallelize the libs tier because of the race conditions it shows now it doesn't compile

I think this is the safest thing to do considering we don't know what kind of other races there can be. The more we move stuff to moz.build and out of the libs tier, the better it will get anyways.

Comment 2

[Mike Hommey [:glandium]]

Different error on the same changeset in https://tbpl.mozilla.org/php/getParsedLog.php?id=28915264&tree=Birch :
Error: /builds/slave/bir-osx64-00000000000000000000/build/obj-firefox/i386/browser/installer/package-manifest:413: File missing in /builds/slave/bir-osx64-00000000000000000000/build/obj-firefox/i386/../x86_64/dist: FirefoxNightly.app/Contents/MacOS/chrome/toolkit/content/global/url-classifier/unittests.xul

This might require to do the same to the tools tier.

Comment 3

[Mike Hommey [:glandium]]

Apparently, the patch fixes problems on android builds too:
red:
https://tbpl.mozilla.org/php/getParsedLog.php?id=28904953&tree=Birch
green:
https://tbpl.mozilla.org/php/getParsedLog.php?id=28918797&tree=Birch

Comment 4

[Mike Hommey [:glandium]]

But it doesn't entirely fix osx :(

Comment 5

[Mike Hommey [:glandium]]

(In reply to Mike Hommey [:glandium] from comment #4)
> But it doesn't entirely fix osx :(

And that's probably because httpd.manifest is done in tools.

Comment 6

[Mike Hommey [:glandium]]

(In reply to Mike Hommey [:glandium] from comment #5)
> (In reply to Mike Hommey [:glandium] from comment #4)
> > But it doesn't entirely fix osx :(
> 
> And that's probably because httpd.manifest is done in tools.

FTR, https://tbpl.mozilla.org/php/getParsedLog.php?id=28919823&tree=Birch

Comment 7

[Mike Hommey [:glandium]]

Attachment: Don't parallelize the libs tier because of the race conditions it shows now it doesn't compile

https://tbpl.mozilla.org/?tree=Birch&rev=08da2afdaeeb

Comment 8

[Mike Hommey [:glandium]]

(In reply to Mike Hommey [:glandium] from comment #7)
> https://tbpl.mozilla.org/?tree=Birch&rev=08da2afdaeeb

The osx tests redness is due to our setup not handling multiple rebuilds properly.

Comment 9

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/adcb20866454