579790 - OS X build error: dist/bin/components/interfaces.manifest.lck exists but stat() failed: Permission denied

Status: RESOLVED FIXED

(Firefox Build System :: General, defect, P2)

Description

[Daniel Holbert [:dholbert]]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1279488380.1279488480.13904.gz
OS X 10.5.2 mozilla-central build on 2010/07/18 14:26:20
s: bm-xserve06


{
/builds/slave/mozilla-central-macosx/build/obj-firefox/ppc/config/nsinstall -L /builds/slave/mozilla-central-macosx/build/obj-firefox/ppc/intl/locale/idl -m 644 _xpidlgen/locale.xpt ../../../dist/bin/components
../../../dist/bin/components/interfaces.manifest.lck exists but stat() failed: Permission denied
make[7]: *** [libs] Error 1
make[6]: *** [libs] Error 2
make[5]: *** [strres_libs] Error 2
}

Looks like a filesystem error perhaps?

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Can't find a lck file now so hard to debug. There's an earlier error on the same slave when building 89284a2b378d at 2010/07/18 07:34:45:
g++-4.2 -arch ppc -o SVGLength.o -c  -fvisibility=hidden ... 
.deps/SVGLength.pp /builds/slave/mozilla-central-macosx/build/content/svg/content/src/SVGLength.cpp
make[8]: *** [SVGLength.o] User defined signal 1
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1279463685.1279463922.32678.gz

I've set this slave to clobber its m-c opt build, but if this continues we might in the market for a disk check, and escalate to a reimage.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Meant to say that builds from this box are otherwise green, hence the clobber having a chance of being effective.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

The clobber was green.

Comment 4

[Daniel Holbert [:dholbert]]

This just happened on bm-xserve21:
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1281484250.1281484991.25373.gz

Comment 5

[Daniel Holbert [:dholbert]]

And again, now on bm-xserve07:
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1281833635.1281833805.10320.gz
OS X 10.5.2 mozilla-central build on 2010/08/14 17:53:55

Comment 6

[Phil Ringnalda (:philor)]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1282268739.1282271589.19249.gz
OS X 10.5.2 mozilla-central build on 2010/08/19 18:45:39  
s: bm-xserve17

adding: install.rdf../../../../dist/bin/chrome.manifest.lck exists but stat() failed: Permission denied

Comment 7

[Phil Ringnalda (:philor)]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1282298188.1282298301.3046.gz
OS X 10.5.2 mozilla-central build on 2010/08/20 02:56:28  
s: bm-xserve07

../../../dist/bin/chrome.manifest.lck exists but stat() failed: Permission denied

Comment 8

[Phil Ringnalda (:philor)]

http://tinderbox.mozilla.org/showlog.cgi?log=TraceMonkey/1282484053.1282484763.28819.gz
OS X 10.5.2 tracemonkey build on 2010/08/22 06:34:13
s: bm-xserve07

../../../dist/bin/chrome.manifest.lck exists but stat() failed: Permission denied

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

I think this is likely to be a race condition with parallel make on fast machines. We use -j4 on both xserves and minis, but xserves do their I/O much faster.

Comment 10

[(not currently active) Ted Mielczarek]

I thought I had wrung all the race condition bugs out of utils.lockFile, but this is a new one. The problem would be right here:
http://mxr.mozilla.org/mozilla-central/source/config/utils.py#92

I guess we can get EACCESS on Mac, too, so we should probably just change that block to be like:
except EnvironmentError, e:
  if e.errno == errno.ENOENT or e.errno == errno.EACCESS:
    continue

Comment 11

[Axel Hecht [:Pike]]

Sounds fair to me.

Comment 12

[Phil Ringnalda (:philor)]

http://tinderbox.mozilla.org/showlog.cgi?log=Jaegermonkey/1282872352.1282873067.3477.gz

Comment 13

[(not currently active) Ted Mielczarek]

Attachment: Handle EACCESS on all platforms

This is the fix I suggested. Kind of hard to test, because you'd have to hit the race condition, but I built successfully with -j4 on Mac and -j8 on Linux on multicore boxes.

Comment 14

[(not currently active) Ted Mielczarek]

Comment on attachment 469868 [details] [diff] [review]
Handle EACCESS on all platforms

This should fix a race condition that manifests as an occasional build error on OS X.

Comment 15

[Daniel Holbert [:dholbert]]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1282921760.1282924569.22090.gz

Comment 16

[Mike Beltzner [:beltzner, not reading bugmail]]

Comment on attachment 469868 [details] [diff] [review]
Handle EACCESS on all platforms

a=beltzner

Comment 17

[(not currently active) Ted Mielczarek]

Pushed to m-c:
http://hg.mozilla.org/mozilla-central/rev/d2800a98321e