Closed Bug 579790 Opened 9 years ago Closed 9 years ago

OS X build error: dist/bin/components/interfaces.manifest.lck exists but stat() failed: Permission denied

Categories

(Firefox Build System :: General, defect, P2)

x86
macOS
defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: dholbert, Assigned: ted)

References

()

Details

(Whiteboard: [buildduty])

Attachments

(1 file)

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1279488380.1279488480.13904.gz
OS X 10.5.2 mozilla-central build on 2010/07/18 14:26:20
s: bm-xserve06


{
/builds/slave/mozilla-central-macosx/build/obj-firefox/ppc/config/nsinstall -L /builds/slave/mozilla-central-macosx/build/obj-firefox/ppc/intl/locale/idl -m 644 _xpidlgen/locale.xpt ../../../dist/bin/components
../../../dist/bin/components/interfaces.manifest.lck exists but stat() failed: Permission denied
make[7]: *** [libs] Error 1
make[6]: *** [libs] Error 2
make[5]: *** [strres_libs] Error 2
}

Looks like a filesystem error perhaps?
Summary: dist/bin/components/interfaces.manifest.lck exists but stat() failed: Permission denied → bm-xserve06 build error: dist/bin/components/interfaces.manifest.lck exists but stat() failed: Permission denied
Can't find a lck file now so hard to debug. There's an earlier error on the same slave when building 89284a2b378d at 2010/07/18 07:34:45:
g++-4.2 -arch ppc -o SVGLength.o -c  -fvisibility=hidden ... 
.deps/SVGLength.pp /builds/slave/mozilla-central-macosx/build/content/svg/content/src/SVGLength.cpp
make[8]: *** [SVGLength.o] User defined signal 1
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1279463685.1279463922.32678.gz

I've set this slave to clobber its m-c opt build, but if this continues we might in the market for a disk check, and escalate to a reimage.
Priority: -- → P2
Whiteboard: [buildduty][badslave?]
Meant to say that builds from this box are otherwise green, hence the clobber having a chance of being effective.
The clobber was green.
And again, now on bm-xserve07:
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1281833635.1281833805.10320.gz
OS X 10.5.2 mozilla-central build on 2010/08/14 17:53:55
Summary: bm-xserve06 build error: dist/bin/components/interfaces.manifest.lck exists but stat() failed: Permission denied → OS X build error: dist/bin/components/interfaces.manifest.lck exists but stat() failed: Permission denied
Whiteboard: [buildduty][badslave?] → [buildduty]
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1282268739.1282271589.19249.gz
OS X 10.5.2 mozilla-central build on 2010/08/19 18:45:39  
s: bm-xserve17

adding: install.rdf../../../../dist/bin/chrome.manifest.lck exists but stat() failed: Permission denied
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1282298188.1282298301.3046.gz
OS X 10.5.2 mozilla-central build on 2010/08/20 02:56:28  
s: bm-xserve07

../../../dist/bin/chrome.manifest.lck exists but stat() failed: Permission denied
http://tinderbox.mozilla.org/showlog.cgi?log=TraceMonkey/1282484053.1282484763.28819.gz
OS X 10.5.2 tracemonkey build on 2010/08/22 06:34:13
s: bm-xserve07

../../../dist/bin/chrome.manifest.lck exists but stat() failed: Permission denied
I think this is likely to be a race condition with parallel make on fast machines. We use -j4 on both xserves and minis, but xserves do their I/O much faster.
Component: Release Engineering → Build Config
Product: mozilla.org → Core
QA Contact: release → build-config
Version: other → Trunk
I thought I had wrung all the race condition bugs out of utils.lockFile, but this is a new one. The problem would be right here:
http://mxr.mozilla.org/mozilla-central/source/config/utils.py#92

I guess we can get EACCESS on Mac, too, so we should probably just change that block to be like:
except EnvironmentError, e:
  if e.errno == errno.ENOENT or e.errno == errno.EACCESS:
    continue
Sounds fair to me.
This is the fix I suggested. Kind of hard to test, because you'd have to hit the race condition, but I built successfully with -j4 on Mac and -j8 on Linux on multicore boxes.
Assignee: nobody → ted.mielczarek
Status: NEW → ASSIGNED
Attachment #469868 - Flags: review?(l10n)
Attachment #469868 - Flags: review?(l10n) → review+
Comment on attachment 469868 [details] [diff] [review]
Handle EACCESS on all platforms

This should fix a race condition that manifests as an occasional build error on OS X.
Attachment #469868 - Flags: approval2.0?
Comment on attachment 469868 [details] [diff] [review]
Handle EACCESS on all platforms

a=beltzner
Attachment #469868 - Flags: approval2.0? → approval2.0+
Pushed to m-c:
http://hg.mozilla.org/mozilla-central/rev/d2800a98321e
Status: ASSIGNED → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.