Closed Bug 557413 Opened 10 years ago Closed 10 years ago

ar: ctypes/libffi/.libs/libffi.a: Resource temporarily unavailable

Categories

(Firefox Build System :: General, defect, major)

x86
macOS
defect
Not set
major

Tracking

(Not tracked)

RESOLVED FIXED
mozilla1.9.3a5

People

(Reporter: jrmuizel, Assigned: dwitte)

References

Details

(Keywords: intermittent-failure)

Attachments

(3 files, 2 obsolete files)

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270514732.1270516404.7315.gz

ar: ctypes/libffi/.libs/libffi.a: Resource temporarily unavailable
Making symlinks to the original object files in the archive libraries ctypes/libffi/.libs/libffi.a
ar: ctypes/libffi/.libs/libffi.a: Resource temporarily unavailable
g++-4.2 -arch i386  -fno-rtti -fno-exceptions -Wall -Wpointer-arith -Woverloaded-virtual -Wsynth -Wno-ctor-dtor-privacy -Wno-non-virtual-dtor -Wcast-align -Wno-invalid-offsetof -Wno-variadic-macros -Wno-long-long -gdwarf-2 -isysroot /Developer/SDKs/MacOSX10.5.sdk -fno-strict-aliasing -fpascal-strings -fno-common -pthread  -DNDEBUG -DTRIMMED -gdwarf-2 -O3 -fstrict-aliasing  -fPIC  -o libmozjs.dylib  jsapi.o jsarena.o jsarray.o jsatom.o jsbool.o jscntxt.o jsdate.o jsdbgapi.o jsdhash.o jsdtoa.o jsemit.o jsexn.o jsfun.o jsgc.o jshash.o jsinterp.o jsinvoke.o jsiter.o jslock.o jslog2.o jsmath.o jsnum.o jsobj.o json.o jsopcode.o jsparse.o jsprf.o jspropertycache.o jspropertytree.o jsregexp.o jsscan.o jsscope.o jsscript.o jsstr.o jstask.o jstypedarray.o jsutil.o jsxdrapi.o jsxml.o prmjtime.o jstracer.o Assembler.o Allocator.o CodeAlloc.o Containers.o Fragmento.o LIR.o njconfig.o RegAlloc.o avmplus.o Nativei386.o jsbuiltins.o VMPI.o ctypes/CTypes.o ctypes/Library.o   debug.o prep_cif.o types.o raw_api.o java_raw_api.o closures.o ffi.o darwin.o ffi64.o darwin64.o  -lobjc -framework Cocoa  -framework ExceptionHandling -Wl,-executable_path,/bin -Wl,-dead_strip -L/builds/moz2_slave/mozilla-central-macosx/build/obj-firefox/i386/dist/lib -lplds4 -lplc4 -lnspr4 -dynamiclib -install_name @executable_path/libmozjs.dylib -compatibility_version 1 -current_version 1 -single_module  -lm  
i686-apple-darwin9-g++-4.2.1: ffi64.o: No such file or directory
i686-apple-darwin9-g++-4.2.1: darwin64.o: No such file or directory
Whiteboard: [orange]
Blocks: 438871
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270560772.1270562293.23511.gz
OS X 10.5.2 mozilla-central build on 2010/04/06 06:32:52
s: bm-xserve19

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270562185.1270563868.28071.gz
OS X 10.5.2 mozilla-central build on 2010/04/06 06:56:25
s: bm-xserve18
Severity: normal → major
I think either we need to figure out a solution to this ASAP or back out that change.
Assignee: nobody → dwitte
I'll take a look when I get in today...
Which is the bug here? "Resource temporarily unavailable" or "No such file or directory"?

Earlier is ranlib: file: .libs/libffi.a(ffi64.o) has no symbols is expected behavior. Are we failing to unpack the object because it doesn't have any symbols?

BTW, does this always happen on the second pass of the universal build? If so, perhaps libffi is polluting the srcdir somehow?
(In reply to comment #4)
> Which is the bug here? "Resource temporarily unavailable" or "No such file or
> directory"?

Both, I think... I'm guessing that the former means that the archive is already open and being written to by the other instance of 'ar', per my race theory. Not sure about the latter... I'd think that one of the ar's succeeds, but maybe the link step itself is racing with 'ar'... (unlikely!?)

> Earlier is ranlib: file: .libs/libffi.a(ffi64.o) has no symbols is expected
> behavior. Are we failing to unpack the object because it doesn't have any
> symbols?

I don't think so; as you say it's expected and this has always been the case when building libffi. The other archive members should definitely have symbols...

> BTW, does this always happen on the second pass of the universal build? If so,
> perhaps libffi is polluting the srcdir somehow?

Looking...
From the full build log (http://tinderbox.mozilla.org/showlog.cgi?tree=Firefox&errorparser=unix&logfile=1270514732.1270516404.7315.gz&buildtime=1270514732&buildname=OS%20X%2010.5.2%20mozilla-central%20build&fulltext=1):

/bin/sh ./libtool --tag=CC   --mode=link gcc-4.2 -arch ppc -Wall -g -fexceptions -Wall -W -Wno-unused -Wpointer-arith -Wcast-align -W -Wno-long-long -gdwarf-2 -isysroot /Developer/SDKs/MacOSX10.5.sdk -fno-strict-aliasing -fpascal-strings -fno-common -pthread -version-info `grep -v '^#' /builds/moz2_slave/mozilla-central-macosx/build/js/src/ctypes/libffi/libtool-version`    -o libffi.la -rpath /usr/local/lib src/debug.lo src/prep_cif.lo src/types.lo src/raw_api.lo src/java_raw_api.lo src/closures.lo              src/powerpc/ffi_darwin.lo src/powerpc/darwin.lo src/powerpc/darwin_closure.lo             
/bin/sh ./libtool --tag=CC   --mode=link gcc-4.2 -arch ppc -Wall -g -fexceptions -Wall -W -Wno-unused -Wpointer-arith -Wcast-align -W -Wno-long-long -gdwarf-2 -isysroot /Developer/SDKs/MacOSX10.5.sdk -fno-strict-aliasing -fpascal-strings -fno-common -pthread   -o libffi_convenience.la  src/debug.lo src/prep_cif.lo src/types.lo src/raw_api.lo src/java_raw_api.lo src/closures.lo              src/powerpc/ffi_darwin.lo src/powerpc/darwin.lo src/powerpc/darwin_closure.lo             
libtool: link: ar cru .libs/libffi_convenience.a  src/debug.o src/prep_cif.o src/types.o src/raw_api.o src/java_raw_api.o src/closures.o src/powerpc/ffi_darwin.o src/powerpc/darwin.o src/powerpc/darwin_closure.o
libtool: link: ar cru .libs/libffi.a  src/debug.o src/prep_cif.o src/types.o src/raw_api.o src/java_raw_api.o src/closures.o src/powerpc/ffi_darwin.o src/powerpc/darwin.o src/powerpc/darwin_closure.o
ranlib: ranlib: file: .libs/libffi.a(raw_api.o) file: .libs/libffi_convenience.a(raw_api.o) has no symbolshas no symbols

ranlib: ranlib: file: .libs/libffi.a(java_raw_api.o) file: .libs/libffi_convenience.a(java_raw_api.o) has no symbolshas no symbols

libtool: link: ranlib .libs/libffi.a
libtool: link: ranlib .libs/libffi_convenience.a
ranlib: ranlib: file: .libs/libffi.a(raw_api.o) file: .libs/libffi_convenience.a(raw_api.o) has no symbolshas no symbols

ranlib: ranlib: file: .libs/libffi.a(java_raw_api.o) file: .libs/libffi_convenience.a(java_raw_api.o) has no symbolshas no symbols

libtool: link: ( cd ".libs" && rm -f "libffi_convenience.la" && ln -s "../libffi_convenience.la" "libffi_convenience.la" )
libtool: link: ( cd ".libs" && rm -f "libffi.la" && ln -s "../libffi.la" "libffi.la" )

I think my race theory is off. The source files in libffi are getting built once, so the js/src makefile's 'export' rule certainly isn't racing. But the link steps in libffi seem to be getting issued twice and execute simultaneously, as clearly evidenced by the garbled output from ranlib toward the end.

Does the definition of SUBMAKE include -j flags? I'm looking at rules.mk now...
For reference, this is the rule that builds libffi:
http://mxr.mozilla.org/mozilla-central/source/js/src/Makefile.in#372
It looks to me as if it's linking the .la and the .a at the same time, which is what I would expect in a parallel build and should be fine unless there are conflicting intermediate files that both link steps create.
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270583838.1270583916.23343.gz
OS X 10.5.2 mozilla-central build on 2010/04/06 12:57:18
s: bm-xserve17
Attached patch possible fix (obsolete) — Splinter Review
This drops the $SUBMAKE bit in favor of an explicit call to $MAKE without any $MAKEFLAGS.

If the problem is building libffi parallel, this should fix it. If not, it'll give us another datapoint...
Attachment #437388 - Flags: review?(ted.mielczarek)
I think you'd be better off just passing -j1 explicitly, not completely removing MAKEFLAGS. You'll be dropping -k or -s flags which people care about.
OK, this one just adds '-j1' -- according to the make manual, the last '-j' takes effect, so this should have the desired effect. Assuming pymake also does so. :)

Otherwise, I can write a sed substitution to pull out '-j' from MAKEFLAGS.
Attachment #437388 - Attachment is obsolete: true
Attachment #437416 - Flags: review?(benjamin)
Attachment #437388 - Flags: review?(ted.mielczarek)
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270623748.1270625205.14554.gz
OS X 10.5.2 mozilla-central build on 2010/04/07 00:02:28
s: bm-xserve19
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270664894.1270664972.21458.gz
OS X 10.5.2 mozilla-central build on 2010/04/07 11:28:14
s: bm-xserve19
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270663780.1270666559.26408.gz
OS X 10.5.2 mozilla-central build on 2010/04/07 11:09:40
s: bm-xserve09
Attachment #437416 - Flags: review?(benjamin) → review+
Landed on m-c. Will let this sync to TM by natural processes.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Flags: in-testsuite-
Target Milestone: --- → mozilla1.9.3a5
This also helps me out, for an otherwise unrelated parallelism reason ;-)
http://tinderbox.mozilla.org/showlog.cgi?tree=Firefox&errorparser=unix&logfile=1270675968.1270676055.22988.gz&buildtime=1270675968&buildname=OS%20X%2010.5.2%20mozilla-central%20build&fulltext=1

Still happening. I'm gonna rejigger dependencies and see if that fixes it...
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
I *think* this does it. I've tested it in all the ways I can think of, and it doesn't break anything. And it will probably fix the problem.

This makes the dependencies very explicit, so that the 'libs' target depends on 'libffi.a', 'libffi.a' depends on 'build_libffi', and 'build_libffi' issues the 'make' command. All three of these are required to get make to do the right thing.
Attachment #437742 - Flags: review?(benjamin)
Attachment #437742 - Flags: review?(benjamin) → review-
Comment on attachment 437742 [details] [diff] [review]
really actually seriously fix deps

>diff --git a/js/src/Makefile.in b/js/src/Makefile.in

>-export::
>+# Specify a 'libs' target dependency on libffi, to make sure it gets built
>+# before libraries are linked. Also specify -j1 so we don't get a parallel
>+# build.
>+libs:: ctypes/libffi/.libs/libffi.$(LIB_SUFFIX)

There is nothing here which prevents us building $(LIBRARY) or $(SHARED_LIBRARY) before this. I don't believe this can work correctly. I think you want:

$(LIBRARY) $(SHARED_LIBRARY): build_libffi
	$(MAKE) -j1 -C ctypes/libffi

But I still don't understand why the export:: thing didn't work. We make pretty strong guarantees that the export:: phase runs before we get anywhere near the libs:: phase.
(In reply to comment #21)
> There is nothing here which prevents us building $(LIBRARY) or
> $(SHARED_LIBRARY) before this. I don't believe this can work correctly. I think
> you want:
> 
> $(LIBRARY) $(SHARED_LIBRARY): build_libffi
>     $(MAKE) -j1 -C ctypes/libffi

This doesn't quite work either. Here's the rub, afaict: we want to make $(LIBRARY) $(SHARED_LIBRARY) depend on libffi.a, and make libffi.a execute a submake which will possibly update it. But make short-circuits the libffi.a rule if the file already exists. Using my 'build_libffi' dummy dependency (like you suggest above, and like part of my patch did) forces make to perform the submake, but then it unconditionally rebuilds $(LIBRARY) $(SHARED_LIBRARY), because it doesn't know that build_libffi produces libffi.a as a result and that this result may or may not be updated. So we really want to say that $(LIBRARY) $(SHARED_LIBRARY) depends on libffi.a, and to rebuild libffi.a you must submake whether or not the file exists, but you must always do so to check if it changes and never short-circuit. Alternatively, $(LIBRARY) $(SHARED_LIBRARY) depends on build_libffi, and to build_libffi you must submake, the result of which is libffi.a; but if libffi.a does not change as a result, then ignore the $(LIBRARY) $(SHARED_LIBRARY) dependency on build_libffi.

As far as I know, there is no way to do either of these things. The fact that js/src/Makefile has no idea of all the source and object dependencies in js/src/ctypes/libffi/Makefile makes it impossible.

Which takes us back to...

> But I still don't understand why the export:: thing didn't work. We make pretty
> strong guarantees that the export:: phase runs before we get anywhere near the
> libs:: phase.

I don't either. I tried putting a 'sleep 5' in libffi's Makefile, and I cannot get the 'libs' target to run concurrently with 'export'. It always completes first.

Let's assume the 'ar: ctypes/libffi/.libs/libffi.a: Resource temporarily unavailable' is the problem. From playing around on my mac, this actually happens during libjs_static.a archiving, not libffi building. So all this build order stuff is probably just a strawman.

I'm gonna have to dig into what exactly the 'resource unavailable' message means. And why it shows up twice in the broken tinderbox logs, but once on my local mac builds (that never break). Further, it only shows up in my local builds with -jN, never with -j1.
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270754696.1270754922.21844.gz
OS X 10.5.2 mozilla-central build on 2010/04/08 12:24:56
s: bm-xserve19
The problem is that js builds both a static and shared library; each of those needs to call up 'ar' to access the guts of libffi.a; and both happen in parallel, so they race and one of them fails. Patch forthcoming.
This reverts the previous change that was landed, and makes $(SHARED_LIBRARY) depend on $(LIBRARY) so that they're not parallelized. I think this is correct since both of them can call up 'ar' regardless of SUB_LOBJS/SUB_SHLOBJS or whatever:

http://mxr.mozilla.org/mozilla-central/source/js/src/config/rules.mk#1173
http://mxr.mozilla.org/mozilla-central/source/js/src/config/rules.mk#1263

I didn't grok all the conditions that each happens under, but this seems safe, at least.

Fixes the race for me locally, tryserver's chewing on it now.
Attachment #437742 - Attachment is obsolete: true
Attachment #437944 - Flags: review?(ted.mielczarek)
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270765622.1270765718.24522.gz
OS X 10.5.2 mozilla-central build on 2010/04/08 15:27:02
s: bm-xserve09
Comment on attachment 437944 [details] [diff] [review]
for real this time
[Checkin: Comment 28]

Note that if you change js/src/config/rules.mk, you need to make the copy in config/rules.mk match exactly or "make check" will fail.
Attachment #437944 - Flags: review?(ted.mielczarek) → review+
Status: REOPENED → RESOLVED
Closed: 10 years ago10 years ago
Resolution: --- → FIXED
Rob, next time m-c is synced to TM you'll pick up this fix for the random red we've been seeing on the Mac opt machines.
Attachment #437416 - Attachment description: possible fix, mk2 → possible fix, mk2 [Checkin: Comment 16]
Attachment #437944 - Attachment description: for real this time → for real this time [Checkin: Comment 28]
Attachment #438028 - Flags: review?(bugspam.Callek) → review+
Comment on attachment 438028 [details] [diff] [review]
(Cv1-CC) Copy it to comm-central
[Checkin: Comment 31]


http://hg.mozilla.org/comm-central/rev/c4701f6cd60f
Attachment #438028 - Attachment description: (Cv1-CC) Copy it to comm-central → (Cv1-CC) Copy it to comm-central [Checkin: Comment 31]
Whiteboard: [orange]
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.