557413 - ar: ctypes/libffi/.libs/libffi.a: Resource temporarily unavailable

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Jeff Muizelaar [:jrmuizel]]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270514732.1270516404.7315.gz

ar: ctypes/libffi/.libs/libffi.a: Resource temporarily unavailable
Making symlinks to the original object files in the archive libraries ctypes/libffi/.libs/libffi.a
ar: ctypes/libffi/.libs/libffi.a: Resource temporarily unavailable
g++-4.2 -arch i386  -fno-rtti -fno-exceptions -Wall -Wpointer-arith -Woverloaded-virtual -Wsynth -Wno-ctor-dtor-privacy -Wno-non-virtual-dtor -Wcast-align -Wno-invalid-offsetof -Wno-variadic-macros -Wno-long-long -gdwarf-2 -isysroot /Developer/SDKs/MacOSX10.5.sdk -fno-strict-aliasing -fpascal-strings -fno-common -pthread  -DNDEBUG -DTRIMMED -gdwarf-2 -O3 -fstrict-aliasing  -fPIC  -o libmozjs.dylib  jsapi.o jsarena.o jsarray.o jsatom.o jsbool.o jscntxt.o jsdate.o jsdbgapi.o jsdhash.o jsdtoa.o jsemit.o jsexn.o jsfun.o jsgc.o jshash.o jsinterp.o jsinvoke.o jsiter.o jslock.o jslog2.o jsmath.o jsnum.o jsobj.o json.o jsopcode.o jsparse.o jsprf.o jspropertycache.o jspropertytree.o jsregexp.o jsscan.o jsscope.o jsscript.o jsstr.o jstask.o jstypedarray.o jsutil.o jsxdrapi.o jsxml.o prmjtime.o jstracer.o Assembler.o Allocator.o CodeAlloc.o Containers.o Fragmento.o LIR.o njconfig.o RegAlloc.o avmplus.o Nativei386.o jsbuiltins.o VMPI.o ctypes/CTypes.o ctypes/Library.o   debug.o prep_cif.o types.o raw_api.o java_raw_api.o closures.o ffi.o darwin.o ffi64.o darwin64.o  -lobjc -framework Cocoa  -framework ExceptionHandling -Wl,-executable_path,/bin -Wl,-dead_strip -L/builds/moz2_slave/mozilla-central-macosx/build/obj-firefox/i386/dist/lib -lplds4 -lplc4 -lnspr4 -dynamiclib -install_name @executable_path/libmozjs.dylib -compatibility_version 1 -current_version 1 -single_module  -lm  
i686-apple-darwin9-g++-4.2.1: ffi64.o: No such file or directory
i686-apple-darwin9-g++-4.2.1: darwin64.o: No such file or directory

Comment 1

[Phil Ringnalda (:philor)]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270560772.1270562293.23511.gz
OS X 10.5.2 mozilla-central build on 2010/04/06 06:32:52
s: bm-xserve19

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270562185.1270563868.28071.gz
OS X 10.5.2 mozilla-central build on 2010/04/06 06:56:25
s: bm-xserve18

Comment 2

[(not currently active) Ted Mielczarek]

I think either we need to figure out a solution to this ASAP or back out that change.

Comment 3

[[e-mail]]

I'll take a look when I get in today...

Comment 4

[Benjamin Smedberg]

Which is the bug here? "Resource temporarily unavailable" or "No such file or directory"?

Earlier is ranlib: file: .libs/libffi.a(ffi64.o) has no symbols is expected behavior. Are we failing to unpack the object because it doesn't have any symbols?

BTW, does this always happen on the second pass of the universal build? If so, perhaps libffi is polluting the srcdir somehow?

Comment 5

[[e-mail]]

(In reply to comment #4)
> Which is the bug here? "Resource temporarily unavailable" or "No such file or
> directory"?

Both, I think... I'm guessing that the former means that the archive is already open and being written to by the other instance of 'ar', per my race theory. Not sure about the latter... I'd think that one of the ar's succeeds, but maybe the link step itself is racing with 'ar'... (unlikely!?)

> Earlier is ranlib: file: .libs/libffi.a(ffi64.o) has no symbols is expected
> behavior. Are we failing to unpack the object because it doesn't have any
> symbols?

I don't think so; as you say it's expected and this has always been the case when building libffi. The other archive members should definitely have symbols...

> BTW, does this always happen on the second pass of the universal build? If so,
> perhaps libffi is polluting the srcdir somehow?

Looking...

Comment 6

[[e-mail]]

From the full build log (http://tinderbox.mozilla.org/showlog.cgi?tree=Firefox&errorparser=unix&logfile=1270514732.1270516404.7315.gz&buildtime=1270514732&buildname=OS%20X%2010.5.2%20mozilla-central%20build&fulltext=1):

/bin/sh ./libtool --tag=CC   --mode=link gcc-4.2 -arch ppc -Wall -g -fexceptions -Wall -W -Wno-unused -Wpointer-arith -Wcast-align -W -Wno-long-long -gdwarf-2 -isysroot /Developer/SDKs/MacOSX10.5.sdk -fno-strict-aliasing -fpascal-strings -fno-common -pthread -version-info `grep -v '^#' /builds/moz2_slave/mozilla-central-macosx/build/js/src/ctypes/libffi/libtool-version`    -o libffi.la -rpath /usr/local/lib src/debug.lo src/prep_cif.lo src/types.lo src/raw_api.lo src/java_raw_api.lo src/closures.lo              src/powerpc/ffi_darwin.lo src/powerpc/darwin.lo src/powerpc/darwin_closure.lo             
/bin/sh ./libtool --tag=CC   --mode=link gcc-4.2 -arch ppc -Wall -g -fexceptions -Wall -W -Wno-unused -Wpointer-arith -Wcast-align -W -Wno-long-long -gdwarf-2 -isysroot /Developer/SDKs/MacOSX10.5.sdk -fno-strict-aliasing -fpascal-strings -fno-common -pthread   -o libffi_convenience.la  src/debug.lo src/prep_cif.lo src/types.lo src/raw_api.lo src/java_raw_api.lo src/closures.lo              src/powerpc/ffi_darwin.lo src/powerpc/darwin.lo src/powerpc/darwin_closure.lo             
libtool: link: ar cru .libs/libffi_convenience.a  src/debug.o src/prep_cif.o src/types.o src/raw_api.o src/java_raw_api.o src/closures.o src/powerpc/ffi_darwin.o src/powerpc/darwin.o src/powerpc/darwin_closure.o
libtool: link: ar cru .libs/libffi.a  src/debug.o src/prep_cif.o src/types.o src/raw_api.o src/java_raw_api.o src/closures.o src/powerpc/ffi_darwin.o src/powerpc/darwin.o src/powerpc/darwin_closure.o
ranlib: ranlib: file: .libs/libffi.a(raw_api.o) file: .libs/libffi_convenience.a(raw_api.o) has no symbolshas no symbols

ranlib: ranlib: file: .libs/libffi.a(java_raw_api.o) file: .libs/libffi_convenience.a(java_raw_api.o) has no symbolshas no symbols

libtool: link: ranlib .libs/libffi.a
libtool: link: ranlib .libs/libffi_convenience.a
ranlib: ranlib: file: .libs/libffi.a(raw_api.o) file: .libs/libffi_convenience.a(raw_api.o) has no symbolshas no symbols

ranlib: ranlib: file: .libs/libffi.a(java_raw_api.o) file: .libs/libffi_convenience.a(java_raw_api.o) has no symbolshas no symbols

libtool: link: ( cd ".libs" && rm -f "libffi_convenience.la" && ln -s "../libffi_convenience.la" "libffi_convenience.la" )
libtool: link: ( cd ".libs" && rm -f "libffi.la" && ln -s "../libffi.la" "libffi.la" )

I think my race theory is off. The source files in libffi are getting built once, so the js/src makefile's 'export' rule certainly isn't racing. But the link steps in libffi seem to be getting issued twice and execute simultaneously, as clearly evidenced by the garbled output from ranlib toward the end.

Does the definition of SUBMAKE include -j flags? I'm looking at rules.mk now...

Comment 7

[[e-mail]]

For reference, this is the rule that builds libffi:
http://mxr.mozilla.org/mozilla-central/source/js/src/Makefile.in#372

Comment 8

[Benjamin Smedberg]

It looks to me as if it's linking the .la and the .a at the same time, which is what I would expect in a parallel build and should be fine unless there are conflicting intermediate files that both link steps create.

Comment 9

[Jonathan Kew [:jfkthame]]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270583838.1270583916.23343.gz
OS X 10.5.2 mozilla-central build on 2010/04/06 12:57:18
s: bm-xserve17

Comment 10

[[e-mail]]

Attachment: possible fix

This drops the $SUBMAKE bit in favor of an explicit call to $MAKE without any $MAKEFLAGS.

If the problem is building libffi parallel, this should fix it. If not, it'll give us another datapoint...

Comment 11

[Benjamin Smedberg]

I think you'd be better off just passing -j1 explicitly, not completely removing MAKEFLAGS. You'll be dropping -k or -s flags which people care about.

Comment 12

[[e-mail]]

Attachment: possible fix, mk2 [Checkin: Comment 16]

OK, this one just adds '-j1' -- according to the make manual, the last '-j' takes effect, so this should have the desired effect. Assuming pymake also does so. :)

Otherwise, I can write a sed substitution to pull out '-j' from MAKEFLAGS.

Comment 13

[Jonathan Kew [:jfkthame]]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270623748.1270625205.14554.gz
OS X 10.5.2 mozilla-central build on 2010/04/07 00:02:28
s: bm-xserve19

Comment 14

[Phil Ringnalda (:philor)]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270664894.1270664972.21458.gz
OS X 10.5.2 mozilla-central build on 2010/04/07 11:28:14
s: bm-xserve19

Comment 15

[Phil Ringnalda (:philor)]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270663780.1270666559.26408.gz
OS X 10.5.2 mozilla-central build on 2010/04/07 11:09:40
s: bm-xserve09

Comment 16

[[e-mail]]

Comment on attachment 437416 [details] [diff] [review]
possible fix, mk2
[Checkin: Comment 16]

http://hg.mozilla.org/mozilla-central/rev/07303da1d826

Comment 17

[[e-mail]]

Landed on m-c. Will let this sync to TM by natural processes.

Comment 18

[[e-mail]]

This also helps me out, for an otherwise unrelated parallelism reason ;-)

Comment 19

[[e-mail]]

http://tinderbox.mozilla.org/showlog.cgi?tree=Firefox&errorparser=unix&logfile=1270675968.1270676055.22988.gz&buildtime=1270675968&buildname=OS%20X%2010.5.2%20mozilla-central%20build&fulltext=1

Still happening. I'm gonna rejigger dependencies and see if that fixes it...

Comment 20

[[e-mail]]

Attachment: really actually seriously fix deps

I *think* this does it. I've tested it in all the ways I can think of, and it doesn't break anything. And it will probably fix the problem.

This makes the dependencies very explicit, so that the 'libs' target depends on 'libffi.a', 'libffi.a' depends on 'build_libffi', and 'build_libffi' issues the 'make' command. All three of these are required to get make to do the right thing.

Comment 21

[Benjamin Smedberg]

Comment on attachment 437742 [details] [diff] [review]
really actually seriously fix deps

>diff --git a/js/src/Makefile.in b/js/src/Makefile.in

>-export::
>+# Specify a 'libs' target dependency on libffi, to make sure it gets built
>+# before libraries are linked. Also specify -j1 so we don't get a parallel
>+# build.
>+libs:: ctypes/libffi/.libs/libffi.$(LIB_SUFFIX)

There is nothing here which prevents us building $(LIBRARY) or $(SHARED_LIBRARY) before this. I don't believe this can work correctly. I think you want:

$(LIBRARY) $(SHARED_LIBRARY): build_libffi
	$(MAKE) -j1 -C ctypes/libffi

But I still don't understand why the export:: thing didn't work. We make pretty strong guarantees that the export:: phase runs before we get anywhere near the libs:: phase.

Comment 22

[[e-mail]]

(In reply to comment #21)
> There is nothing here which prevents us building $(LIBRARY) or
> $(SHARED_LIBRARY) before this. I don't believe this can work correctly. I think
> you want:
> 
> $(LIBRARY) $(SHARED_LIBRARY): build_libffi
>     $(MAKE) -j1 -C ctypes/libffi

This doesn't quite work either. Here's the rub, afaict: we want to make $(LIBRARY) $(SHARED_LIBRARY) depend on libffi.a, and make libffi.a execute a submake which will possibly update it. But make short-circuits the libffi.a rule if the file already exists. Using my 'build_libffi' dummy dependency (like you suggest above, and like part of my patch did) forces make to perform the submake, but then it unconditionally rebuilds $(LIBRARY) $(SHARED_LIBRARY), because it doesn't know that build_libffi produces libffi.a as a result and that this result may or may not be updated. So we really want to say that $(LIBRARY) $(SHARED_LIBRARY) depends on libffi.a, and to rebuild libffi.a you must submake whether or not the file exists, but you must always do so to check if it changes and never short-circuit. Alternatively, $(LIBRARY) $(SHARED_LIBRARY) depends on build_libffi, and to build_libffi you must submake, the result of which is libffi.a; but if libffi.a does not change as a result, then ignore the $(LIBRARY) $(SHARED_LIBRARY) dependency on build_libffi.

As far as I know, there is no way to do either of these things. The fact that js/src/Makefile has no idea of all the source and object dependencies in js/src/ctypes/libffi/Makefile makes it impossible.

Which takes us back to...

> But I still don't understand why the export:: thing didn't work. We make pretty
> strong guarantees that the export:: phase runs before we get anywhere near the
> libs:: phase.

I don't either. I tried putting a 'sleep 5' in libffi's Makefile, and I cannot get the 'libs' target to run concurrently with 'export'. It always completes first.

Let's assume the 'ar: ctypes/libffi/.libs/libffi.a: Resource temporarily unavailable' is the problem. From playing around on my mac, this actually happens during libjs_static.a archiving, not libffi building. So all this build order stuff is probably just a strawman.

I'm gonna have to dig into what exactly the 'resource unavailable' message means. And why it shows up twice in the broken tinderbox logs, but once on my local mac builds (that never break). Further, it only shows up in my local builds with -jN, never with -j1.

Comment 23

[Phil Ringnalda (:philor)]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270754696.1270754922.21844.gz
OS X 10.5.2 mozilla-central build on 2010/04/08 12:24:56
s: bm-xserve19

Comment 24

[[e-mail]]

The problem is that js builds both a static and shared library; each of those needs to call up 'ar' to access the guts of libffi.a; and both happen in parallel, so they race and one of them fails. Patch forthcoming.

Comment 25

[[e-mail]]

Attachment: for real this time [Checkin: Comment 28]

This reverts the previous change that was landed, and makes $(SHARED_LIBRARY) depend on $(LIBRARY) so that they're not parallelized. I think this is correct since both of them can call up 'ar' regardless of SUB_LOBJS/SUB_SHLOBJS or whatever:

http://mxr.mozilla.org/mozilla-central/source/js/src/config/rules.mk#1173
http://mxr.mozilla.org/mozilla-central/source/js/src/config/rules.mk#1263

I didn't grok all the conditions that each happens under, but this seems safe, at least.

Fixes the race for me locally, tryserver's chewing on it now.

Comment 26

[Phil Ringnalda (:philor)]

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1270765622.1270765718.24522.gz
OS X 10.5.2 mozilla-central build on 2010/04/08 15:27:02
s: bm-xserve09

Comment 27

[(not currently active) Ted Mielczarek]

Comment on attachment 437944 [details] [diff] [review]
for real this time
[Checkin: Comment 28]

Note that if you change js/src/config/rules.mk, you need to make the copy in config/rules.mk match exactly or "make check" will fail.

Comment 28

[[e-mail]]

Comment on attachment 437944 [details] [diff] [review]
for real this time
[Checkin: Comment 28]

http://hg.mozilla.org/mozilla-central/rev/4c1ed264c69d

Comment 29

[[e-mail]]

Rob, next time m-c is synced to TM you'll pick up this fix for the random red we've been seeing on the Mac opt machines.

Comment 30

[Serge Gautherie (:sgautherie)]

Attachment: (Cv1-CC) Copy it to comm-central [Checkin: Comment 31]

Comment 31

[Serge Gautherie (:sgautherie)]

Comment on attachment 438028 [details] [diff] [review]
(Cv1-CC) Copy it to comm-central
[Checkin: Comment 31]


http://hg.mozilla.org/comm-central/rev/c4701f6cd60f