869406 - Crash in genrb at haveAliasData - Intermittent OSX "Segmentation fault: 11" while compiling ICU

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=22662396&tree=Mozilla-Inbound

OS X 10.7 mozilla-inbound build on 2013-05-06 19:39:38 PDT for push a28afac94104
slave: bld-lion-r5-002

DYLD_LIBRARY_PATH=../lib:../stubdata:../tools/ctestfw:$DYLD_LIBRARY_PATH  ../bin/genrb --usePoolBundle -k -R -i ./out/build/icudt50l -s /builds/slave/m-in-osx64-0000000000000000000/build/js/src/../../intl/icu/source/data/zone -d ./out/build/icudt50l/zone uz_Arab.txt
make[8]: *** [out/build/icudt50l/dayPeriods.res] Segmentation fault: 11
make[8]: *** Waiting for unfinished jobs....
make[7]: *** [all-recursive] Error 2
make[6]: *** [export] Error 2
make[5]: *** [default] Error 2
make[4]: *** [tier_js] Error 2
make[3]: *** [default] Error 2
make[2]: *** [realbuild] Error 2
make[1]: *** [realbuild] Error 2
make: *** [build] Error 2

Comment 1

[(not currently active) Ted Mielczarek]

If I'm reading this right, this is a segfault in a host tool that ICU builds and runs? If so this isn't actually a build config issue, it's just a bug in ICU.

Comment 2

[Ryan VanderMeulen [:RyanVM]]

https://tbpl.mozilla.org/php/getParsedLog.php?id=22712302&tree=Birch

Comment 3

[Phil Ringnalda (:philor)]

https://tbpl.mozilla.org/php/getParsedLog.php?id=22714529&tree=Cedar

Comment 4

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

I get this occasionally as well - here's a Mac stack:

Thread 0 Crashed:: Dispatch queue: com.apple.main-thread
0   genrb                         	0x0000000100152c60 haveAliasData(UErrorCode*) + 656 (ucnv_io.cpp:295)
1   genrb                         	0x0000000100153c85 ucnv_io_countKnownConverters_50 + 21 (ucnv_io.cpp:1094)
2   genrb                         	0x000000010013a815 initData + 21 (uinit.c:44)
3   genrb                         	0x000000010013a4d8 ucln_mutexedInit_50 + 88 (ucln_cmn.c:41)
4   genrb                         	0x000000010013a7f5 u_init_50 + 37 (uinit.c:58)
5   genrb                         	0x00000001000020bf main + 1391 (genrb.c:261)
6   genrb                         	0x00000001000016c4 start + 52


I guess we should file an upstream bug?

Comment 5

[Jesse Ruderman]

I hit this twice today, once when building the JS shell and once when building Firefox.

Comment 6

[Mike Hommey [:glandium]]

(In reply to Ted Mielczarek [:ted.mielczarek] from comment #1)
> If I'm reading this right, this is a segfault in a host tool that ICU builds
> and runs? If so this isn't actually a build config issue, it's just a bug in
> ICU.

It's actually a very likely race condition in their build system. And it turns out we may not actually be enforcing -j1 on icu build as I'd expect.

Comment 7

[Mike Hommey [:glandium]]

Attachment: Really enforce non-parallel build of ICU

We were really using -j1 on windows, and kind of relying on $(GMAKE) making make ignore -jN because of how $(MAKE) is magic and how not using it can make it not pass flags to a child make. My guess is that this doesn't work properly on mac, and we're still doing parallel icu builds there. The logs suggest that very much, with the crashes refering to commands way before the last commands run for the icu build.

While touching that, only really enforce $(GMAKE) when running with pymake, so that the actually make running is used (no reason to switch to /another/ make if you have several installed ; i hit that problem when trying make 4.0).

Comment 8

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Intl landed in Firefox 22 (supposedly via bug 769872), it'll be fantastic to backport this build tweak back to the branches because it causes intermittent issues with bisection through autoBisect for the fuzzers, along with build issues on TBPL.

Comment 9

[Mike Hommey [:glandium]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/b0e636c410b8

Comment 10

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/b0e636c410b8

Comment 11

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

Mike, do you mind nominating this for backport to the branches? This will help future builds of branch binaries. Thanks!

Comment 12

[Mike Hommey [:glandium]]

(In reply to Gary Kwong [:gkw] [:nth10sd] (still catching up on bugmail) from comment #11)
> Mike, do you mind nominating this for backport to the branches? This will
> help future builds of branch binaries. Thanks!

It's not needed, actually. ICU was only built on nightly builds. See http://hg.mozilla.org/releases/mozilla-aurora/file/75b0b968f3ed/configure.in#l9303
In fact, it wasn't even enabled on nightlies before 26 (m-r and m-b don't have the code linked above)
It is, however, enabled on standalone js builds, but I doubt it's going to be that much of a problem.

Comment 13

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

(In reply to Mike Hommey [:glandium] from comment #12)
> It is, however, enabled on standalone js builds, but I doubt it's going to
> be that much of a problem.

autoBisect works on standalone js builds and this is precisely why I'd prefer for this to be on the branches.

Comment 14

[Mike Hommey [:glandium]]

Comment on attachment 816173 [details] [diff] [review]
Really enforce non-parallel build of ICU

[Approval Request Comment]
Bug caused by bug 724533
User impact if declined: see comment 8
Testing completed (on m-c, etc.): Landed on m-i a couple days ago.
Risk to taking this patch (and alternatives if risky): It doesn't affect firefox builds, as the feature it touches is only enabled on standalone js builds. Even for those, this is only a change in what is used to invoke its build system, which, in the worst case scenario, would blow up visibly on standalone js builds (as opposed to breaking the build in subtle hidden ways).
String or IDL/UUID changes made by this patch: None

Comment 15

[Gary Kwong [:gkw] [:nth10sd] (NOT official MoCo now)]

fwiw, not really needed on release branch I think, since the switchover is going to happen in ~2 weeks, we can live without it there.

Comment 16

[Alex Keybl [:akeybl]]

Comment on attachment 816173 [details] [diff] [review]
Really enforce non-parallel build of ICU

Sounds like a promise of no risk :). We'll take it.

Comment 17

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/e49eac40b0bf
https://hg.mozilla.org/releases/mozilla-beta/rev/b869354c8889
https://hg.mozilla.org/releases/mozilla-esr24/rev/380b31c000fb
https://hg.mozilla.org/releases/mozilla-release/rev/6965a0a490c7