694332 - loads of l10n nightlies failed, central and aurora, several platforms. Infra?

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Axel Hecht [:Pike]]

Is there something wrong on the infra for the l10n nightlies? Burning across the board.

Comment 1

[Armen [:armenzg]]

Marking for clobber and re-triggered.

This is affecting linux64 and macosx64. Various slaves. All repacks. This is not affecting win32.

Interesting. A couple of hours ago linux64-ix-slave08 succeeded for m-a.
Oct 13 07:16 	failure 	Firefox mozilla-central linux64 l10n nightly 	#4089 	Failed make_libmar
Oct 13 05:36 	success 	Firefox mozilla-aurora linux64 l10n nightly 	#2879 	Build successful

I think this is m-c code related but could be wrong.

make[2]: *** [host_mar_create.o] Error 1
make[2]: Leaving directory `/builds/slave/m-cen-lnx64-l10n-ntly/build/mozilla-central/modules/libmar/src'
make[1]: *** [libs] Error 2
make[1]: Leaving directory `/builds/slave/m-cen-lnx64-l10n-ntly/build/mozilla-central/modules/libmar'
make: *** [default] Error 2

Comment 2

[Armen [:armenzg]]

The retriggered job failed after clobber.
This is most likely code changes as the output is different from the successful run and the failing jobs. See below.

--- old 2011-10-13 12:17:27.000000000 -0400
+++ new 2011-10-13 12:17:00.000000000 -0400
@@ -1,4 +1,4 @@
 make -C src libs
 make[2]: Entering directory `/builds/slave/m-cen-lnx64-l10n-ntly/build/mozilla-central/modules/libmar/src'
-/builds/slave/m-cen-lnx64-l10n-ntly/build/mozilla-central/config/nsinstall -R -m 644 libhostmar.a ../../../dist/host/lib
-make[2]: Leaving directory `/builds/slave/m-cen-lnx64-l10n-ntly/build/mozilla-central/modules/libmar/src'
+mar_create.c
+/tools/gcc/bin/gcc -o host_mar_create.o -c  -Wall -W -Wno-unused -Wpointer-arith -Wdeclaration-after-statement -Wcast-align -W -pedantic -Wno-long-long -fno-strict-aliasing -pthread -ffunction-sections -fdata-sections -pipe  -DNDEBUG -DTRIMMED -g -Os -freorder-blocks  -fomit-frame-pointer -DXP_UNIX -Os -freorder-blocks    -I. -I. -I../../../dist/include -I../../../dist/include/nsprpub  `/builds/slave/m-cen-lnx64-l10n-ntly/build/mozilla-central/dist/sdk/bin/nspr-config --prefix=/builds/slave/m-cen-lnx64-l10n-ntly/build/mozilla-central/dist --includedir=/builds/slave/m-cen-lnx64-l10n-ntly/build/mozilla-central/dist/include/nspr --cflags` -I/builds/slave/m-cen-lnx64-l10n-ntly/build/mozilla-central/dist/include/nss     `/builds/slave/m-cen-lnx64-l10n-ntly/build/mozilla-central/dist/sdk/bin/nspr-config --prefix=/builds/slave/m-cen-lnx64-l10n-ntly/build/mozilla-central/dist --includedir=/builds/slave/m-cen-lnx64-l10n-ntly/build/mozilla-central/dist/include/nspr --cflags` /builds/slave/m-cen-lnx64-l10n-ntly/build/mozilla-central/modules/libmar/src/mar_create.c

Comment 3

[Axel Hecht [:Pike]]

It doesn't find ptrypes.h, can you check if it's in the build dir?

Comment 4

[Armen [:armenzg]]

[cltbld@linux64-ix-slave03 ~]$ cd /builds/slave/m-cen-lnx64-l10n-ntly/build/mozilla-central/
[cltbld@linux64-ix-slave03 mozilla-central]$ find . -name ptrypes.h
[cltbld@linux64-ix-slave03 mozilla-central]$ 

I can't even find it on the source tree:
http://mxr.mozilla.org/mozilla-central/find?text=&string=ptrypes.h

Comment 5

[Axel Hecht [:Pike]]

IIII can't dance, IIIII can't type...

prtypes.h it is :-/

Comment 6

[Armen [:armenzg]]

[cltbld@linux64-ix-slave03 mozilla-central]$ find . -name prtypes.h
./config/system_wrappers/prtypes.h
./nsprpub/pr/include/prtypes.h
./ipc/chromium/src/base/third_party/nspr/prtypes.h
./dist/include/nspr/prtypes.h

Comment 7

[Axel Hecht [:Pike]]

OK, this seems to be lack of dist/sdk/bin/nspr-config.

What make targets do we run to compile and set things up these days?

make in nsprpub?

I suspect that's not as good as 

make tier_nsprpub

which adds 

tier_nspr_dirs += config/nspr

in config/nspr/build.mk.

Comment 8

[Armen [:armenzg]]

This seems to be fixed.
http://l10n.mozilla.org/~axel/nightlies/?date=2011-10-16

Comment 9

[Axel Hecht [:Pike]]

I have seen this libmar problem before, and I really think it's a problem on how the nspr step is doing it's work right now.

I'd love for someone with a deeper understanding of how our targets are supposed to work to comment.

AFAICT, we're doing

make -C nsprpub

and may want to 

make tier_nspr

as that includes config/nspr, IIRC.

Ted, Joey?

Comment 10

[Axel Hecht [:Pike]]

Came back today, with vengenance. http://tinderbox.mozilla.org/showlog.cgi?log=Mozilla-l10n-or/1319290588.1319290676.2262.gz&fulltext=1 and http://l10n.mozilla.org/~axel/nightlies/?date=2011-10-22

Comment 11

[Armen [:armenzg]]

Should we move this to Build:Config?

Anything that we can do from releng side?

ted, joey can you have a look at this?

Comment 12

[Rail Aliiev [:rail]]

Is this bug still valid?

Comment 13

[Axel Hecht [:Pike]]

I guess it is, until we know what's actually the root cause of those errors.

Comment 14

[Chris Cooper [:coop] (he/him)]

(In reply to Axel Hecht [:Pike] from comment #13)
> I guess it is, until we know what's actually the root cause of those errors.

FWIW, I think this is our bug. We run 'make' for nsprpub in the BuildStep immediately before 'make libmar.'

Maybe we should try the "make tier_nspr" as Axel suggests?

Comment 15

[Axel Hecht [:Pike]]

Happened today again with a host of mac nightlies, http://l10n.mozilla-community.org/~axel/nightlies/?date=2011-11-08.

Comment 16

[Armen [:armenzg]]

If we switch to make tier_nspr we should remember to adjust the L10n scripts used for releases as well.

Comment 17

[[e-mail] (:bhearsum)]

(In reply to Chris Cooper [:coop] from comment #14)
> (In reply to Axel Hecht [:Pike] from comment #13)
> > I guess it is, until we know what's actually the root cause of those errors.
> 
> FWIW, I think this is our bug. We run 'make' for nsprpub in the BuildStep
> immediately before 'make libmar.'
> 
> Maybe we should try the "make tier_nspr" as Axel suggests?

I was hitting this while testing bug 509158, and after running "make tier_nspr", libmar built fine.

Comment 18

[Chris Cooper [:coop] (he/him)]

Attachment: Use make tier_nspr when building for l10n

This seems to have done the trick in staging.

Another patch for the tools lib coming up shortly.

Comment 19

[Chris Cooper [:coop] (he/him)]

Attachment: Use make tier_nspr when building for l10n (tools)

I'm special-casing nsprpub in the l10n.py lib, which seems IMO like the cleanest way to do this. Let me know if you think differently.

Comment 20

[Armen [:armenzg]]

Comment on attachment 576984 [details] [diff] [review]
Use make tier_nspr when building for l10n (tools)

Excellent! Thanks coop for taking care of this one.

Comment 21

[Axel Hecht [:Pike]]

This didn't land yet, right? Seems to take its toll today again.

Comment 22

[Chris Cooper [:coop] (he/him)]

(In reply to Axel Hecht [:Pike] from comment #21)
> This didn't land yet, right? Seems to take its toll today again.

Correct. I'll start that process today.

Comment 23

[Chris Cooper [:coop] (he/him)]

Comment on attachment 576984 [details] [diff] [review]
Use make tier_nspr when building for l10n (tools)

https://hg.mozilla.org/build/tools/rev/b44116693a53

Comment 24

[Chris Cooper [:coop] (he/him)]

Comment on attachment 576982 [details] [diff] [review]
Use make tier_nspr when building for l10n

https://hg.mozilla.org/build/buildbotcustom/rev/9d1292224c50

Comment 25

[Mike Taylor [:bear]]

this has landed in production as of 2011-12-01 1200 PST