loads of l10n nightlies failed, central and aurora, several platforms. Infra?

RESOLVED FIXED

Status

Release Engineering
General
P2
normal
RESOLVED FIXED
7 years ago
5 years ago

People

(Reporter: Pike, Assigned: coop)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [l10n] try running "make tier_nspr")

Attachments

(2 attachments)

(Reporter)

Description

7 years ago
Is there something wrong on the infra for the l10n nightlies? Burning across the board.
Assignee: nobody → armenzg
Marking for clobber and re-triggered.

This is affecting linux64 and macosx64. Various slaves. All repacks. This is not affecting win32.

Interesting. A couple of hours ago linux64-ix-slave08 succeeded for m-a.
Oct 13 07:16 	failure 	Firefox mozilla-central linux64 l10n nightly 	#4089 	Failed make_libmar
Oct 13 05:36 	success 	Firefox mozilla-aurora linux64 l10n nightly 	#2879 	Build successful

I think this is m-c code related but could be wrong.

make[2]: *** [host_mar_create.o] Error 1
make[2]: Leaving directory `/builds/slave/m-cen-lnx64-l10n-ntly/build/mozilla-central/modules/libmar/src'
make[1]: *** [libs] Error 2
make[1]: Leaving directory `/builds/slave/m-cen-lnx64-l10n-ntly/build/mozilla-central/modules/libmar'
make: *** [default] Error 2
The retriggered job failed after clobber.
This is most likely code changes as the output is different from the successful run and the failing jobs. See below.

--- old 2011-10-13 12:17:27.000000000 -0400
+++ new 2011-10-13 12:17:00.000000000 -0400
@@ -1,4 +1,4 @@
 make -C src libs
 make[2]: Entering directory `/builds/slave/m-cen-lnx64-l10n-ntly/build/mozilla-central/modules/libmar/src'
-/builds/slave/m-cen-lnx64-l10n-ntly/build/mozilla-central/config/nsinstall -R -m 644 libhostmar.a ../../../dist/host/lib
-make[2]: Leaving directory `/builds/slave/m-cen-lnx64-l10n-ntly/build/mozilla-central/modules/libmar/src'
+mar_create.c
+/tools/gcc/bin/gcc -o host_mar_create.o -c  -Wall -W -Wno-unused -Wpointer-arith -Wdeclaration-after-statement -Wcast-align -W -pedantic -Wno-long-long -fno-strict-aliasing -pthread -ffunction-sections -fdata-sections -pipe  -DNDEBUG -DTRIMMED -g -Os -freorder-blocks  -fomit-frame-pointer -DXP_UNIX -Os -freorder-blocks    -I. -I. -I../../../dist/include -I../../../dist/include/nsprpub  `/builds/slave/m-cen-lnx64-l10n-ntly/build/mozilla-central/dist/sdk/bin/nspr-config --prefix=/builds/slave/m-cen-lnx64-l10n-ntly/build/mozilla-central/dist --includedir=/builds/slave/m-cen-lnx64-l10n-ntly/build/mozilla-central/dist/include/nspr --cflags` -I/builds/slave/m-cen-lnx64-l10n-ntly/build/mozilla-central/dist/include/nss     `/builds/slave/m-cen-lnx64-l10n-ntly/build/mozilla-central/dist/sdk/bin/nspr-config --prefix=/builds/slave/m-cen-lnx64-l10n-ntly/build/mozilla-central/dist --includedir=/builds/slave/m-cen-lnx64-l10n-ntly/build/mozilla-central/dist/include/nspr --cflags` /builds/slave/m-cen-lnx64-l10n-ntly/build/mozilla-central/modules/libmar/src/mar_create.c
(Reporter)

Comment 3

7 years ago
It doesn't find ptrypes.h, can you check if it's in the build dir?
[cltbld@linux64-ix-slave03 ~]$ cd /builds/slave/m-cen-lnx64-l10n-ntly/build/mozilla-central/
[cltbld@linux64-ix-slave03 mozilla-central]$ find . -name ptrypes.h
[cltbld@linux64-ix-slave03 mozilla-central]$ 

I can't even find it on the source tree:
http://mxr.mozilla.org/mozilla-central/find?text=&string=ptrypes.h
(Reporter)

Comment 5

7 years ago
IIII can't dance, IIIII can't type...

prtypes.h it is :-/
[cltbld@linux64-ix-slave03 mozilla-central]$ find . -name prtypes.h
./config/system_wrappers/prtypes.h
./nsprpub/pr/include/prtypes.h
./ipc/chromium/src/base/third_party/nspr/prtypes.h
./dist/include/nspr/prtypes.h
(Reporter)

Comment 7

7 years ago
OK, this seems to be lack of dist/sdk/bin/nspr-config.

What make targets do we run to compile and set things up these days?

make in nsprpub?

I suspect that's not as good as 

make tier_nsprpub

which adds 

tier_nspr_dirs += config/nspr

in config/nspr/build.mk.
This seems to be fixed.
http://l10n.mozilla.org/~axel/nightlies/?date=2011-10-16
Status: NEW → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
(Reporter)

Comment 9

7 years ago
I have seen this libmar problem before, and I really think it's a problem on how the nspr step is doing it's work right now.

I'd love for someone with a deeper understanding of how our targets are supposed to work to comment.

AFAICT, we're doing

make -C nsprpub

and may want to 

make tier_nspr

as that includes config/nspr, IIRC.

Ted, Joey?
(Reporter)

Comment 10

7 years ago
Came back today, with vengenance. http://tinderbox.mozilla.org/showlog.cgi?log=Mozilla-l10n-or/1319290588.1319290676.2262.gz&fulltext=1 and http://l10n.mozilla.org/~axel/nightlies/?date=2011-10-22
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Should we move this to Build:Config?

Anything that we can do from releng side?

ted, joey can you have a look at this?
Assignee: armenzg → nobody
Is this bug still valid?
(Reporter)

Comment 13

7 years ago
I guess it is, until we know what's actually the root cause of those errors.
(Assignee)

Comment 14

7 years ago
(In reply to Axel Hecht [:Pike] from comment #13)
> I guess it is, until we know what's actually the root cause of those errors.

FWIW, I think this is our bug. We run 'make' for nsprpub in the BuildStep immediately before 'make libmar.'

Maybe we should try the "make tier_nspr" as Axel suggests?
OS: Mac OS X → All
Priority: -- → P3
Hardware: x86 → All
Whiteboard: [l10n]
(Reporter)

Comment 15

7 years ago
Happened today again with a host of mac nightlies, http://l10n.mozilla-community.org/~axel/nightlies/?date=2011-11-08.
If we switch to make tier_nspr we should remember to adjust the L10n scripts used for releases as well.
Priority: P3 → --
Whiteboard: [l10n] → [l10n] try running "make tier_nspr"
(In reply to Chris Cooper [:coop] from comment #14)
> (In reply to Axel Hecht [:Pike] from comment #13)
> > I guess it is, until we know what's actually the root cause of those errors.
> 
> FWIW, I think this is our bug. We run 'make' for nsprpub in the BuildStep
> immediately before 'make libmar.'
> 
> Maybe we should try the "make tier_nspr" as Axel suggests?

I was hitting this while testing bug 509158, and after running "make tier_nspr", libmar built fine.
(Assignee)

Updated

7 years ago
Assignee: nobody → coop
Priority: -- → P3
(Assignee)

Updated

7 years ago
Status: REOPENED → ASSIGNED
Priority: P3 → P2
(Assignee)

Comment 18

7 years ago
Created attachment 576982 [details] [diff] [review]
Use make tier_nspr when building for l10n

This seems to have done the trick in staging.

Another patch for the tools lib coming up shortly.
Attachment #576982 - Flags: review?(armenzg)
(Assignee)

Comment 19

7 years ago
Created attachment 576984 [details] [diff] [review]
Use make tier_nspr when building for l10n (tools)

I'm special-casing nsprpub in the l10n.py lib, which seems IMO like the cleanest way to do this. Let me know if you think differently.
Attachment #576984 - Flags: review?(armenzg)
Attachment #576982 - Flags: review?(armenzg) → review+
Comment on attachment 576984 [details] [diff] [review]
Use make tier_nspr when building for l10n (tools)

Excellent! Thanks coop for taking care of this one.
Attachment #576984 - Flags: review?(armenzg) → review+
(Reporter)

Comment 21

7 years ago
This didn't land yet, right? Seems to take its toll today again.
(Assignee)

Comment 22

7 years ago
(In reply to Axel Hecht [:Pike] from comment #21)
> This didn't land yet, right? Seems to take its toll today again.

Correct. I'll start that process today.
(Assignee)

Comment 23

7 years ago
Comment on attachment 576984 [details] [diff] [review]
Use make tier_nspr when building for l10n (tools)

https://hg.mozilla.org/build/tools/rev/b44116693a53
Attachment #576984 - Flags: checked-in+
(Assignee)

Comment 24

7 years ago
Comment on attachment 576982 [details] [diff] [review]
Use make tier_nspr when building for l10n

https://hg.mozilla.org/build/buildbotcustom/rev/9d1292224c50
Attachment #576982 - Flags: checked-in+

Comment 25

7 years ago
this has landed in production as of 2011-12-01 1200 PST
(Assignee)

Updated

7 years ago
Status: ASSIGNED → RESOLVED
Last Resolved: 7 years ago7 years ago
Resolution: --- → FIXED
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.