437798 - crazyhorse (Mozilla1.8) went red for no apparent reason

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P1)

Description

[Samuel Sidler (old account; do not CC)]

The build starting at ~8:05am today went red. We really need these machines green this weekend (whether that's IT or build) if we plan on releasing 2.0.0.15 on schedule.

The error I see is: Error: CVS checkout failed.

So, possibly part of bug 435134?

Comment 1

[chizu]

Filesystem corruption, waiting on the second run of fsck.

Comment 2

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Caused by ongoing NetApp woes... see bug#435134 for details.

Comment 3

[[e-mail] (:bhearsum)]

Turns out this system got hit extra-hard by bug 435134. It's going to need some additional coaxing to get working again. Re-opening to track it separately.

Comment 4

[[e-mail] (:bhearsum)]

And for the record, the current issue is ld failing in nspr via sigsegv:
gcc -o plgetopt.o -c      -pipe -Wall -pthread -O2 -gstabs+ -fPIC  -UDEBUG  -DMOZILLA_CLIENT=1 -DNDEBUG=1 -DHAVE_VISIBILITY_HIDDEN_ATTRIBUTE=1 -DXP_UNIX=1 -D_GNU_SOURCE=1 -DHAVE_FCNTL_FILE_LOCKING=1 -DLINUX=1 -Di386=1 -DHAVE_LCHOWN=1 -DHAVE_STRERROR=1 -D_REENTRANT=1  -DFORCE_PR_LOG -D_PR_PTHREADS -UHAVE_CVAR_BUILT_ON_SEM -I/builds/tinderbox/Tb-Mozilla1.8/Linux_2.4.18-14_Depend/mozilla/dist/include/nspr  plgetopt.c
rm -f libplc4.a
/usr/bin/ar cr libplc4.a ./plvrsion.o ./strlen.o ./strcpy.o ./strdup.o ./strcat.o ./strcmp.o ./strccmp.o ./strchr.o ./strpbrk.o ./strstr.o ./strcstr.o ./strtok.o ./base64.o ./plerror.o ./plgetopt.o  
ranlib libplc4.a
rm -f libplc4.so
gcc -shared -Wl,-soname -Wl,libplc4.so -o libplc4.so ./plvrsion.o ./strlen.o ./strcpy.o ./strdup.o ./strcat.o ./strcmp.o ./strccmp.o ./strchr.o ./strpbrk.o ./strstr.o ./strcstr.o ./strtok.o ./base64.o ./plerror.o ./plgetopt.o   -L/builds/tinderbox/Tb-Mozilla1.8/Linux_2.4.18-14_Depend/mozilla/dist/lib -lnspr4
collect2: ld terminated with signal 11 [Segmentation fault]

Clobbering doesn't fix it.

Comment 5

[[e-mail] (:bhearsum)]

Attachment: output from rpm -qa | xargs -n1 rpm -V | grep -v ' c ' | grep -v '.pyc$'

Comment 6

[[e-mail] (:bhearsum)]

I'm still investigating this, but do we have a backup of this VM anywhere?

Comment 7

[[e-mail] (:bhearsum)]

I've tried:
* clobbering
* manually compiling (fails in a different place, same ld crash) on a different partition
* rebooting
* doing a 'e2fsck -f' on all of the drives
* re-installing binutils (using the exact same version, of course)

Heading out for the day, will try more tomorrow.

Comment 8

[[e-mail] (:bhearsum)]

Attachment: strace of ld crash (in a different place, this time)

Comment 9

[[e-mail] (:bhearsum)]

the strace crash log showed warnings about -j3. I turned that off and it made no difference.

Comment 10

[[e-mail] (:bhearsum)]

Mrz/Justin, Do we have a backup of this VM?

I also noticed a second copy, from June 9, on d-fcal-build-002. Any chance we can try booting this up?

Comment 11

[[e-mail] (:bhearsum)]

Tried the image on d-fcal-build-002 -- same problems.

Comment 12

[Justin Fitzhugh]

there are no backups - none was ever requested.  whats the action here?

Comment 13

[matthew zeier [:mrz]]

What's the issue, with / or /builds?  

I can create a new / and you can try to copy tools off the old drive to the new.

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

This box is using ccache for some reason, unlike every other linux box still operational. Renaming the cache in /builds/ccache to cache.borked got us past the error creating libplc4.so in nspr. The build continues ...

Comment 15

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Build ran clean, and machine is now reporting "green" on tinderbox. Looks like corrupted ccache was the last problem with crazyhorse. Now very happy to close this bug. :-) 

(While investigating, Justin also fixed some VM config silliness, which we agree would not cause the problems seen above, but were good to fix once discovered.)