585365 - Some of the L10n win repack build slaves are failing

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Vlado Valastiak [:wladow]]

Some of the L10n win repack build slaves are failing with:

...
./allmakefiles.sh: line 114: ./toolkit/toolkit-makefiles.sh: No such file or directory
program finished with exit code 1

see http://tinderbox.mozilla.org/showlog.cgi?log=Mozilla-l10n/1281187619.1281187752.22270.gz and many more reports at http://tinderbox.mozilla.org/Mozilla-l10n/

Reading through the todays repacks run these slaves seem to be affected:
win32-slave12
win32-slave13
win32-slave19

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

I've set these machines to clobber. 

coop/armen, could we try to figure out why this is happening ? Or figure out a way to clobber on the first nightly build a slave does ? Maybe just '-t 18' instead of '-t 168' for the periodic clobber timeout.

Comment 2

[Vlado Valastiak [:wladow]]

Today's nightly failed again with the same error:
http://tinderbox.mozilla.org/showlog.cgi?log=Mozilla-l10n-sk/1281360117.1281360271.25173.gz

Comment 3

[Armen [:armenzg]]

I will check what is going on in the morning.

Comment 4

[Armen [:armenzg]]

This seems to be happening with the slaves you mention.
I have put them on the side until I look further into it.

I can't reach at the moment moz2-win32-slave13 but let's leave it in the pool and see what happens.

Comment 5

[Armen [:armenzg]]

OK so we are having trouble with moz2-win32-slave[12,13,16,19]

Not sure if clobberer works properly with L10n nightly builds (I don't know the system myself). It might not work because the slave returns None as the last clobbered date:
mozilla-central-win32-nightly:Our last clobber date:  None
mozilla-central-win32-nightly:Server clobber date:    2010-08-09 12:54:58

We also run out of space in one of the slaves when checking out after having 2.06 GB of space available. I will post a couple of patches to address that.

I will clobber manually the mentioned slaves.

NOTE: This is the issue we are hitting:
./allmakefiles.sh: line 114: ./toolkit/toolkit-makefiles.sh: No such file or directory

Comment 6

[Armen [:armenzg]]

Attachment: Default l10n space variable

Comment 7

[Armen [:armenzg]]

Attachment: make use of the l10n space variable

Comment 8

[Armen [:armenzg]]

Attachment: make use of the l10n space variable

This patch is the correct one.

Comment 9

[Armen [:armenzg]]

(In reply to comment #6)
> Created attachment 464411 [details] [diff] [review]
> Default l10n space variable
To note I am increasing it from 2GB to 3GB.

Comment 10

[Armen [:armenzg]]

(In reply to comment #5) 
> Not sure if clobberer works properly with L10n nightly builds (I don't know the
> system myself). It might not work because the slave returns None as the last
> clobbered date:
> mozilla-central-win32-nightly:Our last clobber date:  None
> mozilla-central-win32-nightly:Server clobber date:    2010-08-09 12:54:58
This is invalid as it has nothing to do with L10n. Please ignore.

Comment 11

[Armen [:armenzg]]

I have clobbered manually the mentioned machines and hopefully I have not missed any. The slaves are back in the pool.
I will check in the morning.

Comment 12

[Vlado Valastiak [:wladow]]

Looking at todays tinderbox waterfall the slave05 might need a clobber as well

Comment 13

[Armen [:armenzg]]

Right. Done.

Comment 14

[Armen [:armenzg]]

Changed reviewer since coop is away until next week.

There were only 2 repacks that failed today and they seems to be valid failures (ca and rm).

I am still not sure why clobberer did not work the first time. Let me give a try.

Comment 15

[Armen [:armenzg]]

Well it seems that clobber does work.
So let's hope that removing 2GB instead of 2GB will prevent us from getting into this state again.

mozilla-central-win32-l10n-nightly:Our last clobber date:  2010-08-09 12:54:58
mozilla-central-win32-l10n-nightly:Server clobber date:    2010-08-11 12:37:10
mozilla-central-win32-l10n-nightly:Server is forcing a clobber, initiated by [e-mail]
mozilla-central-win32-l10n-nightly:Clobbering...

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

Armen, I don't see anything here that indicates why increasing the allocated space to 3G will help. Could you fill me in ? We've had concerns in the past about the bash code we use to pulling mercurial not being robust enough, I figured that was the issue here.

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 464411 [details] [diff] [review]
Default l10n space variable

Ah, you did find one instance of running out of space.

Comment 18

[Armen [:armenzg]]

Yeah on comment#5.

We should separately fix bug 554438.

Comment 19

[Armen [:armenzg]]

Comment on attachment 464411 [details] [diff] [review]
Default l10n space variable

http://hg.mozilla.org/build/buildbot-configs/rev/b8473368a7d0

Comment 20

[Armen [:armenzg]]

Comment on attachment 464417 [details] [diff] [review]
make use of the l10n space variable

http://hg.mozilla.org/build/buildbotcustom/rev/e17933bb8fe7

Comment 21

[Armen [:armenzg]]

This has also landed today which should help avoid other L10n-broken scenarios:
http://hg.mozilla.org/build/buildbotcustom/rev/91f1cc47614b