377911 - Tinderboxes don't remove old packages when talkback disabled

Status: VERIFIED FIXED

(Release Engineering :: General, defect, P2)

Description

[Stefan Sitter [:ssitter]]

Since several days the 'Linux lt18-linux-tbox Dep Sb-Release' tinderbox [http://tinderbox.mozilla.org/Sunbird-Mozilla1.8/] fails on hourly builds while the nightly builds seem to work fine. The error messages in the log files look like:

    cvs [checkout aborted]: cannot close 
    mozilla/content/xml/document/test/CVS/Root: No space left on device
or
    ranlib: libgklayout.a: No space left on device
    gmake[4]: *** [libgklayout.a] Error 1

This also makes some l10n tinderboxen fail, e.g. 'Linux lt18-linux-tbox Clbr Sb-Moz1.8-l10n de' tinderbox [http://tinderbox.mozilla.org/Mozilla1.8-l10n-de/].

Expected Results:
Have enough device space on tinderbox to make all the builds succeed.

Since we are short before release of Sunbird 0.5 every tinderbox is needed.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Looks like this directory 
  /builds/tinderbox/Lt-Mozilla1.8/Linux_2.4.21-37.EL_Depend
is filling up with old builds (YYYYMMDDHH/packages). There is 4.9 GB used by dirs going back to 2006030712.

Tinderbox should be autocleaning that. I'll remove all but the directories from April, and investigate.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

This accumulation problem seems to be happening with a number of other boxes:
 * solaria - win32 lightning moz1.8 & xulrunner 1.8.0
 * sb-linux-tbox - linux lightning trunk
 * sb-win32-tbox - win32 lightning trunk
These either have larger drives or less builds so it's not been a problem yet.

The Sunbird builds on the same boxes aren't showing the same buildup, so it must be a config issue. This is the code that isn't being hit
 http://lxr.mozilla.org/mozilla/source/tools/tinderbox/post-mozilla-rel.pl#1352

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: patch

There's not much point conditionally removing the builds if we always save them. Not sure about the side affects here, although it looks like we're already removing packages and symbols on the release tbox with this code.

For the record, Chase changed this in rev 1.83

Comment 4

[Chris Cooper [:coop] (he/him)]

Since *all* the Sunbird VMs seem to be affected, can we establish why $cachebuild isn't being set correctly (or being set to 0) and hence not triggering this code for only this product?

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 262031 [details] [diff] [review]
patch

It turns out that cachebuild is being set, and the find call is being made. Eg:

sb-win32-tbox Depend Lt-Nightly
http://tinderbox.mozilla.org/showlog.cgi?log=Sunbird/1180605840.32256.gz&fulltext=1
...
find d:/builds/tinderbox/Lightning-Trunk/WINNT_5.2_Depend -type d -name "200*-*" -maxdepth 1 -prune -mtime +7 -print | xargs rm -rf
...

It just doesn't do anything useful. I get warning when I try it on the prompt, complaining about the order of arguments/options, so this could be an issue with newer versions of find.

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

It all depends on Talkback, yay! If it's enabled, then this happens:      (Firefox Mozilla1.8 win32 log)

Symbol transfer completed.
Cleaning up...
Remove any folders older than 7 days...
/bin/find `cygpath -u /cygdrive/c/builds/tinderbox/Fx-Mozilla1.8/WINNT_5.2_Depend/mozilla/..` -type d -name "200???????" -prune -mtime +7 -maxdepth 1 -print | xargs rm -rf
/bin/find `cygpath -u /cygdrive/c/builds/tinderbox/Fx-Mozilla1.8/WINNT_5.2_Depend/mozilla/..` -type f -name "200???????.tar.bz2" -prune -mtime +7 -maxdepth 1 -print | xargs rm -rf
make: Leaving directory `/cygdrive/c/builds/tinderbox/Fx-Mozilla1.8/WINNT_5.2_Depend/mozilla/fullsoft'

It's the first call to find doing the work. 

If talkback isn't enabled then there is only the call in comment #5, and the name argument isn't going to match the dirs we'd like to remove (no - in the name). Should we tweak that argument to fix this ?

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Prevent non-talkback builds from accumulating

I really hope we aren't using tinderbox in 2010, but just in case.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

I meant to say that I can't find any directories in the 200*-* style in the likes of pacifica-vm:/cygdrive/c/builds/tinderbox/Fx-Mozilla1.8/WINNT_5.2_Depend ($mozilla_build_dir), and checked about half a dozen tinderboxes of varying age.

Comment 9

[J. Paul Reed [:preed]]

Sorry for the bugspam; these are now P2 in the New View of the World (tm).

Comment 10

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Prevent non-talkback builds from accumulating - v2

This is the same find command as the previous patch, but the argument order is shuffled to avoid this warning from Cygwin find 4.2.27:

find: warning: you have specified -maxdepth option after a non-option argument -type, but options are not positional (-maxdepth affects tests specified before it as well as those specified after it). Please specify options before other arguments.

So now they are in the order 
 options (maxdepth), tests (type, name, mtime), actions (prune, print).

I'd like to fix this bug soon, as it affects the Firefox trunk since Talkback was disabled, and other apps are moving that way too.

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

Checking in post-mozilla-rel.pl;
/cvsroot/mozilla/tools/tinderbox/post-mozilla-rel.pl,v  <--  post-mozilla-rel.pl
new revision: 1.124; previous revision: 1.123
done

Will verify after tomorrow's nightlies (fx-linux-tbox has packages going back to 20070624).

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

Verified on fx-linux-tbox and lt-linux-tbox that only a weeks worth of packages are present. The latter has 5.8 GB in use & 5.3 GB free on the /builds partition.