531685 - Build problems on linux and win32 due to insufficient disk space

Status: RESOLVED FIXED

(Calendar :: Build Config, defect)

Description

[Stefan Sitter [:ssitter]]

Nightly builds have been missing for some days, e.g. Lightning linux comm-1.9.1 since 25-Nov-2009.

Currently 3 boxes are burning on <http://tinderbox.mozilla.org/showbuilds.cgi?tree=Sunbird> and 6 boxes are burning on <http://tinderbox.mozilla.org/showbuilds.cgi?tree=Calendar1.0>

Error messages indicate that this is caused by insufficient disk space on both linux and win32 build servers.

Comment 1

[Tony Mechelynck [:tonymec]]

In addition, Linux hourlies have now gone missing, see http://ftp.mozilla.org/pub/mozilla.org/calendar/sunbird/tinderbox-builds/comm-1.9.1-linux/ and http://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/tinderbox-builds/comm-1.9.1/linux-xpi-linux/ which ought not to be empty.

Comment 2

[Philippe M. Chiasson (:gozer)]

Side effect of release builds eating up disk space and not releasing it, as we haven't managed to successfully complete the release. I'll have a quick look to see if there are any old, unused leftover build directories I could clean up.

Comment 3

[Philippe M. Chiasson (:gozer)]

Freed up some space on the linux builder, keeping the linux_build results.

Size Used  Free  Use%
30G   23G  5.2G  82% /builds

Hope it's enough.

Comment 4

[Tony Mechelynck [:tonymec]]

(In reply to comment #3)
> Freed up some space on the linux builder, keeping the linux_build results.
> 
> Size Used  Free  Use%
> 30G   23G  5.2G  82% /builds
> 
> Hope it's enough.

Well, it changed nothing about the nightlies (which aren't on the same machine IIUC), but at least the hourlies are indeed back, so "nightly testers" can use that for the time being. People won't all have the same build, but at least severe bugs can (I hope) be crushed in time.

Comment 5

[Philippe M. Chiasson (:gozer)]

(In reply to comment #4)
> (In reply to comment #3)
> > Freed up some space on the linux builder, keeping the linux_build results.
> > 
> > Size Used  Free  Use%
> > 30G   23G  5.2G  82% /builds
> > 
> > Hope it's enough.
> 
> Well, it changed nothing about the nightlies (which aren't on the same machine
> IIUC)

Nightlies are on the same shared machines, and should show up starting with this evenining.

, but at least the hourlies are indeed back, so "nightly testers" can use
> that for the time being. People won't all have the same build, but at least
> severe bugs can (I hope) be crushed in time.

Good, this should normally be a much smaller time window, but getting the first ever Sunbird/Lightninig build thru via release automation is proving to be more time consuming that hoped.

Comment 6

[Tony Mechelynck [:tonymec]]

W32: nightlies built, 2009-12-03 06:57 PST
Mac: nightlies built, 2009-12-03 08:48 PST
Linux: still waiting, 2009-12-04 already.

Comment 7

[Tony Mechelynck [:tonymec]]

Hourlies have stopped being built on all three Tier1 platforms. Latest tinderbox-builds directories:

http://ftp.mozilla.org/pub/mozilla.org/calendar/sunbird/tinderbox-builds/comm-1.9.1-linux/1259853429/ dated 2009-12-03 07:23 PST
http://ftp.mozilla.org/pub/mozilla.org/calendar/sunbird/tinderbox-builds/comm-1.9.1-macosx/1259838000/ dated 2009-12-03 08:48 PST
http://ftp.mozilla.org/pub/mozilla.org/calendar/sunbird/tinderbox-builds/comm-1.9.1-win32/1259859318/ dated 2009-12-03 09:19 PST

http://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/tinderbox-builds/comm-1.9.1/linux-xpi-linux/1259912157/ dated 2009-12-03 23:46 PST
http://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/tinderbox-builds/comm-1.9.1/macosx-xpi-macosx/1259911532/ dated 2009-12-03 23:44 PST
http://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/tinderbox-builds/comm-1.9.1/win32-xpi-win32/1259912402/ dated 2009-12-03 23:58 PST

No comm-central-sunbird-* left for any platform

http://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/tinderbox-builds/comm-central/linux-xpi-linux/1259924402/ dated 2009-12-04 04:28 PST (the most recent one of all ATM)
http://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/tinderbox-builds/comm-central/macosx-xpi-macosx/1259914126/ dated 2009-12-04 00:24 PST
http://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/tinderbox-builds/comm-central/win32-xpi-win32/1259914126/ dated 2009-12-04 00:29 PST

This is no reproach but a status check. Let's hope the release finally happens, and that the builders can be restarted thereafter (possibly after finding some disk space for them somewhere).

Comment 8

[Philippe M. Chiasson (:gozer)]

Looks like this isn't directly related to space issues because of releases. Since the release has been delayed some and will need respinning, I cleaned up the build areas on each calendar build box, so they have plenty of space now. Looks like they are failing to compile alltogether.

make[1]: Leaving directory `/e/buildbot/comm-1.9.1-sunbird-win32-nightly/build/objdir-tb'
make[4]: *** No rule to make target `/e/buildbot/comm-1.9.1-sunbird-win32-nightly/build/calendar/sunbird/branding/nightly/Makefile.in', needed by `calendar/sunbird/branding/nightly/Makefile'.  Stop.
make[3]: *** [export_tier_app] Error 2

for example:

Win32: http://build.mozillamessaging.com/buildbot/production/builders/WINNT%205.2%20comm-1.9.1%20sunbird%20nightly/builds/19/steps/compile/logs/stdio
Mac: http://build.mozillamessaging.com/buildbot/production/builders/MacOSX%2010.5%20comm-1.9.1%20sunbird%20nightly/builds/19/steps/compile/logs/stdio

Comment 9

[Stefan Sitter [:ssitter]]

Yes. Some days later new error Bug 533070 has been introduced by Bug 532656.

Comment 10

[Stefan Sitter [:ssitter]]

I'm resolving this bug as WORKSFORME for now. 
Lightning builds are back and Sunbird is failing due to Bug 533070.

Comment 11

[Stefan Sitter [:ssitter]]

Reopening. Lightning builds for linux and win32 are failing again.

> fatal error LNK1180: insufficient disk space to complete link
http://tinderbox.mozilla.org/showlog.cgi?log=Calendar1.0/1260929681.1260930528.18150.gz#err0