Closed Bug 531685 Opened 11 years ago Closed 11 years ago

Build problems on linux and win32 due to insufficient disk space

Categories

(Calendar :: Build Config, defect)

defect
Not set
major

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: ssitter, Assigned: gozer)

Details

Nightly builds have been missing for some days, e.g. Lightning linux comm-1.9.1 since 25-Nov-2009.

Currently 3 boxes are burning on <http://tinderbox.mozilla.org/showbuilds.cgi?tree=Sunbird> and 6 boxes are burning on <http://tinderbox.mozilla.org/showbuilds.cgi?tree=Calendar1.0>

Error messages indicate that this is caused by insufficient disk space on both linux and win32 build servers.
Side effect of release builds eating up disk space and not releasing it, as we haven't managed to successfully complete the release. I'll have a quick look to see if there are any old, unused leftover build directories I could clean up.
Freed up some space on the linux builder, keeping the linux_build results.

Size Used  Free  Use%
30G   23G  5.2G  82% /builds

Hope it's enough.
(In reply to comment #3)
> Freed up some space on the linux builder, keeping the linux_build results.
> 
> Size Used  Free  Use%
> 30G   23G  5.2G  82% /builds
> 
> Hope it's enough.

Well, it changed nothing about the nightlies (which aren't on the same machine IIUC), but at least the hourlies are indeed back, so "nightly testers" can use that for the time being. People won't all have the same build, but at least severe bugs can (I hope) be crushed in time.
(In reply to comment #4)
> (In reply to comment #3)
> > Freed up some space on the linux builder, keeping the linux_build results.
> > 
> > Size Used  Free  Use%
> > 30G   23G  5.2G  82% /builds
> > 
> > Hope it's enough.
> 
> Well, it changed nothing about the nightlies (which aren't on the same machine
> IIUC)

Nightlies are on the same shared machines, and should show up starting with this evenining.

, but at least the hourlies are indeed back, so "nightly testers" can use
> that for the time being. People won't all have the same build, but at least
> severe bugs can (I hope) be crushed in time.

Good, this should normally be a much smaller time window, but getting the first ever Sunbird/Lightninig build thru via release automation is proving to be more time consuming that hoped.
W32: nightlies built, 2009-12-03 06:57 PST
Mac: nightlies built, 2009-12-03 08:48 PST
Linux: still waiting, 2009-12-04 already.
OS: All → Linux
Hardware: All → x86
Whiteboard: [fixed except Linux]
Hourlies have stopped being built on all three Tier1 platforms. Latest tinderbox-builds directories:

http://ftp.mozilla.org/pub/mozilla.org/calendar/sunbird/tinderbox-builds/comm-1.9.1-linux/1259853429/ dated 2009-12-03 07:23 PST
http://ftp.mozilla.org/pub/mozilla.org/calendar/sunbird/tinderbox-builds/comm-1.9.1-macosx/1259838000/ dated 2009-12-03 08:48 PST
http://ftp.mozilla.org/pub/mozilla.org/calendar/sunbird/tinderbox-builds/comm-1.9.1-win32/1259859318/ dated 2009-12-03 09:19 PST

http://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/tinderbox-builds/comm-1.9.1/linux-xpi-linux/1259912157/ dated 2009-12-03 23:46 PST
http://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/tinderbox-builds/comm-1.9.1/macosx-xpi-macosx/1259911532/ dated 2009-12-03 23:44 PST
http://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/tinderbox-builds/comm-1.9.1/win32-xpi-win32/1259912402/ dated 2009-12-03 23:58 PST

No comm-central-sunbird-* left for any platform

http://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/tinderbox-builds/comm-central/linux-xpi-linux/1259924402/ dated 2009-12-04 04:28 PST (the most recent one of all ATM)
http://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/tinderbox-builds/comm-central/macosx-xpi-macosx/1259914126/ dated 2009-12-04 00:24 PST
http://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/tinderbox-builds/comm-central/win32-xpi-win32/1259914126/ dated 2009-12-04 00:29 PST

This is no reproach but a status check. Let's hope the release finally happens, and that the builders can be restarted thereafter (possibly after finding some disk space for them somewhere).
Looks like this isn't directly related to space issues because of releases. Since the release has been delayed some and will need respinning, I cleaned up the build areas on each calendar build box, so they have plenty of space now. Looks like they are failing to compile alltogether.

make[1]: Leaving directory `/e/buildbot/comm-1.9.1-sunbird-win32-nightly/build/objdir-tb'
make[4]: *** No rule to make target `/e/buildbot/comm-1.9.1-sunbird-win32-nightly/build/calendar/sunbird/branding/nightly/Makefile.in', needed by `calendar/sunbird/branding/nightly/Makefile'.  Stop.
make[3]: *** [export_tier_app] Error 2

for example:

Win32: http://build.mozillamessaging.com/buildbot/production/builders/WINNT%205.2%20comm-1.9.1%20sunbird%20nightly/builds/19/steps/compile/logs/stdio
Mac: http://build.mozillamessaging.com/buildbot/production/builders/MacOSX%2010.5%20comm-1.9.1%20sunbird%20nightly/builds/19/steps/compile/logs/stdio
Yes. Some days later new error Bug 533070 has been introduced by Bug 532656.
I'm resolving this bug as WORKSFORME for now. 
Lightning builds are back and Sunbird is failing due to Bug 533070.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → WORKSFORME
Reopening. Lightning builds for linux and win32 are failing again.

> fatal error LNK1180: insufficient disk space to complete link
http://tinderbox.mozilla.org/showlog.cgi?log=Calendar1.0/1260929681.1260930528.18150.gz#err0
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---
Whiteboard: [fixed except Linux]
Status: REOPENED → RESOLVED
Closed: 11 years ago11 years ago
Resolution: --- → FIXED
OS: Linux → All
Hardware: x86 → All
Assignee: nobody → gozer
You need to log in before you can comment on or make changes to this bug.