Build problems on linux and win32 due to insufficient disk space

RESOLVED FIXED

Status

Calendar
Build Config
--
major
RESOLVED FIXED
8 years ago
6 years ago

People

(Reporter: Stefan Sitter, Assigned: gozer)

Tracking

Details

(Reporter)

Description

8 years ago
Nightly builds have been missing for some days, e.g. Lightning linux comm-1.9.1 since 25-Nov-2009.

Currently 3 boxes are burning on <http://tinderbox.mozilla.org/showbuilds.cgi?tree=Sunbird> and 6 boxes are burning on <http://tinderbox.mozilla.org/showbuilds.cgi?tree=Calendar1.0>

Error messages indicate that this is caused by insufficient disk space on both linux and win32 build servers.
In addition, Linux hourlies have now gone missing, see http://ftp.mozilla.org/pub/mozilla.org/calendar/sunbird/tinderbox-builds/comm-1.9.1-linux/ and http://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/tinderbox-builds/comm-1.9.1/linux-xpi-linux/ which ought not to be empty.
(Assignee)

Comment 2

8 years ago
Side effect of release builds eating up disk space and not releasing it, as we haven't managed to successfully complete the release. I'll have a quick look to see if there are any old, unused leftover build directories I could clean up.
(Assignee)

Comment 3

8 years ago
Freed up some space on the linux builder, keeping the linux_build results.

Size Used  Free  Use%
30G   23G  5.2G  82% /builds

Hope it's enough.
(In reply to comment #3)
> Freed up some space on the linux builder, keeping the linux_build results.
> 
> Size Used  Free  Use%
> 30G   23G  5.2G  82% /builds
> 
> Hope it's enough.

Well, it changed nothing about the nightlies (which aren't on the same machine IIUC), but at least the hourlies are indeed back, so "nightly testers" can use that for the time being. People won't all have the same build, but at least severe bugs can (I hope) be crushed in time.
(Assignee)

Comment 5

8 years ago
(In reply to comment #4)
> (In reply to comment #3)
> > Freed up some space on the linux builder, keeping the linux_build results.
> > 
> > Size Used  Free  Use%
> > 30G   23G  5.2G  82% /builds
> > 
> > Hope it's enough.
> 
> Well, it changed nothing about the nightlies (which aren't on the same machine
> IIUC)

Nightlies are on the same shared machines, and should show up starting with this evenining.

, but at least the hourlies are indeed back, so "nightly testers" can use
> that for the time being. People won't all have the same build, but at least
> severe bugs can (I hope) be crushed in time.

Good, this should normally be a much smaller time window, but getting the first ever Sunbird/Lightninig build thru via release automation is proving to be more time consuming that hoped.
W32: nightlies built, 2009-12-03 06:57 PST
Mac: nightlies built, 2009-12-03 08:48 PST
Linux: still waiting, 2009-12-04 already.
OS: All → Linux
Hardware: All → x86
Whiteboard: [fixed except Linux]
Hourlies have stopped being built on all three Tier1 platforms. Latest tinderbox-builds directories:

http://ftp.mozilla.org/pub/mozilla.org/calendar/sunbird/tinderbox-builds/comm-1.9.1-linux/1259853429/ dated 2009-12-03 07:23 PST
http://ftp.mozilla.org/pub/mozilla.org/calendar/sunbird/tinderbox-builds/comm-1.9.1-macosx/1259838000/ dated 2009-12-03 08:48 PST
http://ftp.mozilla.org/pub/mozilla.org/calendar/sunbird/tinderbox-builds/comm-1.9.1-win32/1259859318/ dated 2009-12-03 09:19 PST

http://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/tinderbox-builds/comm-1.9.1/linux-xpi-linux/1259912157/ dated 2009-12-03 23:46 PST
http://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/tinderbox-builds/comm-1.9.1/macosx-xpi-macosx/1259911532/ dated 2009-12-03 23:44 PST
http://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/tinderbox-builds/comm-1.9.1/win32-xpi-win32/1259912402/ dated 2009-12-03 23:58 PST

No comm-central-sunbird-* left for any platform

http://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/tinderbox-builds/comm-central/linux-xpi-linux/1259924402/ dated 2009-12-04 04:28 PST (the most recent one of all ATM)
http://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/tinderbox-builds/comm-central/macosx-xpi-macosx/1259914126/ dated 2009-12-04 00:24 PST
http://ftp.mozilla.org/pub/mozilla.org/calendar/lightning/tinderbox-builds/comm-central/win32-xpi-win32/1259914126/ dated 2009-12-04 00:29 PST

This is no reproach but a status check. Let's hope the release finally happens, and that the builders can be restarted thereafter (possibly after finding some disk space for them somewhere).
(Assignee)

Comment 8

8 years ago
Looks like this isn't directly related to space issues because of releases. Since the release has been delayed some and will need respinning, I cleaned up the build areas on each calendar build box, so they have plenty of space now. Looks like they are failing to compile alltogether.

make[1]: Leaving directory `/e/buildbot/comm-1.9.1-sunbird-win32-nightly/build/objdir-tb'
make[4]: *** No rule to make target `/e/buildbot/comm-1.9.1-sunbird-win32-nightly/build/calendar/sunbird/branding/nightly/Makefile.in', needed by `calendar/sunbird/branding/nightly/Makefile'.  Stop.
make[3]: *** [export_tier_app] Error 2

for example:

Win32: http://build.mozillamessaging.com/buildbot/production/builders/WINNT%205.2%20comm-1.9.1%20sunbird%20nightly/builds/19/steps/compile/logs/stdio
Mac: http://build.mozillamessaging.com/buildbot/production/builders/MacOSX%2010.5%20comm-1.9.1%20sunbird%20nightly/builds/19/steps/compile/logs/stdio
(Reporter)

Comment 9

8 years ago
Yes. Some days later new error Bug 533070 has been introduced by Bug 532656.
(Reporter)

Comment 10

8 years ago
I'm resolving this bug as WORKSFORME for now. 
Lightning builds are back and Sunbird is failing due to Bug 533070.
Status: NEW → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → WORKSFORME
(Reporter)

Comment 11

8 years ago
Reopening. Lightning builds for linux and win32 are failing again.

> fatal error LNK1180: insufficient disk space to complete link
http://tinderbox.mozilla.org/showlog.cgi?log=Calendar1.0/1260929681.1260930528.18150.gz#err0
Status: RESOLVED → REOPENED
Resolution: WORKSFORME → ---
Whiteboard: [fixed except Linux]
(Assignee)

Updated

8 years ago
Status: REOPENED → RESOLVED
Last Resolved: 8 years ago8 years ago
Resolution: --- → FIXED
OS: Linux → All
Hardware: x86 → All

Updated

6 years ago
Assignee: nobody → gozer
You need to log in before you can comment on or make changes to this bug.