The default bug view has changed. See this FAQ.

mozilla-central OS X builders out of disk space

RESOLVED FIXED

Status

Release Engineering
Other
--
blocker
RESOLVED FIXED
9 years ago
4 years ago

People

(Reporter: ted, Assigned: nthomas)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(URL)

(Reporter)

Description

9 years ago
ar cr libgkconxultmpl_s.a nsContentSupportMap.o nsContentTestNode.o nsInstantiationNode.o nsTreeRows.o nsRDFConInstanceTestNode.o nsRDFConMemberTestNode.o nsRDFPropertyTestNode.o nsRDFBinding.o nsRDFQuery.o nsResourceSet.o nsRuleNetwork.o nsTemplateMatch.o nsTemplateRule.o nsXULContentBuilder.o nsXULContentUtils.o nsXULTreeBuilder.o nsXULSortService.o nsXULTemplateBuilder.o nsXULTemplateQueryProcessorRDF.o nsXULTemplateResultRDF.o nsXULTemplateResultSetRDF.o nsXMLBinding.o nsXULTemplateQueryProcessorXML.o nsXULTemplateResultXML.o nsXULTemplateQueryProcessorStorage.o nsXULTemplateResultStorage.o  
ar: libgkconxultmpl_s.a: No space left on device

I think we need some cleanup or something here.
So it went green for one cycle (thought it was fixed, and landed stuff), and now it's hit the same problem again.
This happened two cycles in a row now; I think that means it holds the tree closed.
Severity: normal → blocker
And now it succeeded again; not sure if that's because something was cleaned up or if it's just random.
Severity: blocker → critical

Comment 4

9 years ago
Disk space issues fall outside of the scope of IT's ability to support build machines, since we have no idea what is safe to remove or not.  Passing over to release engineering to handle.
Assignee: server-ops → nobody
Component: Server Operations: Tinderbox Maintenance → Release Engineering: Maintenance
QA Contact: mrz → release
Two red in a row again -> tree effectively closed.
Severity: critical → blocker
(Reporter)

Comment 6

9 years ago
there's more than one build slave reporting to this column, which is why this looks intermittent. 
(Assignee)

Comment 7

9 years ago
Pulling bm-xserve16 from the slave pool for the moment.
Assignee: nobody → nthomas
(Assignee)

Comment 8

9 years ago
Rehabilitated by cleaning up intermediate files from 3.1a1 build2:
  cd /builds/releases/firefox-3.1a1-build2/build/mozilla-central/fx-objdir
  find -E . -iregex '.*\.(i|s|mii|ii)$' -exec rm -v {} \;

Back in the pool now. (actually a couple of hours ago)
Status: NEW → RESOLVED
Last Resolved: 9 years ago
Resolution: --- → FIXED
Looks like it ran out of space again. Part of the problem here is that if a nightly build fails it doesn't clean up after itself, which ends up using up to 15G of space that isn't normally used. Getting https://bugzilla.mozilla.org/show_bug.cgi?id=421534 resolved will help this a lot, in the meantime maybe we can get some additional space on these xserves? Our current clean-up job, shown ini comment#8 doesn't get all of the files -save-temps creates, maybe we can improve that a bit.

To fix bm-xserve16 for now, I cleaned out mozilla-central-macosx-nightly/build and actionmonkey-macosx-nightly/build, which freed up 20G. bm-xserve17 has about 25G free. I looked around and noticed that bm-xserve17 had its printer drivers deleted to free up almost 4G. I did the same on 16, 18, and 19.

Updated

8 years ago
Component: Release Engineering: Maintenance → Release Engineering
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.