Last Comment Bug 448115 - mozilla-central OS X builders out of disk space
: mozilla-central OS X builders out of disk space
Product: Release Engineering
Classification: Other
Component: Other (show other bugs)
: other
: x86 Mac OS X
-- blocker (vote)
: ---
Assigned To: Nick Thomas [:nthomas]
Depends on:
  Show dependency treegraph
Reported: 2008-07-26 07:56 PDT by Ted Mielczarek [:ted.mielczarek]
Modified: 2013-08-12 21:54 PDT (History)
5 users (show)
See Also:
Crash Signature:
QA Whiteboard:
Iteration: ---
Points: ---


Description User image Ted Mielczarek [:ted.mielczarek] 2008-07-26 07:56:12 PDT
ar cr libgkconxultmpl_s.a nsContentSupportMap.o nsContentTestNode.o nsInstantiationNode.o nsTreeRows.o nsRDFConInstanceTestNode.o nsRDFConMemberTestNode.o nsRDFPropertyTestNode.o nsRDFBinding.o nsRDFQuery.o nsResourceSet.o nsRuleNetwork.o nsTemplateMatch.o nsTemplateRule.o nsXULContentBuilder.o nsXULContentUtils.o nsXULTreeBuilder.o nsXULSortService.o nsXULTemplateBuilder.o nsXULTemplateQueryProcessorRDF.o nsXULTemplateResultRDF.o nsXULTemplateResultSetRDF.o nsXMLBinding.o nsXULTemplateQueryProcessorXML.o nsXULTemplateResultXML.o nsXULTemplateQueryProcessorStorage.o nsXULTemplateResultStorage.o  
ar: libgkconxultmpl_s.a: No space left on device

I think we need some cleanup or something here.
Comment 1 User image David Baron :dbaron: ⌚️UTC-8 2008-07-26 09:29:43 PDT
So it went green for one cycle (thought it was fixed, and landed stuff), and now it's hit the same problem again.
Comment 2 User image David Baron :dbaron: ⌚️UTC-8 2008-07-26 10:42:42 PDT
This happened two cycles in a row now; I think that means it holds the tree closed.
Comment 3 User image David Baron :dbaron: ⌚️UTC-8 2008-07-26 11:50:53 PDT
And now it succeeded again; not sure if that's because something was cleaned up or if it's just random.
Comment 4 User image Mark Smith [:xb95] 2008-07-26 12:00:53 PDT
Disk space issues fall outside of the scope of IT's ability to support build machines, since we have no idea what is safe to remove or not.  Passing over to release engineering to handle.
Comment 5 User image David Baron :dbaron: ⌚️UTC-8 2008-07-26 13:47:39 PDT
Two red in a row again -> tree effectively closed.
Comment 6 User image Ted Mielczarek [:ted.mielczarek] 2008-07-26 16:57:30 PDT
there's more than one build slave reporting to this column, which is why this looks intermittent. 
Comment 7 User image Nick Thomas [:nthomas] 2008-07-27 00:51:55 PDT
Pulling bm-xserve16 from the slave pool for the moment.
Comment 8 User image Nick Thomas [:nthomas] 2008-07-27 04:25:26 PDT
Rehabilitated by cleaning up intermediate files from 3.1a1 build2:
  cd /builds/releases/firefox-3.1a1-build2/build/mozilla-central/fx-objdir
  find -E . -iregex '.*\.(i|s|mii|ii)$' -exec rm -v {} \;

Back in the pool now. (actually a couple of hours ago)
Comment 9 User image Ben Hearsum (:bhearsum) 2008-07-28 05:04:37 PDT
Looks like it ran out of space again. Part of the problem here is that if a nightly build fails it doesn't clean up after itself, which ends up using up to 15G of space that isn't normally used. Getting resolved will help this a lot, in the meantime maybe we can get some additional space on these xserves? Our current clean-up job, shown ini comment#8 doesn't get all of the files -save-temps creates, maybe we can improve that a bit.

To fix bm-xserve16 for now, I cleaned out mozilla-central-macosx-nightly/build and actionmonkey-macosx-nightly/build, which freed up 20G. bm-xserve17 has about 25G free. I looked around and noticed that bm-xserve17 had its printer drivers deleted to free up almost 4G. I did the same on 16, 18, and 19.

Note You need to log in before you can comment on or make changes to this bug.