448115 - mozilla-central OS X builders out of disk space

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[(not currently active) Ted Mielczarek]

ar cr libgkconxultmpl_s.a nsContentSupportMap.o nsContentTestNode.o nsInstantiationNode.o nsTreeRows.o nsRDFConInstanceTestNode.o nsRDFConMemberTestNode.o nsRDFPropertyTestNode.o nsRDFBinding.o nsRDFQuery.o nsResourceSet.o nsRuleNetwork.o nsTemplateMatch.o nsTemplateRule.o nsXULContentBuilder.o nsXULContentUtils.o nsXULTreeBuilder.o nsXULSortService.o nsXULTemplateBuilder.o nsXULTemplateQueryProcessorRDF.o nsXULTemplateResultRDF.o nsXULTemplateResultSetRDF.o nsXMLBinding.o nsXULTemplateQueryProcessorXML.o nsXULTemplateResultXML.o nsXULTemplateQueryProcessorStorage.o nsXULTemplateResultStorage.o  
ar: libgkconxultmpl_s.a: No space left on device

I think we need some cleanup or something here.

Comment 1

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

So it went green for one cycle (thought it was fixed, and landed stuff), and now it's hit the same problem again.

Comment 2

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

This happened two cycles in a row now; I think that means it holds the tree closed.

Comment 3

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

And now it succeeded again; not sure if that's because something was cleaned up or if it's just random.

Comment 4

[Mark Smith [:xb95]]

Disk space issues fall outside of the scope of IT's ability to support build machines, since we have no idea what is safe to remove or not.  Passing over to release engineering to handle.

Comment 5

[David Baron :dbaron: (⌚️UTC-4, no longer working on Mozilla)]

Two red in a row again -> tree effectively closed.

Comment 6

[(not currently active) Ted Mielczarek]

there's more than one build slave reporting to this column, which is why this looks intermittent. 

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Pulling bm-xserve16 from the slave pool for the moment.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Rehabilitated by cleaning up intermediate files from 3.1a1 build2:
  cd /builds/releases/firefox-3.1a1-build2/build/mozilla-central/fx-objdir
  find -E . -iregex '.*\.(i|s|mii|ii)$' -exec rm -v {} \;

Back in the pool now. (actually a couple of hours ago)

Comment 9

[[e-mail] (:bhearsum)]

Looks like it ran out of space again. Part of the problem here is that if a nightly build fails it doesn't clean up after itself, which ends up using up to 15G of space that isn't normally used. Getting https://bugzilla.mozilla.org/show_bug.cgi?id=421534 resolved will help this a lot, in the meantime maybe we can get some additional space on these xserves? Our current clean-up job, shown ini comment#8 doesn't get all of the files -save-temps creates, maybe we can improve that a bit.

To fix bm-xserve16 for now, I cleaned out mozilla-central-macosx-nightly/build and actionmonkey-macosx-nightly/build, which freed up 20G. bm-xserve17 has about 25G free. I looked around and noticed that bm-xserve17 had its printer drivers deleted to free up almost 4G. I did the same on 16, 18, and 19.