795776 - b2g builds don't check for clobber or ensure free space before starting compile

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Nick Thomas [:nthomas] (UTC+12)]

eg bld-linux64-ec2-051 failed a b2g_services-central_panda_dep build starting at 2012-09-30 18:57:47, running out of space while packaging. Subsequently failed 7 non-mozharness jobs in cloning build tools or setting a toolsdir property, and then magically had space free to save itself.

Buildbot factory jobs make a call to tools/buildfarm/maintenance/purge_builds.py. Looks like we don't check for a clobber either.

Comment 2

[(no longer active)]

(In reply to comment #1)
> *** Bug 795781 has been marked as a duplicate of this bug. ***
> 
> Referenced Bugs:
> 
> https://bugzilla.mozilla.org/show_bug.cgi?id=795781
> [Bug 795781] "No space left on device" on some builds

FWIW not all of those failures were on b2g.

Comment 3

[Phil Ringnalda (:philor)]

Ehsan's comment that inline-history ate was trying to point out that his dupe was about things other than b2g builds, but the problem here is not just that the b2g build fails, it's that it fails by using up all available space, and then the next job on that slave, Linux desktop or Android or whatever, can't even get started far enough to purge, which only takes it a second or two, and then it takes the next Linux desktop job and breaks it, and then all the Android jobs and breaks them, letting a single slave eat every single mock job on a push in just a few seconds.

Comment 4

[(no longer active)]

(In reply to Phil Ringnalda (:philor) from comment #3)
> Ehsan's comment that inline-history ate

Shoot!  Filed bug 796043 about that!

Comment 5

[Chris AtLee [:catlee]]

Attachment: purging support

will attempt clobberer next

Comment 6

[Aki Sasaki (not active)]

Comment on attachment 666784 [details] [diff] [review]
purging support

>+external_tools_path = os.path.join(
>+    os.path.abspath(os.path.dirname(os.path.dirname(mozharness.__file__))),
>+    'external_tools',
>+)

Nice, I was going to go by the script path, but this looks better.

Comment 7

[Chris AtLee [:catlee]]

Attachment: clobberer support

Comment 8

[Aki Sasaki (not active)]

Comment on attachment 671481 [details] [diff] [review]
clobberer support

>+    def clobberer(self):
>+        c = self.config
>+        assert self.buildbot_config

I'd prefer a self.fatal() here for consistency.