1035609 - Clobber b2g hazard build before running

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Steve Fink [:sfink] [:s:]]

Running out of space.

Comment 1

[Steve Fink [:sfink] [:s:]]

Attachment: Clobber b2g hazard build before running

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

This is causing quite a lot of slaves to run out of disk space, resulting in (at least) a few burnt builds in each case. Can we get a fix in on Tuesday ?

Comment 3

[Steve Fink [:sfink] [:s:]]

(In reply to Nick Thomas [:nthomas] from comment #2)
> This is causing quite a lot of slaves to run out of disk space, resulting in
> (at least) a few burnt builds in each case. Can we get a fix in on Tuesday ?

Yes, sorry. I recognize the severity of this, since I did the same d*mn thing with the desktop builds initially. You'd think I'd learn...

(I think the problem is that the purge is really annoying for testing on a private slave, since it tends to wipe out all the artifacts of the other stuff you're testing. So I disable it "temporarily".)

Comment 4

[Steve Fink [:sfink] [:s:]]

Attachment: Clobber b2g hazard build before running

Now with 100% more testing.

I tried doing this in buildb2gbase, but the scripts have different notions of what upload dirs to always clobber.

Comment 5

[Steve Fink [:sfink] [:s:]]

Thanks, pushed to default branch at https://hg.mozilla.org/build/mozharness/rev/c75efd186ebf

Comment 6

[Steve Fink [:sfink] [:s:]]

Triggered at build at https://tbpl.mozilla.org/?tree=Ash&rev=620766b18864&jobname=b2g-haz

I see it already plowed over the ruined carcasses of 4 out of space slaves. I can't access the build log until it's done, but someone could get it from the buildmaster to verify that it's clobbering if they wanted to merge it to production sooner.

Comment 7

[Steve Fink [:sfink] [:s:]]

The pile of corpses has reached 23 slaves now. Seems like tbpl should be showing the purple miasma. Merged to production 'cause rail said so.

https://hg.mozilla.org/build/mozharness/rev/f9886b14df1c

Comment 8

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Are these some new fallout from this?
https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=bld-linux64-spot-110
https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=bld-linux64-spot-111
https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=bld-linux64-spot-115

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

I suspect not, because these slaves can't make a connection to hg.mozilla.org despite reboots.

Comment 10

[Steve Fink [:sfink] [:s:]]

To the extent that I can access those urls, it looks like things have been worked out now?

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

bug 1036176 tracks the hg issue. I think we're done with this bug.