607940 - builds should halt most any pre-build steps fail

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[[e-mail] (:bhearsum)]

In bug 607928 someone reported a build that he thought failed in count_and_reboot.py. There was actually quite a few things that went wrong prior to that:
- Failed to completely, remove previous tools directory because of a read-only filesystem
- Failed to clone new tools directory
- Failed to run clobberer
- Failed to run purge_builds

The count_and_reboot failure is the machine being and unable to run the reboot script, because it doesn't exist.

All but the clobberer failure, and possibly it too, should've halted the build, turned it purple, and marked it as needing to be retried.

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #1)
> In bug 607928 someone reported a build that he thought failed in
> count_and_reboot.py. There was actually quite a few things that went wrong
> prior to that:
> - Failed to completely, remove previous tools directory because of a read-only
> filesystem
> - Failed to clone new tools directory
> - Failed to run clobberer
> - Failed to run purge_builds
> 
> The count_and_reboot failure is the machine being and unable to run the reboot
> script, because it doesn't exist.
This feels like something that puppet should deploy on all slaves for us?



> All but the clobberer failure, and possibly it too, should've halted the build,
> turned it purple, and marked it as needing to be retried.

Comment 2

[Chris Cooper [:coop] (he/him)]

(In reply to Ben Hearsum [:bhearsum] from comment #0)
> - Failed to completely, remove previous tools directory because of a
> read-only filesystem
> - Failed to clone new tools directory

bug 712205

> - Failed to run clobberer
> - Failed to run purge_builds

Do we do this already?

Comment 3

[[e-mail] (:bhearsum)]

Cloning failures generally get caught by one of the error regexes these days. Purge errors have their own regex that catches them. MozillaClobber is flunkOnFailure=False. I don't think there's anything left to do here that isn't tracked elsewhere.