builds should halt most any pre-build steps fail

RESOLVED FIXED

Status

Release Engineering
General
P3
normal
RESOLVED FIXED
8 years ago
14 days ago

People

(Reporter: bhearsum, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [automation][cleanup][retry])

(Reporter)

Description

8 years ago
In bug 607928 someone reported a build that he thought failed in count_and_reboot.py. There was actually quite a few things that went wrong prior to that:
- Failed to completely, remove previous tools directory because of a read-only filesystem
- Failed to clone new tools directory
- Failed to run clobberer
- Failed to run purge_builds

The count_and_reboot failure is the machine being and unable to run the reboot script, because it doesn't exist.

All but the clobberer failure, and possibly it too, should've halted the build, turned it purple, and marked it as needing to be retried.
(In reply to comment #1)
> In bug 607928 someone reported a build that he thought failed in
> count_and_reboot.py. There was actually quite a few things that went wrong
> prior to that:
> - Failed to completely, remove previous tools directory because of a read-only
> filesystem
> - Failed to clone new tools directory
> - Failed to run clobberer
> - Failed to run purge_builds
> 
> The count_and_reboot failure is the machine being and unable to run the reboot
> script, because it doesn't exist.
This feels like something that puppet should deploy on all slaves for us?



> All but the clobberer failure, and possibly it too, should've halted the build,
> turned it purple, and marked it as needing to be retried.
Priority: -- → P4

Comment 2

6 years ago
(In reply to Ben Hearsum [:bhearsum] from comment #0)
> - Failed to completely, remove previous tools directory because of a
> read-only filesystem
> - Failed to clone new tools directory

bug 712205

> - Failed to run clobberer
> - Failed to run purge_builds

Do we do this already?
Component: Release Engineering → Release Engineering: Automation
OS: Mac OS X → All
Priority: P4 → --
QA Contact: release → catlee
Whiteboard: [automation][cleanup][retry]

Updated

6 years ago
Priority: -- → P3
(Reporter)

Comment 3

6 years ago
Cloning failures generally get caught by one of the error regexes these days. Purge errors have their own regex that catches them. MozillaClobber is flunkOnFailure=False. I don't think there's anything left to do here that isn't tracked elsewhere.
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
(Assignee)

Updated

5 years ago
Product: mozilla.org → Release Engineering
(Assignee)

Updated

14 days ago
Component: General Automation → General
Product: Release Engineering → Release Engineering
You need to log in before you can comment on or make changes to this bug.