Closed Bug 1259599 Opened 4 years ago Closed 4 years ago

release runner shouldn't retry automatically with different graph ID

Categories

(Release Engineering :: Release Automation: Other, defect)

defect
Not set

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: rail, Assigned: rail)

References

Details

Attachments

(1 file)

In 46.0b5 we release runner tried to submit the graph, but failed multiple times (503 from TC). After some retries it exited non zero and let supervisor restart the service. The second run picked up the same release request from ship-it and submitted a new graph with different graph ID. Apparently TC created a graph for the first submission regardless of the error it returned, so we ended up with 2 graphs running in parallel.

Release runner should mark the release failed and not reuse the same release request until we verify that the first graph id is not running.
See Also: → 1259627
Assignee: nobody → rail
Comment on attachment 8735428 [details]
MozReview Request: Bug 1259599 - release runner shouldn't retry automatically with different graph ID r=Callek

https://reviewboard.mozilla.org/r/42781/#review39209

::: buildfarm/release/release-runner.py:602
(Diff revision 1)
> -            rr.update_status(release, 'Failed to start release promotion')
> -            log.exception("Failed to start release promotion for {}: ".format(release))
> +            rr.mark_as_failed(
> +                release,
> +                'Failed to start release promotion (graph ID: %s)' % graph_id)
> +            log.exception(
> +                "Failed to start release promotion for graph {} {}: ".format(
> +                    graph_id, release))

nit - mixing .format() and % formatting on two consecutive lines
Attachment #8735428 - Flags: review?(bugspam.Callek) → review+
Comment on attachment 8735428 [details]
MozReview Request: Bug 1259599 - release runner shouldn't retry automatically with different graph ID r=Callek

(In reply to Justin Wood (:Callek) [back on Mar 21] from comment #2)
> nit - mixing .format() and % formatting on two consecutive lines

I fixed this in the final push:
https://hg.mozilla.org/build/tools/rev/e38d178b287a
Attachment #8735428 - Flags: checked-in+
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
See Also: → 1269627
You need to log in before you can comment on or make changes to this bug.