1259599 - release runner shouldn't retry automatically with different graph ID

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect)

Description

[Rail Aliiev [:rail]]

In 46.0b5 we release runner tried to submit the graph, but failed multiple times (503 from TC). After some retries it exited non zero and let supervisor restart the service. The second run picked up the same release request from ship-it and submitted a new graph with different graph ID. Apparently TC created a graph for the first submission regardless of the error it returned, so we ended up with 2 graphs running in parallel.

Release runner should mark the release failed and not reuse the same release request until we verify that the first graph id is not running.

Comment 1

[Rail Aliiev [:rail]]

Attachment: MozReview Request: Bug 1259599 - release runner shouldn't retry automatically with different graph ID r=Callek

Review commit: https://reviewboard.mozilla.org/r/42781/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/42781/

Comment 2

[Justin Wood (:Callek)]

Comment on attachment 8735428 [details]
MozReview Request: Bug 1259599 - release runner shouldn't retry automatically with different graph ID r=Callek

https://reviewboard.mozilla.org/r/42781/#review39209

::: buildfarm/release/release-runner.py:602
(Diff revision 1)
> -            rr.update_status(release, 'Failed to start release promotion')
> -            log.exception("Failed to start release promotion for {}: ".format(release))
> +            rr.mark_as_failed(
> +                release,
> +                'Failed to start release promotion (graph ID: %s)' % graph_id)
> +            log.exception(
> +                "Failed to start release promotion for graph {} {}: ".format(
> +                    graph_id, release))

nit - mixing .format() and % formatting on two consecutive lines

Comment 3

[Rail Aliiev [:rail]]

Comment on attachment 8735428 [details]
MozReview Request: Bug 1259599 - release runner shouldn't retry automatically with different graph ID r=Callek

(In reply to Justin Wood (:Callek) [back on Mar 21] from comment #2)
> nit - mixing .format() and % formatting on two consecutive lines

I fixed this in the final push:
https://hg.mozilla.org/build/tools/rev/e38d178b287a