1764371 - reruns of release promotion action tasks shouldn't unblock tasks created by a previous run

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Julien Cristau [:jcristau]]

When an relpro action task fails, it may already have created tasks, which will be unblocked if/when it's rerun successfully. The new run will create duplicate tasks, which is not safe.
We could either 1) add a breakpoint task that blocks release tasks on the specific run of the action task being completed, or 2) have the relpro action task cancel any existing tasks in its task group before creating new ones.

Comment 1

[Aki Sasaki (not active)]

or 3) have relpro fail if there are existing unscheduled tasks in its task group.

Comment 2

[Julien Cristau [:jcristau]]

Attachment: Bug 1764371 - relpro: cancel existing tasks in the task group on rerun

Release promotion action tasks aren't atomic, so they may schedule some
tasks, then fail. The scheduled tasks depend on the action task so
normally they never run and everything's fine.
However if the action task is rerun, if the rerun succeeds it unblocks
both the tasks it scheduled and the ones scheduled by previous runs,
which may not be safe.
Prevent this by explicitly cancelling existing tasks in the group before
anything else.

Comment 3

[Julien Cristau [:jcristau]]

Attachment: Bug 1764371 - relpro: check for existing tasks in the task group on rerun

Release promotion action tasks aren't atomic, so they may schedule some
tasks, then fail. The scheduled tasks depend on the action task so
normally they never run and everything's fine.
However if the action task is rerun, if the rerun succeeds it unblocks
both the tasks it scheduled and the ones scheduled by previous runs,
which may not be safe.
Prevent this by explicitly checking for existing tasks in the group
before anything else, and returning an error if any are incomplete.

Comment 4

[Julien Cristau [:jcristau]]

I've got tentative patches for 2 and 3. 1 sounds nicer but also probably more work. wdyt?

Comment 5

[Aki Sasaki (not active)]

Nice! Have you been able to test this?

I'm leaning towards 3, because of this hypothetical scenario:

• we are running a valid and wanted release promotion graph
• someone accidentally force-reruns the release promotion action

which would lead to the in-progress graph getting cancelled, possibly with partial uploads to archive.m.o, and we need a build 2.

Comment 6

[Julien Cristau [:jcristau]]

(In reply to Aki Sasaki [:aki] (he/him) (UTC-6) from comment #5)

Nice! Have you been able to test this?

Not yet, planning to do that this week.

I'm leaning towards 3, because of this hypothetical scenario:

• we are running a valid and wanted release promotion graph
• someone accidentally force-reruns the release promotion action

which would lead to the in-progress graph getting cancelled, possibly with partial uploads to archive.m.o, and we need a build 2.

Makes sense, I guess I didn't think about force-reruns.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2aa73f78422b
relpro: check for existing tasks in the task group on rerun r=releng-reviewers,hneiva

Comment 8

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/2aa73f78422b