1473734 - Gecko Decision Task needs to be more resilient

Status: RESOLVED DUPLICATE of bug 1462323

(Firefox Build System :: Task Configuration, task)

Description

[(Away)]

The rate of decision task failures on try is too high in recent times.

I push to try several times a day, and given the length of our builds, I usually send off my push and then leave to do something else -- eat lunch, commute, etc. Imagine my frustration and lost productivity when I come back to a "Task failed" email.

Considering that a decision task failure completely prevents the push from making progress, we really need to ensure a high success rate.

Can we auto-retry the decision task a couple times? This gets the build going again most of the time.

But there are also sometimes more widespread infra issues where a retry won't help. Can we work on making the decision task's dependencies more resilient? As a starting point, hg.mo seems to be a big contributor to failures.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

It sounds like the root cause here is bug 1462323.

The trouble with automatic retries is that some failure modes occur after the decision task has created dependent tasks.  When the retry succeeds, those dependent tasks then execute as well, sometimes doubling or tripling the number of tasks.  We actually removed automatic retries because of this issue.