Status

defect
P1
critical
RESOLVED FIXED
3 months ago
3 months ago

People

(Reporter: ahal, Assigned: ahal)

Tracking

(Regression)

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

This looks like a potential repeat of bug 1541917 where changes to task labels have invalidated SETA's history and is causing havoc in CI. This is likely regressed by bug 1541527, though Joel said SETA itself seems to be broken. It's not clear why that happened.

Potential fixes include running a migration on the SETA database (like we did in bug 1541917) or partially backing out bug 1541527 by adding "-e10s" back to the task label.

Though we'll need to investigate why SETA proper isn't working.

Severity: normal → critical
Priority: -- → P1
Type: task → defect

This is clearly a regression from the e10s change:
https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=linux64%2Cdebug%2Cmochitest%2Cquantum&group_state=expanded&tochange=03c29de4a29d9d188490b5eca34bc9e9a720ec36&fromchange=4b43bc5647121840f9997d985e0bc3d64fb5c2b4

Unfortunately some kind of latent bug seems to have been triggered in SETA, so at the moment SETA isn't working at all. In other words, even backing out that change wouldn't fix the issue. Joel is investigating.

I have a patch that reverts the task label changes of bug 1541527 ready to go when we figure out why the service fell over:
https://hg.mozilla.org/try/rev/036d130d3adfd6903f6bd599c2363ef63b36a96c

Pushed by ahalberstadt@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/415d4eb9182a
[ci] Add '-e10s' back to task labels to satisfy SETA, r=jmaher
Keywords: leave-open
Pushed by ahalberstadt@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/58593f23158c
[ci] Add '-e10s' back to task labels to satisfy SETA, r=jmaher
Pushed by ahalberstadt@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/549d05befaa0
[ci] Add '-e10s' back to task labels to satisfy SETA, r=jmaher
Pushed by dluca@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/9066ae618a1e
[ci] Add '-e10s' back to task labels to satisfy SETA, r=jmaher CLOSED TREE

Backlogs look reasonable now, and SETA should learn to triage the new "-1proc" tasks in time. This can be closed.

N.B that the underlying problem hasn't been solved.. namely the problem of performing mass task name changes without causing unsustainable backlogs.

Status: NEW → RESOLVED
Closed: 3 months ago
Keywords: leave-open
Resolution: --- → FIXED

FWIW, xpcshell doesn't need the -1proc label.

it doesn't run in multiprocess, so technically it isn't e10s; we could say the same about many other jobs, but it is good to be clear on what is run with multiple processes.

I'm pretty sure there are xpcshell tests which use multiple processes too.

This is going to be fixed in bug 1544816. Essentially "1proc" is an artifact of how tasks are configured. In xpcshell's case:
https://searchfox.org/mozilla-central/rev/d33d470140ce3f9426af523eaa8ecfa83476c806/taskcluster/ci/test/xpcshell.yml#58

I'm pretty sure we can remove that, but just want to be sure there isn't some real reason we had that in.

You need to log in before you can comment on or make changes to this bug.