run xpcshell tests 'sequentially' when they fail frequently in parallel
Categories
(Testing :: XPCShell Harness, task)
Tracking
(firefox105 fixed)
Tracking | Status | |
---|---|---|
firefox105 | --- | fixed |
People
(Reporter: jmaher, Assigned: jmaher)
References
Details
Attachments
(1 file)
xpcshell runs all tests in parallel, then any failures that happen will be retries sequentially. This results in 95% of the failures going away- I find that many of these tests are actually failing 25-100% of the time and often timing out.
The end result of this bug is to edit xpcshell.ini manifest files and add run-sequentially
key to the tests which are most prone to retrying (yet not failing all the time)
Assignee | ||
Comment 1•2 years ago
|
||
to find these just run xpcshell tests without retry=True and it will report failures in parallel. Patterns emerge quickly.
These align with a bigQuery find showing which testnames failed once but not twice in xpcshell tasks.
One risk here is that some new tests will become high intermittent, effectively becoming disabled- I view this as we will be finding the real errors, not masking them.
Assignee | ||
Comment 2•2 years ago
|
||
Updated•2 years ago
|
Pushed by jmaher@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/9fe9e9146b01 Run frequent xpcshell parallel failing tests sequential by default. r=ahal,necko-reviewers,extension-reviewers,application-update-reviewers,bytesized,rpl,kershaw
Comment 4•2 years ago
|
||
bugherder |
Comment 5•2 years ago
|
||
After this bug landed, failed tests are running twice even if --verbose
is set. That's not intended, is it?
Assignee | ||
Comment 6•2 years ago
|
||
this was intended. Is this causing a problem for you, or hiding problems?
Comment 7•2 years ago
|
||
It just seems pointless. I now get all the same information I got before, but twice. Which makes it more confusing to scroll back through the output to find what happened.
Assignee | ||
Comment 8•2 years ago
|
||
I spent some time looking at xpcshell jobs and I am not seeing this duplicated error. Maybe I am looking in the wrong place, can you give an example?
Description
•