1781782 - run xpcshell tests 'sequentially' when they fail frequently in parallel

Status: RESOLVED FIXED

(Testing :: XPCShell Harness, task)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

xpcshell runs all tests in parallel, then any failures that happen will be retries sequentially. This results in 95% of the failures going away- I find that many of these tests are actually failing 25-100% of the time and often timing out.

The end result of this bug is to edit xpcshell.ini manifest files and add run-sequentially key to the tests which are most prone to retrying (yet not failing all the time)

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

to find these just run xpcshell tests without retry=True and it will report failures in parallel. Patterns emerge quickly.

These align with a bigQuery find showing which testnames failed once but not twice in xpcshell tasks.

One risk here is that some new tests will become high intermittent, effectively becoming disabled- I view this as we will be finding the real errors, not masking them.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: Bug 1781782 - Run frequent xpcshell parallel failing tests sequential by default. r=ahal!

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9fe9e9146b01
Run frequent xpcshell parallel failing tests sequential by default. r=ahal,necko-reviewers,extension-reviewers,application-update-reviewers,bytesized,rpl,kershaw

Comment 4

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/9fe9e9146b01

Comment 5

[Geoff Lankow (:darktrojan)]

After this bug landed, failed tests are running twice even if --verbose is set. That's not intended, is it?

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

this was intended. Is this causing a problem for you, or hiding problems?

Comment 7

[Geoff Lankow (:darktrojan)]

It just seems pointless. I now get all the same information I got before, but twice. Which makes it more confusing to scroll back through the output to find what happened.

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

I spent some time looking at xpcshell jobs and I am not seeing this duplicated error. Maybe I am looking in the wrong place, can you give an example?