967588 - Unhide desktop B2G mochitests on trunk when their failure rate meets visibility standards

Status: RESOLVED FIXED

(Tree Management Graveyard :: Visibility Requests, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

At the moment, desktop B2G mochitests are timing out somewhere in the ballpark of 30-50% of the time they run. This does not meet visibility standards. I have hidden them on all trees. This bug tracks making them reliable enough to meet visibility standards.

Comment 1

[Ryan VanderMeulen [:RyanVM]]

Just a few logs from the last few hours:
https://tbpl.mozilla.org/php/getParsedLog.php?id=34065972&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=34064263&tree=Mozilla-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=34066188&tree=B2g-Inbound
https://tbpl.mozilla.org/php/getParsedLog.php?id=34057285&tree=B2g-Inbound

There are bugs on file for some of the failures, but most often, they're generically starred and retriggered.

Comment 2

[Ryan VanderMeulen [:RyanVM]]

Also, reftests aren't faring much better. I haven't hidden them yet, but they may also be heading that direction if things don't change.

Comment 3

[Ryan VanderMeulen [:RyanVM]]

Actually, b2g26 and b2g28 look pretty green, so I've left them visible there for now. This is trunk-only at the moment.

Comment 4

[Phil Ringnalda (:philor)]

Whoever works on this: once you're getting close on whatever twig you use to green them up, you'll need to let me know to stop blind-starring them with whatever word occurs to me on all the trunk trees, because I'll just be wiping them out of my way without looking at them.

Comment 5

[Jonathan Griffin (:jgriffin)]

This looks

Comment 6

[Jonathan Griffin (:jgriffin)]

These timeouts appear to be quite random.  We're getting stack traces from them when we kill them after they time out, but the stack traces that I've looked at don't seem to be similar.

Comment 7

[Phil Ringnalda (:philor)]

An interesting experiment for someone with a loaner slave would be to run desktop mochitest *unchunked* on it ten or twenty times, and see how it likes that. b2g desktop mochitest-only-chunk is just 300K tests, and all of desktop-desktop is more like 500K, but still, we're running half again more tests in this than in the biggest desktop-desktop chunk.

Comment 8

[Andrew Halberstadt [:ahal]]

Apparently these got better when we switched back from the supposedly faster m3 class slaves to the m1 ones and are now unhidden.

Comment 9

[Phil Ringnalda (:philor)]

They got slightly better but not at all cured, but we haven't switched back, we've only temporarily switched one set of slaves, the ones with -spot- in their name. These were only unhidden on b2g-inbound and only temporarily, because someone landed bustage to them in the last 30 seconds before I was going to bed last night and I wanted whoever the next sheriff was going to be to know that there was bustage they shouldn't merge around.

Comment 10

[Phil Ringnalda (:philor)]

Unhidden. Time to start aggressively disabling tests on them.

Comment 11

[David Baron :dbaron: (⌚️UTC-5, no longer working on Mozilla)]

bug 969590 indicates the m3.medium instances were gone as of a few hours before philor's comment, fwiw.