1188052 - Frequent OSX 10.6 debug dom/plugins mochitest hangs leading to harness timeouts

Status: RESOLVED INCOMPLETE

(Core Graveyard :: Plug-ins, defect)

Description

[Ryan VanderMeulen [:RyanVM]]

Not a new problem exactly, but it seems to have gotten worse lately. Given that it happens only on OSX 10.6 debug, aklotz has already told me that he wouldn't be heartbroken if we just skipped these tests on that combo.

Comment 1

[Pulsebot]

https://hg.mozilla.org/mozilla-central/rev/7d5d2d96f19b

Comment 2

[Pulsebot]

https://hg.mozilla.org/mozilla-central/rev/2e835ae56ddd

Comment 3

[Pulsebot]

https://hg.mozilla.org/mozilla-central/rev/19c842fad15c

Comment 4

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

test_beaconCookies.html started permafailing after dom/plugins was skipped, so I'm skipping it on 10.6 also and hoping the carnage stops there.
https://hg.mozilla.org/mozilla-central/rev/19c842fad15c

Comment 5

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

The failures looked like https://treeherder.mozilla.org/logviewer.html#?job_id=12170800&repo=mozilla-inbound

Comment 6

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

I had to back out all of this because m(oth) was still permafailing and I'm tired of fighting it tonight.

Comment 7

[Ryan VanderMeulen [:RyanVM]]

Which leaves us with the exciting conundrum of what to do next. We have a test chunk that isn't meeting visibility standards (>5% failure rate) and disabling our way to victory has proven to be horrendously ill-fated.

Aaron, any suggestions?

Comment 8

[Pulsebot]

Backout:
https://hg.mozilla.org/mozilla-central/rev/b2fdef69c4d2
https://hg.mozilla.org/mozilla-central/rev/6f55f64bf973
https://hg.mozilla.org/mozilla-central/rev/33dc8a83cfc0

Comment 9

[(No longer employed by Mozilla) Aaron Klotz]

Do we have any data as to when these timeouts spiked?

Comment 10

[Ryan VanderMeulen [:RyanVM]]

Kyle, is that something ActiveData could help with?

Comment 11

[Kyle Lahnakoski [:ekyle]]

Let me look into the the duration for these tests/jobs/chunks and I will get back to you.  We do not have visibility to the timeouts, but maybe we can see a pattern in the holes.

Comment 12

[Kyle Lahnakoski [:ekyle]]

How far back do we expect this problem to be?  I looked back to June 15 and I see nothing that would indicate a change in behavior, nor any creeping change that may hit a timeout limit.  I am still looking at some smaller anomalies.

Comment 13

[Ryan VanderMeulen [:RyanVM]]

The problem is that these issues have always been around to some extent (seriously, look at how old most of the deps are), so it's hard to get a feel for when things got markedly worse.

Comment 14

[Ryan VanderMeulen [:RyanVM]]

Going through the various deps, I'd say to maybe start looking around mid-July, but that's a bit hand-wavey.

Comment 15

[Kyle Lahnakoski [:ekyle]]

I broke down average run times for all tests [1].  They all seem quite stable.  

* dom/plugins/test/mochitest/test_bug1165981.html was added around July 21st, 
  but its runtime is very short ad makes no overall impact
* There were failures and slowdowns around July 5 and July 6, but normal afterward

Am I looking at the right thing (example [2])?

* All tests atarting with "dom/plugins"
* in the "mochitest" suite
* using the "debug" build
* and run on "yosemite"

[1] http://activedata.allizom.org/tools/query.html#query_id=dw285BlR
[2] http://activedata.allizom.org/tools/query.html#query_id=u1c1+Hi2

Comment 16

[Benjamin Smedberg]

Resolving old bugs which are likely not relevant any more, since NPAPI plugins are deprecated.

Comment 17

[Firefox Bug Husbandry Bot]

Removing leave-open keyword from resolved bugs, per :sylvestre.