1076786 - Web-platform-tests are too intermittent to enable on OSX

Status: RESOLVED FIXED

(Testing :: web-platform-tests, defect)

Description

[James Graham [:jgraham]]

For some reason OSX seems to show more instability than other platforms. This suggests that the problem is either with marionette or core gecko on that platform. The instability isn't obviously limited to one set of tests. I spent some time with a slave trying to reproduce this and the most suspicious thing was that when a test unexpectedly timed out, it seemed to be impossible to load any further pages in that browser session without clearing the cache.

If this is a gecko problem I expect I will need some insight from gecko hackers to resolve it.

Comment 1

[James Graham [:jgraham]]

This improved greatly when jdm patched sptserve to queue more connections.

Based on cedar testing, I now think these are stable enough to enable on 10.8 (but not 10.6).

Comment 2

[James Graham [:jgraham]]

Attachment: wpt_osx.diff

Enable wpt on OSX 10.8 for non-cedar trees. Tested using braindump and it seems to do the right thing.

Comment 3

[James Graham [:jgraham]]

Attachment: wpt_osx.diff

This didn't land before the 38 uplift to aurora, so attached a rebased version of the patch that should only enable things for >= 39 (this time I plan to land it right away).

Comment 4

[James Graham [:jgraham]]

https://hg.mozilla.org/build/buildbot-configs/rev/ce99eb0dbeb7

Comment 5

[James Graham [:jgraham]]

https://hg.mozilla.org/build/buildbot-configs/rev/43c35b6c987d

Comment 6

[Ryan VanderMeulen [:RyanVM]]

So...2 10.8 W2 runs on inbound so far, both failing. I'm going to be hiding it now.

https://treeherder.mozilla.org/logviewer.html#?job_id=6987876&repo=mozilla-inbound

Comment 7

[Rail Aliiev [:rail]]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/43c35b6c987d

Comment 8

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/b1c87ca5d769

Comment 9

[James Graham [:jgraham]]

I think this should be unhidden now.