1433276 - adjust non-e10s test scheduling

Status: RESOLVED FIXED

(Testing :: General, enhancement)

Description

[Joel Maher ( :jmaher ) (UTC -8) (PTO'ish until July 5)]

we have retained non-e10s test scheduling for a few reasons:
1) android still ships with non-e10s by default
2) not all desktop tests run on android, for example web-platform-tests which have unique service worker code
3) users can easily turn off e10s to use addons or adjust their browser (not a good reason, but still valid)

We chose windows7-debug originally as this provided coverage of assertions and debug information to developers while gaining coverage on windows.  Now that we are many months past the initial decision we are looking at what remains for non-e10s coverage and the top 3 reasons still exist.

After some discussion we find that linux32-debug will give us great coverage.  Ideally in coming months #2 will not be an issue and we can then make a decision that if we have enough tests running on android (geckoview in the future) that we can be confident our browser is stable then we could be closer to making a decision to not run non-e10s on desktop builds.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8) (PTO'ish until July 5)]

this is:
mochitest-plain
mochitest-browser-chrome
mochitest-clipboard
mochitest-devtools-chrome
mochitest-gpu
mochitest-webgl
reftest
reftest-no-accel
crashtest
jsreftest
web-platform-tests
web-platform-tests-reftests


I wonder if we need:
jsreftest
reftest-no-accel
mochitest-devtools-chrome
mochitest-browser-chrome

:snorp, a small attempt at reducing the tests by 4 types- what do you think?

Comment 2

[James Willcox (:snorp) ([e-mail]) (he/him)]

Yeah, I think excluding those should be fine.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8) (PTO'ish until July 5)]

one last question- should this be on linux32 (stylo by default) or linux32-stylo-disabled ?

Comment 4

[Joel Maher ( :jmaher ) (UTC -8) (PTO'ish until July 5)]

Attachment: move non-e10s testing to linux32-debug

this patch will work for linux32/stylo- if we need stylo-disabled instead, I can adjust it quickly.

The test failures are ones that were failing in the same way on win7-debug non-e10s

Comment 5

[Chris Peterson [:cpeterson]]

(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #3)
> one last question- should this be on linux32 (stylo by default) or
> linux32-stylo-disabled ?

Just testing linux32 with Stylo default is adequate. We don't need to test non-e10s stylo-disabled because Stylo is already enabled by default on Android in Nightly.

Comment 6

[Andrew Halberstadt [:ahal]]

Comment on attachment 8945806 [details] [diff] [review]
move non-e10s testing to linux32-debug

Review of attachment 8945806 [details] [diff] [review]:
-----------------------------------------------------------------

It was hard to see what changed in the reftest.list, splinter kind of munged things up. Also, aren't there windows annotations in the web-platform manifests that should be removed? (I'm not familiar with these files)

The taskcluster changes look good though.

Comment 7

[Joel Maher ( :jmaher ) (UTC -8) (PTO'ish until July 5)]

the web-platform-tests are really hard to edit- it seems as though we have every possibly config there and sometimes old ones we haven't ran for a long time.  I would like to clean them up..i.e. if this is 80% of our cases, default to "timeout" or "crash" or "...", then exceptions for the passes.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/9d96681c74f2
adjust non-e10s test scheduling. r=ahal

Comment 9

[Narcis Beleuzu [:NarcisB]]

:jmaher, starting with this push, an almost perma failure starts to occur on dom/serviceworkers/test/test_workerupdatefoundevent.html
Could you please take a look?

Treeherder: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=runnable&filter-resultStatus=success&fromchange=4637ab31ae110a8747761ea00814463c5855b107&filter-searchStr=b5df8d5c2639fef31a716cd37d027da3f49f9643

Log: https://treeherder.mozilla.org/logviewer.html#?job_id=158871318&repo=mozilla-inbound&lineNumber=2146

Comment 10

[Joel Maher ( :jmaher ) (UTC -8) (PTO'ish until July 5)]

I landed a quick fix to disable this one test.  Looking at the log:
[task 2018-01-27T16:00:39.317Z] 16:00:39     INFO - GECKO(2223) | ++DOCSHELL 0xcc70b800 == 7 [pid = 2223] [id = {ac16d669-f467-4f37-8180-b51cf69a634a}]
[task 2018-01-27T16:00:39.319Z] 16:00:39     INFO - GECKO(2223) | ++DOMWINDOW == 15 (0xccac5160) [pid = 2223] [serial = 847] [outer = (nil)]
[task 2018-01-27T16:00:39.322Z] 16:00:39     INFO - GECKO(2223) | ++DOMWINDOW == 16 (0xcc70c000) [pid = 2223] [serial = 848] [outer = 0xccac5160]
[task 2018-01-27T16:00:39.481Z] 16:00:39     INFO - GECKO(2223) | [2223, Main Thread] WARNING: FIXME: Not implemented!: file /builds/worker/workspace/build/src/dom/serviceworkers/ServiceWorkerRegistration.cpp, line 1031
[task 2018-01-27T16:01:00.474Z] 16:01:00     INFO - GECKO(2223) | --DOMWINDOW == 15 (0xcc626400) [pid = 2223] [serial = 845] [outer = (nil)] [url = http://mochi.test:8888/tests/SimpleTest/iframe-between-tests.html]
[task 2018-01-27T16:01:09.304Z] 16:01:09     INFO - GECKO(2223) | JavaScript error: http://mochi.test:8888/tests/dom/serviceworkers/test/worker_updatefoundevent.js, line 9: ReferenceError: reject is not defined
[task 2018-01-27T16:01:44.966Z] 16:01:44     INFO - GECKO(2223) | --DOMWINDOW == 14 (0xd7195800) [pid = 2223] [serial = 844] [outer = (nil)] [url = http://mochi.test:8888/tests/dom/serviceworkers/test/test_worker_reference_gc_timeout.html]
[task 2018-01-27T16:06:06.121Z] 16:06:06     INFO - TEST-INFO | started process screentopng
[task 2018-01-27T16:06:06.125Z] 16:06:06     INFO -  /builds/worker/workspace/build/tests/bin/screentopng: error while loading shared libraries: libXss.so.1: cannot open shared object file: No such file or directory
[task 2018-01-27T16:06:06.127Z] 16:06:06     INFO - TEST-INFO | screentopng: exit 127
[task 2018-01-27T16:06:06.128Z] 16:06:06     INFO - TEST-UNEXPECTED-FAIL | dom/serviceworkers/test/test_workerupdatefoundevent.html | Test timed out.

I see:
JavaScript error: http://mochi.test:8888/tests/dom/serviceworkers/test/worker_updatefoundevent.js, line 9: ReferenceError: reject is not defined

the source is here:
https://searchfox.org/mozilla-central/source/dom/workers/test/serviceworkers/worker_updatefoundevent.js#9

:bkelly- I see you wrote the code that calls reject()- we have been discussing where to run non-e10s tests and linux32/debug seemed like the most appropriate place given current code conditions, etc.- could you look into fixing this test?

Comment 11

[Tiberius Oros[:tiberius_oros]]

https://hg.mozilla.org/mozilla-central/rev/9d96681c74f2
https://hg.mozilla.org/mozilla-central/rev/1d32e08b9e14

Comment 12

[Ben Kelly [:bkelly, not reviewing]]

Redirect NI to Marion to find someone to look at comment 10.  The code looks just busted there.  It should be using throw() instead of the undefined reject().  Its unclear to me if the test will pass if we fix that, though.

Comment 13

[Marion Daly [:mdaly]]

:catalin, can you take a look at this please?

Comment 14

[Cătălin Badea (:catalinb)]

filed a new bug to re-enable this.