Closed Bug 1433276 Opened 2 years ago Closed 2 years ago

adjust non-e10s test scheduling

Categories

(Testing :: General, enhancement)

enhancement
Not set

Tracking

(firefox60 fixed)

RESOLVED FIXED
mozilla60
Tracking Status
firefox60 --- fixed

People

(Reporter: jmaher, Assigned: jmaher)

References

(Depends on 1 open bug)

Details

Attachments

(1 file)

we have retained non-e10s test scheduling for a few reasons:
1) android still ships with non-e10s by default
2) not all desktop tests run on android, for example web-platform-tests which have unique service worker code
3) users can easily turn off e10s to use addons or adjust their browser (not a good reason, but still valid)

We chose windows7-debug originally as this provided coverage of assertions and debug information to developers while gaining coverage on windows.  Now that we are many months past the initial decision we are looking at what remains for non-e10s coverage and the top 3 reasons still exist.

After some discussion we find that linux32-debug will give us great coverage.  Ideally in coming months #2 will not be an issue and we can then make a decision that if we have enough tests running on android (geckoview in the future) that we can be confident our browser is stable then we could be closer to making a decision to not run non-e10s on desktop builds.
this is:
mochitest-plain
mochitest-browser-chrome
mochitest-clipboard
mochitest-devtools-chrome
mochitest-gpu
mochitest-webgl
reftest
reftest-no-accel
crashtest
jsreftest
web-platform-tests
web-platform-tests-reftests


I wonder if we need:
jsreftest
reftest-no-accel
mochitest-devtools-chrome
mochitest-browser-chrome

:snorp, a small attempt at reducing the tests by 4 types- what do you think?
Flags: needinfo?(snorp)
Yeah, I think excluding those should be fine.
Flags: needinfo?(snorp)
one last question- should this be on linux32 (stylo by default) or linux32-stylo-disabled ?
this patch will work for linux32/stylo- if we need stylo-disabled instead, I can adjust it quickly.

The test failures are ones that were failing in the same way on win7-debug non-e10s
Assignee: nobody → jmaher
Status: NEW → ASSIGNED
(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #3)
> one last question- should this be on linux32 (stylo by default) or
> linux32-stylo-disabled ?

Just testing linux32 with Stylo default is adequate. We don't need to test non-e10s stylo-disabled because Stylo is already enabled by default on Android in Nightly.
Attachment #8945806 - Flags: review?(ahalberstadt)
Comment on attachment 8945806 [details] [diff] [review]
move non-e10s testing to linux32-debug

Review of attachment 8945806 [details] [diff] [review]:
-----------------------------------------------------------------

It was hard to see what changed in the reftest.list, splinter kind of munged things up. Also, aren't there windows annotations in the web-platform manifests that should be removed? (I'm not familiar with these files)

The taskcluster changes look good though.
Attachment #8945806 - Flags: review?(ahalberstadt) → review+
the web-platform-tests are really hard to edit- it seems as though we have every possibly config there and sometimes old ones we haven't ran for a long time.  I would like to clean them up..i.e. if this is 80% of our cases, default to "timeout" or "crash" or "...", then exceptions for the passes.
Pushed by jmaher@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/9d96681c74f2
adjust non-e10s test scheduling. r=ahal
I landed a quick fix to disable this one test.  Looking at the log:
[task 2018-01-27T16:00:39.317Z] 16:00:39     INFO - GECKO(2223) | ++DOCSHELL 0xcc70b800 == 7 [pid = 2223] [id = {ac16d669-f467-4f37-8180-b51cf69a634a}]
[task 2018-01-27T16:00:39.319Z] 16:00:39     INFO - GECKO(2223) | ++DOMWINDOW == 15 (0xccac5160) [pid = 2223] [serial = 847] [outer = (nil)]
[task 2018-01-27T16:00:39.322Z] 16:00:39     INFO - GECKO(2223) | ++DOMWINDOW == 16 (0xcc70c000) [pid = 2223] [serial = 848] [outer = 0xccac5160]
[task 2018-01-27T16:00:39.481Z] 16:00:39     INFO - GECKO(2223) | [2223, Main Thread] WARNING: FIXME: Not implemented!: file /builds/worker/workspace/build/src/dom/serviceworkers/ServiceWorkerRegistration.cpp, line 1031
[task 2018-01-27T16:01:00.474Z] 16:01:00     INFO - GECKO(2223) | --DOMWINDOW == 15 (0xcc626400) [pid = 2223] [serial = 845] [outer = (nil)] [url = http://mochi.test:8888/tests/SimpleTest/iframe-between-tests.html]
[task 2018-01-27T16:01:09.304Z] 16:01:09     INFO - GECKO(2223) | JavaScript error: http://mochi.test:8888/tests/dom/serviceworkers/test/worker_updatefoundevent.js, line 9: ReferenceError: reject is not defined
[task 2018-01-27T16:01:44.966Z] 16:01:44     INFO - GECKO(2223) | --DOMWINDOW == 14 (0xd7195800) [pid = 2223] [serial = 844] [outer = (nil)] [url = http://mochi.test:8888/tests/dom/serviceworkers/test/test_worker_reference_gc_timeout.html]
[task 2018-01-27T16:06:06.121Z] 16:06:06     INFO - TEST-INFO | started process screentopng
[task 2018-01-27T16:06:06.125Z] 16:06:06     INFO -  /builds/worker/workspace/build/tests/bin/screentopng: error while loading shared libraries: libXss.so.1: cannot open shared object file: No such file or directory
[task 2018-01-27T16:06:06.127Z] 16:06:06     INFO - TEST-INFO | screentopng: exit 127
[task 2018-01-27T16:06:06.128Z] 16:06:06     INFO - TEST-UNEXPECTED-FAIL | dom/serviceworkers/test/test_workerupdatefoundevent.html | Test timed out.

I see:
JavaScript error: http://mochi.test:8888/tests/dom/serviceworkers/test/worker_updatefoundevent.js, line 9: ReferenceError: reject is not defined

the source is here:
https://searchfox.org/mozilla-central/source/dom/workers/test/serviceworkers/worker_updatefoundevent.js#9

:bkelly- I see you wrote the code that calls reject()- we have been discussing where to run non-e10s tests and linux32/debug seemed like the most appropriate place given current code conditions, etc.- could you look into fixing this test?
Flags: needinfo?(jmaher) → needinfo?(bkelly)
https://hg.mozilla.org/mozilla-central/rev/9d96681c74f2
https://hg.mozilla.org/mozilla-central/rev/1d32e08b9e14
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla60
Redirect NI to Marion to find someone to look at comment 10.  The code looks just busted there.  It should be using throw() instead of the undefined reject().  Its unclear to me if the test will pass if we fix that, though.
Flags: needinfo?(bkelly) → needinfo?(mdaly)
:catalin, can you take a look at this please?
Flags: needinfo?(mdaly) → needinfo?(catalin.badea392)
Depends on: 1434599
filed a new bug to re-enable this.
Flags: needinfo?(catalin.badea392)
You need to log in before you can comment on or make changes to this bug.