Preload Firefox instances in wpt harness
Categories
(Testing :: web-platform-tests, enhancement)
Tracking
(firefox76 fixed)
| Tracking | Status | |
|---|---|---|
| firefox76 | --- | fixed |
People
(Reporter: jgraham, Assigned: jgraham)
References
(Blocks 1 open bug)
Details
Attachments
(4 files)
Flamegraph (before - slightly outdated)
Looking at the wptrunner_instruments.txt data files, it seems wptrunner spends around 20% of the time in between-tests tasks. Most of this is starting up and shutting down Firefox.
One way to reduce this is to start Firefox before it's required so that when tests are running there's another instance either starting or started and waiting for the next time there's a browser restart. One disadvantage is that it's possible for the additional activity starting Firefox in the background to affect some tests, but this seems reasonably unlikley unless we're right at the edge of the resource limits.
A prototype of this approach suggests it can reduce the between-test time to 7% (linux64-opt), saving about an hour of total machine time for a full run on a single platform.
|
Assignee | |
Comment 1•4 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=dba0a3228a3295903c68bb2bae64cd5ff2157801 although this does more cleanup of old profiles, so could regress the perf a little from 7% (that could perhaps happen on a background thread).
|
|
Assignee | |
Comment 2•4 years ago
|
||
|
|
Assignee | |
Comment 3•4 years ago
|
||
|
|
Assignee | |
Comment 4•4 years ago
|
||
Attached flamegraphs. Time is number of samples interpreted as milliseconds.
The try run looks OK except for a few problems on Windows that need to be investigated.
|
|
Assignee | |
Comment 5•4 years ago
|
||
The wpt harness spends a lot of time — about 20% of the total runtime
on Linux64 — between tests. The majority of this is shutting down and
restarting the Firefox instance under test. To reduce this
overhead we add the option to preload a Firefox instance which can be
immediately used when a new session is required. This makes the slow
process of creating a profile and starting the browser
asynchronous. Testing shows that this cuts the between-test time to
between a half and a third of the previous value.
The cost of this change is that we are adding background load at the
time tests are running, which could influence the results and add
randomness. However that seems unlikely unless we are right on the
edge of the system performance.
|
||
Updated•4 years ago
|
Pushed by james@hoppipolla.co.uk: https://hg.mozilla.org/integration/autoland/rev/1da2a939d9cf Preload Firefox instances in wpt harness, r=karlcow
Failed to create upstream wpt PR due to merge conflicts. This requires fixup from a wpt sync admin.
Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/22156 for changes under testing/web-platform/tests
Can't merge web-platform-tests PR due to failing upstream checks: Github PR https://github.com/web-platform-tests/wpt/pull/22156 * Community-TC (pull_request) (https://community-tc.services.mozilla.com/tasks/groups/cTQNj0AWRjW9MpW4aBPTjQ)
|
||
Comment 10•4 years ago
|
||
Backed out changeset 1da2a939d9cf (Bug 1618913) for web platform tests failures complaining about activeWindow
Backout link: https://hg.mozilla.org/integration/autoland/rev/4f63cf32fb281bd62fd02f1fafd2bbae8d4f4cea
Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=292455074&repo=autoland&lineNumber=3667
[task 2020-03-10T13:37:27.798Z] 13:37:27 INFO - TEST-START | /css/CSS2/tables/border-collapse-dynamic-colgroup-003.xht
[task 2020-03-10T13:37:27.846Z] 13:37:27 INFO - PID 3844 | 1583847447837 Marionette INFO Testing http://web-platform.test:8000/css/CSS2/tables/border-collapse-dynamic-colgroup-003.xht == http://web-platform.test:8000/css/CSS2/tables/border-collapse-dynamic-colgroup-003-ref.xht
[task 2020-03-10T13:37:27.973Z] 13:37:27 INFO - TEST-UNEXPECTED-ERROR | /css/CSS2/tables/border-collapse-dynamic-colgroup-003.xht | Testing http://web-platform.test:8000/css/CSS2/tables/border-collapse-dynamic-colgroup-003.xht == http://web-platform.test:8000/css/CSS2/tables/border-collapse-dynamic-colgroup-003-ref.xht
[task 2020-03-10T13:37:27.973Z] 13:37:27 INFO - TypeError: setting getter-only property "activeWindow"
[task 2020-03-10T13:37:27.973Z] 13:37:27 INFO - ensureFocus@chrome://marionette/content/reftest.js:490:7
[task 2020-03-10T13:37:27.973Z] 13:37:27 INFO - screenshot@chrome://marionette/content/reftest.js:567:12
[task 2020-03-10T13:37:27.973Z] 13:37:27 INFO - async*compareUrls@chrome://marionette/content/reftest.js:418:26
[task 2020-03-10T13:37:27.973Z] 13:37:27 INFO - runTest@chrome://marionette/content/reftest.js:322:33
[task 2020-03-10T13:37:27.974Z] 13:37:27 INFO - async*run/testRunner<@chrome://marionette/content/reftest.js:263:29
[task 2020-03-10T13:37:27.974Z] 13:37:27 INFO - run@chrome://marionette/content/reftest.js:280:7
[task 2020-03-10T13:37:27.974Z] 13:37:27 INFO - GeckoDriver.prototype.runReftest@chrome://marionette/content/driver.js:3693:32
[task 2020-03-10T13:37:27.974Z] 13:37:27 INFO - despatch@chrome://marionette/content/server.js:305:40
[task 2020-03-10T13:37:27.974Z] 13:37:27 INFO - execute@chrome://marionette/content/server.js:275:16
[task 2020-03-10T13:37:27.974Z] 13:37:27 INFO - onPacket/<@chrome://marionette/content/server.js:248:20
[task 2020-03-10T13:37:27.974Z] 13:37:27 INFO - onPacket@chrome://marionette/content/server.js:249:9
[task 2020-03-10T13:37:27.975Z] 13:37:27 INFO - _onJSONObjectReady/<@chrome://marionette/content/transport.js:501:20
[task 2020-03-10T13:37:27.975Z] 13:37:27 INFO - REFTEST IMAGE 1 (TEST): data:image/png;base64,
[task 2020-03-10T13:37:27.975Z] 13:37:27 INFO - REFTEST IMAGE 2 (REFERENCE): data:image/png;base64,
[task 2020-03-10T13:37:27.975Z] 13:37:27 INFO - TEST-INFO took 173ms
Upstream PR was closed without merging
|
|
Assignee | |
Comment 12•4 years ago
|
||
This case was missed in bug 1618798
|
|
Assignee | |
Updated•4 years ago
|
|
||
Comment 13•4 years ago
|
||
Pushed by james@hoppipolla.co.uk: https://hg.mozilla.org/integration/autoland/rev/cf6095b14995 Preload Firefox instances in wpt harness, r=karlcow https://hg.mozilla.org/integration/autoland/rev/98e984041e7f Fix use of activeWindow in marionette, r=marionette-reviewers,maja_zf
|
||
Comment 14•4 years ago
|
||
| bugherder | ||
https://hg.mozilla.org/mozilla-central/rev/cf6095b14995
https://hg.mozilla.org/mozilla-central/rev/98e984041e7f
Upstream PR merged by moz-wptsync-bot
|
|
||
Comment 16•4 years ago
|
||
Pushed by wptsync@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/95ce2ba147c6 [wpt PR 22156] - [Gecko Bug 1618913] Preload Firefox instances in wpt harness, a=testonly
|
|
||
Comment 17•4 years ago
|
||
| bugherder | ||
Description
•