Closed Bug 1618913 Opened 8 months ago Closed 8 months ago

Preload Firefox instances in wpt harness

Categories

(Testing :: web-platform-tests, enhancement)

Version 3
enhancement
Not set
normal

Tracking

(firefox76 fixed)

RESOLVED FIXED
mozilla76
Tracking Status
firefox76 --- fixed

People

(Reporter: jgraham, Assigned: jgraham)

References

(Blocks 1 open bug)

Details

Attachments

(4 files)

Looking at the wptrunner_instruments.txt data files, it seems wptrunner spends around 20% of the time in between-tests tasks. Most of this is starting up and shutting down Firefox.

One way to reduce this is to start Firefox before it's required so that when tests are running there's another instance either starting or started and waiting for the next time there's a browser restart. One disadvantage is that it's possible for the additional activity starting Firefox in the background to affect some tests, but this seems reasonably unlikley unless we're right at the edge of the resource limits.

A prototype of this approach suggests it can reduce the between-test time to 7% (linux64-opt), saving about an hour of total machine time for a full run on a single platform.

https://treeherder.mozilla.org/#/jobs?repo=try&revision=dba0a3228a3295903c68bb2bae64cd5ff2157801 although this does more cleanup of old profiles, so could regress the perf a little from 7% (that could perhaps happen on a background thread).

Attached image Flamegraph (after)

Attached flamegraphs. Time is number of samples interpreted as milliseconds.

The try run looks OK except for a few problems on Windows that need to be investigated.

The wpt harness spends a lot of time — about 20% of the total runtime
on Linux64 — between tests. The majority of this is shutting down and
restarting the Firefox instance under test. To reduce this
overhead we add the option to preload a Firefox instance which can be
immediately used when a new session is required. This makes the slow
process of creating a profile and starting the browser
asynchronous. Testing shows that this cuts the between-test time to
between a half and a third of the previous value.

The cost of this change is that we are adding background load at the
time tests are running, which could influence the results and add
randomness. However that seems unlikely unless we are right on the
edge of the system performance.

Assignee: nobody → james
Status: NEW → ASSIGNED
Pushed by james@hoppipolla.co.uk:
https://hg.mozilla.org/integration/autoland/rev/1da2a939d9cf
Preload Firefox instances in wpt harness, r=karlcow
Failed to create upstream wpt PR due to merge conflicts. This requires fixup from a wpt sync admin.
Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/22156 for changes under testing/web-platform/tests

Backed out changeset 1da2a939d9cf (Bug 1618913) for web platform tests failures complaining about activeWindow

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=web%2Cplatform&fromchange=4815ea533d316c528538c04d0736ef5fee228944&tochange=aca7870ec9360ec8062b3300aea1dfd70a2b1652&selectedJob=292455074

Backout link: https://hg.mozilla.org/integration/autoland/rev/4f63cf32fb281bd62fd02f1fafd2bbae8d4f4cea

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=292455074&repo=autoland&lineNumber=3667

[task 2020-03-10T13:37:27.798Z] 13:37:27     INFO - TEST-START | /css/CSS2/tables/border-collapse-dynamic-colgroup-003.xht
[task 2020-03-10T13:37:27.846Z] 13:37:27     INFO - PID 3844 | 1583847447837	Marionette	INFO	Testing http://web-platform.test:8000/css/CSS2/tables/border-collapse-dynamic-colgroup-003.xht == http://web-platform.test:8000/css/CSS2/tables/border-collapse-dynamic-colgroup-003-ref.xht
[task 2020-03-10T13:37:27.973Z] 13:37:27     INFO - TEST-UNEXPECTED-ERROR | /css/CSS2/tables/border-collapse-dynamic-colgroup-003.xht | Testing http://web-platform.test:8000/css/CSS2/tables/border-collapse-dynamic-colgroup-003.xht == http://web-platform.test:8000/css/CSS2/tables/border-collapse-dynamic-colgroup-003-ref.xht
[task 2020-03-10T13:37:27.973Z] 13:37:27     INFO - TypeError: setting getter-only property "activeWindow"
[task 2020-03-10T13:37:27.973Z] 13:37:27     INFO - ensureFocus@chrome://marionette/content/reftest.js:490:7
[task 2020-03-10T13:37:27.973Z] 13:37:27     INFO - screenshot@chrome://marionette/content/reftest.js:567:12
[task 2020-03-10T13:37:27.973Z] 13:37:27     INFO - async*compareUrls@chrome://marionette/content/reftest.js:418:26
[task 2020-03-10T13:37:27.973Z] 13:37:27     INFO - runTest@chrome://marionette/content/reftest.js:322:33
[task 2020-03-10T13:37:27.974Z] 13:37:27     INFO - async*run/testRunner<@chrome://marionette/content/reftest.js:263:29
[task 2020-03-10T13:37:27.974Z] 13:37:27     INFO - run@chrome://marionette/content/reftest.js:280:7
[task 2020-03-10T13:37:27.974Z] 13:37:27     INFO - GeckoDriver.prototype.runReftest@chrome://marionette/content/driver.js:3693:32
[task 2020-03-10T13:37:27.974Z] 13:37:27     INFO - despatch@chrome://marionette/content/server.js:305:40
[task 2020-03-10T13:37:27.974Z] 13:37:27     INFO - execute@chrome://marionette/content/server.js:275:16
[task 2020-03-10T13:37:27.974Z] 13:37:27     INFO - onPacket/<@chrome://marionette/content/server.js:248:20
[task 2020-03-10T13:37:27.974Z] 13:37:27     INFO - onPacket@chrome://marionette/content/server.js:249:9
[task 2020-03-10T13:37:27.975Z] 13:37:27     INFO - _onJSONObjectReady/<@chrome://marionette/content/transport.js:501:20
[task 2020-03-10T13:37:27.975Z] 13:37:27     INFO - REFTEST   IMAGE 1 (TEST): data:image/png;base64,
[task 2020-03-10T13:37:27.975Z] 13:37:27     INFO - REFTEST   IMAGE 2 (REFERENCE): data:image/png;base64,
[task 2020-03-10T13:37:27.975Z] 13:37:27     INFO - TEST-INFO took 173ms
Upstream PR was closed without merging

This case was missed in bug 1618798

Flags: needinfo?(james)
Pushed by james@hoppipolla.co.uk:
https://hg.mozilla.org/integration/autoland/rev/cf6095b14995
Preload Firefox instances in wpt harness, r=karlcow
https://hg.mozilla.org/integration/autoland/rev/98e984041e7f
Fix use of activeWindow in marionette, r=marionette-reviewers,maja_zf
Status: ASSIGNED → RESOLVED
Closed: 8 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla76
Upstream PR merged by moz-wptsync-bot
Pushed by wptsync@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/95ce2ba147c6
[wpt PR 22156] - [Gecko Bug 1618913] Preload Firefox instances in wpt harness, a=testonly
You need to log in before you can comment on or make changes to this bug.