1618913 - Preload Firefox instances in wpt harness

Status: RESOLVED FIXED

(Testing :: web-platform-tests, enhancement)

Description

[James Graham [:jgraham]]

Looking at the wptrunner_instruments.txt data files, it seems wptrunner spends around 20% of the time in between-tests tasks. Most of this is starting up and shutting down Firefox.

One way to reduce this is to start Firefox before it's required so that when tests are running there's another instance either starting or started and waiting for the next time there's a browser restart. One disadvantage is that it's possible for the additional activity starting Firefox in the background to affect some tests, but this seems reasonably unlikley unless we're right at the edge of the resource limits.

A prototype of this approach suggests it can reduce the between-test time to 7% (linux64-opt), saving about an hour of total machine time for a full run on a single platform.

Comment 1

[James Graham [:jgraham]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=dba0a3228a3295903c68bb2bae64cd5ff2157801 although this does more cleanup of old profiles, so could regress the perf a little from 7% (that could perhaps happen on a background thread).

Comment 2

[James Graham [:jgraham]]

Attachment: Flamegraph (before - slightly outdated)

Comment 3

[James Graham [:jgraham]]

Attachment: Flamegraph (after)

Comment 4

[James Graham [:jgraham]]

Attached flamegraphs. Time is number of samples interpreted as milliseconds.

The try run looks OK except for a few problems on Windows that need to be investigated.

Comment 5

[James Graham [:jgraham]]

Attachment: Bug 1618913- Preload Firefox instances in wpt harness,

The wpt harness spends a lot of time — about 20% of the total runtime
on Linux64 — between tests. The majority of this is shutting down and
restarting the Firefox instance under test. To reduce this
overhead we add the option to preload a Firefox instance which can be
immediately used when a new session is required. This makes the slow
process of creating a profile and starting the browser
asynchronous. Testing shows that this cuts the between-test time to
between a half and a third of the previous value.

The cost of this change is that we are adding background load at the
time tests are running, which could influence the results and add
randomness. However that seems unlikely unless we are right on the
edge of the system performance.

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1da2a939d9cf
Preload Firefox instances in wpt harness, r=karlcow

Comment 7

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Failed to create upstream wpt PR due to merge conflicts. This requires fixup from a wpt sync admin.

Comment 8

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Created web-platform-tests PR https://github.com/web-platform-tests/wpt/pull/22156 for changes under testing/web-platform/tests

Comment 9

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Can't merge web-platform-tests PR due to failing upstream checks:
Github PR https://github.com/web-platform-tests/wpt/pull/22156
* Community-TC (pull_request) (https://community-tc.services.mozilla.com/tasks/groups/cTQNj0AWRjW9MpW4aBPTjQ)

Comment 10

[Bogdan Tara[:bogdan_tara | bogdant]]

Backed out changeset 1da2a939d9cf (Bug 1618913) for web platform tests failures complaining about activeWindow

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&searchStr=web%2Cplatform&fromchange=4815ea533d316c528538c04d0736ef5fee228944&tochange=aca7870ec9360ec8062b3300aea1dfd70a2b1652&selectedJob=292455074

Backout link: https://hg.mozilla.org/integration/autoland/rev/4f63cf32fb281bd62fd02f1fafd2bbae8d4f4cea

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=292455074&repo=autoland&lineNumber=3667

[task 2020-03-10T13:37:27.798Z] 13:37:27     INFO - TEST-START | /css/CSS2/tables/border-collapse-dynamic-colgroup-003.xht
[task 2020-03-10T13:37:27.846Z] 13:37:27     INFO - PID 3844 | 1583847447837	Marionette	INFO	Testing http://web-platform.test:8000/css/CSS2/tables/border-collapse-dynamic-colgroup-003.xht == http://web-platform.test:8000/css/CSS2/tables/border-collapse-dynamic-colgroup-003-ref.xht
[task 2020-03-10T13:37:27.973Z] 13:37:27     INFO - TEST-UNEXPECTED-ERROR | /css/CSS2/tables/border-collapse-dynamic-colgroup-003.xht | Testing http://web-platform.test:8000/css/CSS2/tables/border-collapse-dynamic-colgroup-003.xht == http://web-platform.test:8000/css/CSS2/tables/border-collapse-dynamic-colgroup-003-ref.xht
[task 2020-03-10T13:37:27.973Z] 13:37:27     INFO - TypeError: setting getter-only property "activeWindow"
[task 2020-03-10T13:37:27.973Z] 13:37:27     INFO - ensureFocus@chrome://marionette/content/reftest.js:490:7
[task 2020-03-10T13:37:27.973Z] 13:37:27     INFO - screenshot@chrome://marionette/content/reftest.js:567:12
[task 2020-03-10T13:37:27.973Z] 13:37:27     INFO - async*compareUrls@chrome://marionette/content/reftest.js:418:26
[task 2020-03-10T13:37:27.973Z] 13:37:27     INFO - runTest@chrome://marionette/content/reftest.js:322:33
[task 2020-03-10T13:37:27.974Z] 13:37:27     INFO - async*run/testRunner<@chrome://marionette/content/reftest.js:263:29
[task 2020-03-10T13:37:27.974Z] 13:37:27     INFO - run@chrome://marionette/content/reftest.js:280:7
[task 2020-03-10T13:37:27.974Z] 13:37:27     INFO - GeckoDriver.prototype.runReftest@chrome://marionette/content/driver.js:3693:32
[task 2020-03-10T13:37:27.974Z] 13:37:27     INFO - despatch@chrome://marionette/content/server.js:305:40
[task 2020-03-10T13:37:27.974Z] 13:37:27     INFO - execute@chrome://marionette/content/server.js:275:16
[task 2020-03-10T13:37:27.974Z] 13:37:27     INFO - onPacket/<@chrome://marionette/content/server.js:248:20
[task 2020-03-10T13:37:27.974Z] 13:37:27     INFO - onPacket@chrome://marionette/content/server.js:249:9
[task 2020-03-10T13:37:27.975Z] 13:37:27     INFO - _onJSONObjectReady/<@chrome://marionette/content/transport.js:501:20
[task 2020-03-10T13:37:27.975Z] 13:37:27     INFO - REFTEST   IMAGE 1 (TEST): data:image/png;base64,
[task 2020-03-10T13:37:27.975Z] 13:37:27     INFO - REFTEST   IMAGE 2 (REFERENCE): data:image/png;base64,
[task 2020-03-10T13:37:27.975Z] 13:37:27     INFO - TEST-INFO took 173ms

Comment 11

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Upstream PR was closed without merging

Comment 12

[James Graham [:jgraham]]

Attachment: Bug 1618913 - Fix use of activeWindow in marionette,

This case was missed in bug 1618798

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/cf6095b14995
Preload Firefox instances in wpt harness, r=karlcow
https://hg.mozilla.org/integration/autoland/rev/98e984041e7f
Fix use of activeWindow in marionette, r=marionette-reviewers,maja_zf

Comment 14

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/cf6095b14995
https://hg.mozilla.org/mozilla-central/rev/98e984041e7f

Comment 15

[Web Platform Test Sync Bot [:wpt-sync] (Matrix: #interop:mozilla.org)]

Upstream PR merged by moz-wptsync-bot

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/95ce2ba147c6
[wpt PR 22156] - [Gecko Bug 1618913] Preload Firefox instances in wpt harness, a=testonly

Comment 17

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/95ce2ba147c6