1690746 - Browsertime benchmark measurements are generated only after the page load event

Status: RESOLVED FIXED

(Testing :: Raptor, task, P2)

Description

[Florin Strugariu [:Bebe]]

Browsertime benchmark measurements are gathered using the --script option of browsertime.

  --script                                      Add custom Javascript to run after the page has finished loading to collect metrics. If a single js file is specified, it will be included in the category named "custom" in the output json. Pass a folder to include all .js scripts in the folder, and have the folder name be the category. Note that --script can be passed multiple times.

https://searchfox.org/mozilla-central/source/testing/raptor/browsertime/browsertime_benchmark.js#2

As the doc says this JS code is run after the page has finished loading but some of our benchmarks post the data during the page load.
Because of this the measurements are not collected by browsertime and the test fails.

A good example of this is assoted-doom.
You can see here that the postToRaptor method happens during the onload event
https://github.com/mozilla/perf-automation/blob/master/benchmarks/assorted-dom/analyze-results.js#L302

Manually triggering the postToRaptor method triggers the benchmark script form browsertime.

We need a way to gather this data in browsertime before the page has finished loading

Comment 1

[Andrew Creskey [:acreskey]]

One way to fix this could be to use a custom pageCompleteCheck script where we registered the raptor-benchmark message handler (only once :).
https://github.com/sitespeedio/browsertime/blob/e1925f4c4e6c696efb16dec431fe6f815a7264a6/lib/support/cli.js#L714-L717

Another option -- I think the primary browsertime script could add the event handler.
Is this the one used by the browsertime-raptor benchmarks?
https://searchfox.org/mozilla-central/source/testing/raptor/browsertime/browsertime_scenario.js#8

Comment 2

[Florin Strugariu [:Bebe]]

Thanks for the suggestions. I will try to hack something to see if it works.

:Andrew can you check that the browsertime_benchmark script is correct(it works but should it be like that? with a Promise)
https://searchfox.org/mozilla-central/source/testing/raptor/browsertime/browsertime_benchmark.js#2

one other option is to modify the way benchmarks send the data. Could we use the browser internal "userTimings"
https://github.com/sitespeedio/browsertime/blob/main/browserscripts/timings/userTimings.js
can we store custom data in to Performance API?
https://developer.mozilla.org/en-US/docs/Web/API/Performance

Comment 3

[Greg Mierzwinski [:sparky]]

The userTimings entries is what you'd want to use here IMO.

Comment 4

[Andrew Creskey [:acreskey]]

I'm +1 on the userTimings, I think that's cleaner than the message passing.

(In reply to Florin Strugariu [:Bebe] (needinfo me) from comment #2)

:Andrew can you check that the browsertime_benchmark script is correct(it works but should it be like that? with a Promise)
https://searchfox.org/mozilla-central/source/testing/raptor/browsertime/browsertime_benchmark.js#2

Florin, I've never used the --script option, but it seems to me from reading the documents that it could just run the window.addEventListener code without the need for a promise.
** not an expert here.

Comment 5

[Florin Strugariu [:Bebe]]

Attachment: Bug 1690746 - Browsertime benchmark measurements are generated only after the page load event

Comment 6

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Attachment: Bug 1690746 - Browsertime benchmark measurements are generated only after the page load event r?#perftest

Comment 7

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Attachment: Bug 1690746 - Migrate ARES6 - fix r?#perftest

Depends on D106923

Comment 8

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Attachment: Bug 1690746 - Migrate motionmark - fix r?#perftest

Depends on D106930

Comment 9

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Attachment: Bug 1690746 - Migrate stylebench - fix r?#perftest

Depends on D106931

Comment 10

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Attachment: Bug 1690746 - Migrate speedometer - fix r?#perftest

Depends on D106932

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1c9c16a4ccdb
Browsertime benchmark measurements are generated only after the page load event r=perftest-reviewers,alexandru.irimovici,sparky
https://hg.mozilla.org/integration/autoland/rev/1691447765ea
Migrate ARES6 - fix r=perftest-reviewers,Bebe
https://hg.mozilla.org/integration/autoland/rev/6e574ae3b986
Migrate motionmark - fix r=perftest-reviewers,Bebe
https://hg.mozilla.org/integration/autoland/rev/d40e89f84d6f
Migrate stylebench - fix r=perftest-reviewers,Bebe
https://hg.mozilla.org/integration/autoland/rev/654339860958
Migrate speedometer - fix r=perftest-reviewers,Bebe

Comment 12

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/1c9c16a4ccdb
https://hg.mozilla.org/mozilla-central/rev/1691447765ea
https://hg.mozilla.org/mozilla-central/rev/6e574ae3b986
https://hg.mozilla.org/mozilla-central/rev/d40e89f84d6f
https://hg.mozilla.org/mozilla-central/rev/654339860958

Comment 13

[Greg Mierzwinski [:sparky]]

Another regression from this bug is that we can no longer run multiple page-cycles on benchmarks