Closed Bug 1690746 Opened 4 years ago Closed 3 years ago

Browsertime benchmark measurements are generated only after the page load event

Categories

(Testing :: Raptor, task, P2)

Default
task

Tracking

(firefox89 fixed)

RESOLVED FIXED
89 Branch
Tracking Status
firefox89 --- fixed

People

(Reporter: Bebe, Assigned: Bebe)

References

(Regressed 1 open bug)

Details

Attachments

(5 files, 1 obsolete file)

Browsertime benchmark measurements are gathered using the --script option of browsertime.

  --script                                      Add custom Javascript to run after the page has finished loading to collect metrics. If a single js file is specified, it will be included in the category named "custom" in the output json. Pass a folder to include all .js scripts in the folder, and have the folder name be the category. Note that --script can be passed multiple times.

https://searchfox.org/mozilla-central/source/testing/raptor/browsertime/browsertime_benchmark.js#2

As the doc says this JS code is run after the page has finished loading but some of our benchmarks post the data during the page load.
Because of this the measurements are not collected by browsertime and the test fails.

A good example of this is assoted-doom.
You can see here that the postToRaptor method happens during the onload event
https://github.com/mozilla/perf-automation/blob/master/benchmarks/assorted-dom/analyze-results.js#L302

Manually triggering the postToRaptor method triggers the benchmark script form browsertime.

We need a way to gather this data in browsertime before the page has finished loading

Flags: needinfo?(gmierz2)
Flags: needinfo?(acreskey)

One way to fix this could be to use a custom pageCompleteCheck script where we registered the raptor-benchmark message handler (only once :).
https://github.com/sitespeedio/browsertime/blob/e1925f4c4e6c696efb16dec431fe6f815a7264a6/lib/support/cli.js#L714-L717

Another option -- I think the primary browsertime script could add the event handler.
Is this the one used by the browsertime-raptor benchmarks?
https://searchfox.org/mozilla-central/source/testing/raptor/browsertime/browsertime_scenario.js#8

Flags: needinfo?(acreskey)

Thanks for the suggestions. I will try to hack something to see if it works.

:Andrew can you check that the browsertime_benchmark script is correct(it works but should it be like that? with a Promise)
https://searchfox.org/mozilla-central/source/testing/raptor/browsertime/browsertime_benchmark.js#2

one other option is to modify the way benchmarks send the data. Could we use the browser internal "userTimings"
https://github.com/sitespeedio/browsertime/blob/main/browserscripts/timings/userTimings.js
can we store custom data in to Performance API?
https://developer.mozilla.org/en-US/docs/Web/API/Performance

Flags: needinfo?(acreskey)

The userTimings entries is what you'd want to use here IMO.

Flags: needinfo?(gmierz2)

I'm +1 on the userTimings, I think that's cleaner than the message passing.

(In reply to Florin Strugariu [:Bebe] (needinfo me) from comment #2)

:Andrew can you check that the browsertime_benchmark script is correct(it works but should it be like that? with a Promise)
https://searchfox.org/mozilla-central/source/testing/raptor/browsertime/browsertime_benchmark.js#2

Florin, I've never used the --script option, but it seems to me from reading the documents that it could just run the window.addEventListener code without the need for a promise.
** not an expert here.

Flags: needinfo?(acreskey)
Severity: -- → S3
Priority: -- → P2
Assignee: nobody → fstrugariu
Status: NEW → ASSIGNED
Attachment #9204828 - Attachment is obsolete: true
Attachment #9206363 - Attachment description: Bug 1690746 - Migrate ARES6 - fix → Bug 1690746 - Migrate ARES6 - fix r?#perftest
Attachment #9206365 - Attachment description: Bug 1690746 - Migrate motionmark - fix → Bug 1690746 - Migrate motionmark - fix r?#perftest
Attachment #9206366 - Attachment description: Bug 1690746 - Migrate stylebench - fix → Bug 1690746 - Migrate stylebench - fix r?#perftest
Attachment #9206367 - Attachment description: Bug 1690746 - Migrate speedometer - fix → Bug 1690746 - Migrate speedometer - fix r?#perftest
Pushed by aionescu@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/1c9c16a4ccdb
Browsertime benchmark measurements are generated only after the page load event r=perftest-reviewers,alexandru.irimovici,sparky
https://hg.mozilla.org/integration/autoland/rev/1691447765ea
Migrate ARES6 - fix r=perftest-reviewers,Bebe
https://hg.mozilla.org/integration/autoland/rev/6e574ae3b986
Migrate motionmark - fix r=perftest-reviewers,Bebe
https://hg.mozilla.org/integration/autoland/rev/d40e89f84d6f
Migrate stylebench - fix r=perftest-reviewers,Bebe
https://hg.mozilla.org/integration/autoland/rev/654339860958
Migrate speedometer - fix r=perftest-reviewers,Bebe
Regressions: 1701166

Another regression from this bug is that we can no longer run multiple page-cycles on benchmarks

Regressions: 1702689
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: