1768204 - Run browsertime by default instead of webextension

Status: RESOLVED FIXED

(Testing :: Raptor, task, P2)

Description

[Greg Mierzwinski [:sparky]]

This bug is for changing our code to run browsertime by default rather than webextension.

See here for a possible patch: https://treeherder.mozilla.org/jobs?repo=try&tier=1%2C2%2C3&revision=513d08c4a2b7cbf514019a635a6cf3acdce4e6e5

The patch adds a browsertime_required attribute to the test manifests. If set to false, then browsertime won't be forced.

Alternatively, we could add a webext_allowed attribute that is more descriptive of what this flag does and instead sets the browsertime flag to false.

Comment 1

[Greg Mierzwinski [:sparky]]

./mach raptor -t amazon
^ uses webextension

./mach raptor --browsertime -t amazon
^ uses browsertime

After the bug is complete, ./mach raptor -t amazon will use browsertime by default

Comment 2

[Myeongjun Go]

Attachment: Bug 1768204 - Run browsertime by default instead of webextension. r=sparky

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/33aa05fe4379
Run browsertime by default instead of webextension. r=sparky,perftest-reviewers

Comment 4

[Cristina Cozmuta (:CrissCozmuta)]

https://hg.mozilla.org/mozilla-central/rev/33aa05fe4379

Comment 5

[Acasandrei Beatrice (needinfo me)]

== Change summary for alert #34696 (as of Thu, 30 Jun 2022 15:12:09 GMT) ==

Improvements:

Ratio	Test	Platform	Options	Absolute values (old vs new)
33%	welcome FirstVisualChange	windows10-64-shippable-qr	cold fission webrender	120.00 -> 80.00
23%	welcome ContentfulSpeedIndex	windows10-64-shippable-qr	cold fission webrender	298.00 -> 228.08
20%	welcome loadtime	linux1804-64-shippable-qr	fission warm webrender	43.00 -> 34.58
15%	welcome loadtime	linux1804-64-shippable-qr	cold fission webrender	75.21 -> 64.00
14%	welcome loadtime	windows10-64-shippable-qr	cold fission webrender	60.96 -> 52.29
...	...	...	...	...
4%	welcome fcp	linux1804-64-shippable-qr	fission warm webrender	51.08 -> 49.00

For up to date results, see: https://treeherder.mozilla.org/perfherder/alerts?id=34696

Comment 6

[Greg Mierzwinski [:sparky]]

This is an odd improvement, but we'll accept it as wontfix. Videos are all looking good, and I don't see any issues.