2008355 - Consider enabling trackingProtection and safebrowsing in Browsertime pageload tests

Status: NEW

(Testing :: Raptor, enhancement, P3)

Description

[Andrew Creskey [:acreskey]]

As far as I can tell, both TrackingProtection and Safebrowsing are disabled in our CI Browsertime pageload tests
https://github.com/sitespeedio/browsertime/blob/fd66684edf12d7e879ad6c6703e189219dde7f9f/lib/support/cli.js#L586-L597
(Although the configuration options are complex, and they could be re-enable via some other mechanism).

In bug 2008101 we discovered a major longstanding performance issue in Fenix in which tracking protection options were misconfigured.

It would be helpful to be able to detect improvements and regressions in this area, but we would have to enable tracking protection and safebrowsing in a realistic manner in CI.
Note that initially these were disabled by Peter H due to noise issues.

Related bugs:
bug 1572713
bug 1599508

Browsertime options:
https://github.com/sitespeedio/browsertime/issues/1211

In-tree disablement of safebrowsing in perf profile:
https://searchfox.org/firefox-main/rev/b82cded8c5b732c2ea15b7871d14e13b5fadeffd/testing/profiles/perf/user.js#17-29

Comment 1

[Andrew Creskey [:acreskey]]

I tried to run local browsertime tests passing in the options to not disable safebrowsing and tracking protection

--firefox.disableSafeBrowsing false 
--firefox.disableTrackingProtection false 

However I was still not able to reproduce the behaviour that I see in Fenix outside of browsertime.

Fenix Nightly
https://share.firefox.dev/4q4jRhu
Note all the tailed requests and the 5-6 second delays on them

Fenix Nightly, via browsertime
https://share.firefox.dev/3L7xnlb
Note the lack of tailed requests and no 5-6 second delays

Comment 2

[Greg Mierzwinski [:sparky]]

:acreskey, do you have a sense of how important it might be to get this issue resolved with a targeted test?

Comment 3

[Andrew Creskey [:acreskey]]

(In reply to Greg Mierzwinski [:sparky] from comment #2)

:acreskey, do you have a sense of how important it might be to get this issue resolved with a targeted test?

I think that ideally we make all navigation tests run with tracking protection and safebrowsing systems enabled.
(Since we ship these to Firefox users and they can directly impact performance).

But in terms of making our navigation tests more representative of user experience, I think this particular bug is less important than bug 1973522, for instance (if that helps!).

Comment 4

[Greg Mierzwinski [:sparky]]

Thanks for the info! Yes, that helps :)