1599508 - Enabling Safebrowsing is introducing significant noise in network throttled pageload tests

Status: NEW

(Core :: Performance: Navigation, defect, P3)

Description

[Andrew Creskey [:acreskey]]

Attachment: The outliers in this plot are not present with safebrowsing disabled.

When running pageload tests with safebrowsing enabled in network throttled pageload tests (e.g. 100ms rtt), we see a very significant increase in noise.

As Safebrowsing is required to fully test tracking protection and other url classification features, this is a problem.

Disabling Safebrowsing greatly reduces the number of outlier results (see attached), thus reducing the noise.

In browsertime we are disabling Safebrowsing through these prefs:
https://github.com/mozilla/browsertime/blob/6819c7f79392876255712aaac2629fa5e4d5b126/lib/firefox/webdriver/index.js#L137

Comment 1

[Andrew Creskey [:acreskey]]

As discussed, should we disable safebrowsing in browsertime by default?

Comment 2

[Julien Wajsberg [:julienw]]

Hey Andrew,
Could you please have a look at adding a priority to this bug if it's still something we want to do? Thanks!

Comment 3

[Andrew Creskey [:acreskey]]

Although we are not yet running throttled network performance tests in automation, I believe we will still run into this problem once we do.
So keeping open, but at reduced priority.