1636461 - Enable tracking protection for Fenix pageload performance tests

Status: NEW

(Testing :: Raptor, defect, P2)

Description

[Andrew Creskey [:acreskey]]

Currently the raptor-browsertime pageload tests disable TrackingProtection (and the underlying Safebrowsing database) in Fenix.

See the perf/user.js profile.

And Browsertime also currently does this by default.

However Fenix currently defaults to Strict Tracking Protection which blocks resource requests and has been shown to improve visual metrics by approximately 6% over a broad range of pages.

On some sites the improvements can be up to 50% as the majority of network requests are skipped and thus the content resources are fetched sooner.

So Fenix pageload tests are not currently representative of the user-experienced performance.

Enabling TrackingProtection in Browsertime will require this change that's currently in review:
https://github.com/sitespeedio/browsertime/pull/1272
And providing these flags:
--firefox.disableSafeBrowsing false --firefox.disableTrackingProtection false

I'm not sure the best approach for separating out the the prefs for the perf/user.js
https://searchfox.org/mozilla-central/source/testing/profiles/perf/user.js

Comment 1

[Andrew Creskey [:acreskey]]

:sparky was looking at a regression in ContentfulSpeedIndex in Fenix and it appears that tracking protection may in fact be enabled in tests as this coincides with the date that TP was changed from Strict to Standard.
https://treeherder.mozilla.org/perf.html#/graphs?series=mozilla-central,2441205,1,13&selected=2441205,1158745317
In looking at the resourceCount metrics he was able to see the increase.

Comment 2

[Greg Mierzwinski [:sparky]]

:bebe, could you verify if the tracking protection is on?

Comment 3

[Florin Strugariu [:Bebe]]

Ran a small browsertime page load test and opened about:config to check the settings.
comparing the user.js seetings here are the results:

user.js > Browsertime device seetings
"privacy.trackingprotection.annotate_channels", false -> None
"privacy.trackingprotection.enabled", false -> false (not default)
"privacy.trackingprotection.introURL", "http://127.0.0.1/trackingprotection/tour" -> http://127.0.0.1/trackingprotection/tour (not default)
"privacy.trackingprotection.pbmode.enabled", false -> false (not default)

Will attach screenshots

Comment 4

[Florin Strugariu [:Bebe]]

Attachment: Screenshot_20201007-171711.png

Comment 5

[Florin Strugariu [:Bebe]]

Attachment: Screenshot_20201007-171717.png

Comment 6

[Andrew Creskey [:acreskey]]

One option to compare what is actually being blocked (since I believe the app can override the preferences) is to compare these tracking protection tests in a clean nightly vs in the performance tests.

https://senglehardt.com/test/trackingprotection/test_pages/tracking_protection.html

Comment 7

[Andrew Creskey [:acreskey]]

Florin, from Comment 4, it looks like this is disabled.

Comment 8

[Andrew Creskey [:acreskey]]

Attachment: fenix nightly tracking protection behaviour

When running the latest Fenix nightly, outside of the test framework, these are the results.

(From this test https://senglehardt.com/test/trackingprotection/test_pages/tracking_protection.html)

Note that we no longer default Fenix to "strict" tracking protection which blocks many resources.

Comment 9

[Alexandru Ionescu (needinfo me) [:alexandrui]]

any particular reason you needinfoed me, Kash?

Comment 10

[Kash Shampur [:kshampur] ⌚EST]

Ah sorry :alexandrui, forgot to follow up with a message!

This bug came up during triage meeting this week and I believe we thought you would have further context on this or if/when we should tackle this

Comment 11

[Alexandru Ionescu (needinfo me) [:alexandrui]]

acreskey, is this still relevant?

Comment 12

[Andrew Creskey [:acreskey]]

I would say that this is still relevant because of the large impact that tracking protection has on performance.

However, it's somewhat less relevant because Fenix no longer defaults to Strick tracking protection.

I know that from a test in the fall of 2021, tracking protection on desktop is working correctly in automation.
I'm not sure if that's the case for Fenix.
I can verify in a couple of weeks, or if someone is available perhaps they can test themselves.
I think the easiest way is to note the results of this site in a fresh install of Fenix: https://senglehardt.com/test/trackingprotection/test_pages/tracking_protection.html
And then to compare those to that page when loaded through test automation.
They should match.

Comment 13

[Alex Finder]

We still don't enable tracking protection for Fenix through the browsertime flag.