1380359 - Request for performance testing on Cliqz

Status: RESOLVED FIXED

(Firefox :: General, enhancement)

Description

[:shell escalante]

Hi there,

Sorry, I'm sure this is under the wrong component.  It's a PI request.

Description:  We want to test out the performance implications of the Cliqz add-on.  Implications of integrating Cliqz with Firefox by default is the decision that needs to be made. 

I believe we're looking for perfherder like results - that can compare timing changes between different configurations.

Likely limit scope to Windows (for simplicity - more is better, but Windows is P1).  Comparing 3 configuration:

Scenario 1: Firefox without Cliqz - (baseline to compare against 2 & 3)

Scenario 2: Firefox with Cliqz and a pref switched to disable multi (baseline to see the multi doesn’t break Cliqz, so we feel good about 57 roll-out.  compare against 3 only).  pref to force this configuration (multi off while e10s enabled) set  dom.ipc.multiOptOut" to 1

Scenario 3: Firefox with Cliqz and a pref switched to enable multi (baseline to see if turning on Multi helps/hurts cliqz and where changes are needed.  compare against 1 & 2). Pref to force this configuration is setting dom.ipc.processCount to 4

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

Scenario 2 (dom.ipc.multiOptOut=1 vs Baseline):
https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=3ac5e491eec6&newProject=try&newRevision=aac727f7c72884fe9a019003dfb0f895e38638f8&framework=1
* 7 improvements (6 linux, 1 osx)
* 1 regression (win7)

Scenario 3.1 (dom.ipc.processCount=4 vs dom.ipc.multiOptOut=1):
https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=aac727f7c728&newProject=try&newRevision=224e64c0ee8ef71a6dac7035258cca1a7fea1294&framework=1
* reversing the improvements/regressions from scenario 2
* a few extra changes, but these tests are bi-modal

Scenario 3.2 (dom.ipc.processCount=4 vs Baseline):
https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=3ac5e491eec6&newProject=try&newRevision=224e64c0ee8e&framework=1
* no changes


I am not sure if this will answer your questions or if the data makes sense- but this is what our set of Talos tests on all platforms we currently run on produce.

:shell, do you need anything else here?

Comment 2

[Panos Astithas (he/him) [:past] (please ni?)]

Joel, did the try pushes contain the Cliqz add-on? I'm looking e.g. at the parent changesets of this and can't find anything related:

https://hg.mozilla.org/try/rev/224e64c0ee8ef71a6dac7035258cca1a7fea1294

Comment 3

[:shell escalante]

Hi Joel,

My question is the same as Panos in Comment 2.  If there was the Cliqz add-on installed during the 3 scenarios outlined in 
Comment 1.

Comment 4

[:shell escalante]

and sorry - the comparisons between configurations that Joel outlined are what we are looking for - if that was the question to me.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

I didn't do anything to add Cliqz, just the prefs outlined in comment 0.  How do I build firefox locally to have Cliqz?  This would be required for a push to try to get performance numbers.

Comment 6

[Panos Astithas (he/him) [:past] (please ni?)]

I'll try to post a patch that adds Cliqz tomorrow.

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

I got a .xpi of cliqz and pushed to try again.  I didn't do osx this time due to large backlogs, but I did linux and windows.

Scenario 2 (dom.ipc.multiOptOut=1 vs Baseline):
https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=d12fad3e0ad3&newProject=try&newRevision=8d02de465f45168ed893a7c823e05f361a95c341&framework=1
* sessionrestore (startup) - solid 8% regression
* ts_paint (startup) - 25% windows (9% when using a webextension- so 16% is webextension overhead)
* tp5 responsiveness - 50% windows/ 80% linux
* fileIO + memory - 8% memory increase, 2MB fileIO increase
* tart - 8% improvement
* damp - 6% improvement

Scenario 3.1 (dom.ipc.processCount=4 vs dom.ipc.multiOptOut=1):
https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=8d02de465f45&newProject=try&newRevision=babc5d7cc2e1c7513eb828584489853ab8b70712&framework=1
* tart - 8% (tab animation) - reversed from scenario 2
* damp - 6% (devtools) - reversed from scenario 2

Scenario 3.2 (dom.ipc.processCount=4 vs Baseline):
https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=d12fad3e0ad3&newProject=try&newRevision=babc5d7cc2e1c7513eb828584489853ab8b70712&framework=1
* sessionrestore (startup) - solid 8% regression
* ts_paint (startup) - 25% windows (9% when using a webextension- so 16% is webextension overhead)
* tp5 responsiveness - 50% windows/ 80% linux
* fileIO + memory - 8% memory increase, 2MB fileIO increase

scenario 3.2 is the same as Scenario 2 except for the reverted regressions seen in 3.1.  Overall this follows a similar pattern we saw when I accidentally did this without cliqz.

I think the memory and fileIO might be necessary- the fileIO is on the main thread, would it be possible to make this on a nonmain thread?

As for startup, there is work to do here- I know the screenshots team did a lot of work here.

The other item is responsiveness- this is a large change: https://wiki.mozilla.org/Buildbot/Talos/Tests#Responsiveness

Comment 8

[:shell escalante]

based on these results and Cliqz functionality and internal perf testing we are not concerned about Cliqz running with multi-processes.

Cliqz team with Panos is going over the results of Scenario 3.2 from comment 7 to see how to improve the performance.

Joel has offered to help if there is questions on interpreting the performance data.