1219301 - [Raptor] Consider enabling of javascript.options.discardSystemSource preference for the Raptor profile

Status: RESOLVED DUPLICATE of bug 1219176

(Firefox OS Graveyard :: Gaia, defect)

Description

[Oleg Zasypkin [:azasypkin]]

We've find out that "make raptor" disables "javascript.options.discardSystemSource" preference that per bug 990353 should have saved some memory.

This preference is enabled for both production and engineering PVT builds, so it's actually the thing user lives with and it would probably more precise to measure memory and cold launch with this preference enabled.

Let's measure!

Comment 1

[Oleg Zasypkin [:azasypkin]]

ni? myself to run raptor with and without this pref for at least Messages app.

Comment 2

[:Eli Perelman]

Going this route we would need to remove DEVICE_DEBUG=1 from `make raptor` and replace any required preferences from it with what we specifically need. Here is the current preferences set through DEVICE_DEBUG:

dom.apps.developer_mode true
network.disable.ipc.security true
dom.webcomponents.enabled true
devtools.debugger.prompt-connection false
devtools.debugger.forbid-certified-apps false
javascript.options.discardSystemSource false
b2g.adb.timeout 0

I feel like right now the only things we really may need are the adb timeout, web components, and developer mode, even though I have no idea what developer mode is.

Comment 3

[Oleg Zasypkin [:azasypkin]]

So here are results:

make raptor -> make reference-workload-light -> wait until workload is fully propagated -> raptor test coldlaunch --app sms --runs 30

I made "make raptor" before base) and 3) only (same profile, just enabled pref).

Not sure how to explain _stable_ difference in visuallyLoaded with and without --memory-delay, but RSS looks better in all cases.

base) master, memory-delay 0
| Metric                | Mean     | Median   | Min    | Max    | StdDev  | 95% Bound |
| --------------------- | -------- | -------- | ------ | ------ | ------- | --------- |
| navigationLoaded      | 968.767  | 975.500  | 865    | 1061   | 49.952  | 986.642   |
| willRenderThreads     | 1017.733 | 1026.500 | 918    | 1119   | 50.512  | 1035.809  |
| navigationInteractive | 1020.667 | 1030     | 921    | 1122   | 50.348  | 1038.684  |
| visuallyLoaded        | 1324.700 | 1314     | 1197   | 1705   | 89.523  | 1356.735  |
| contentInteractive    | 1927.933 | 1906.500 | 1788   | 2427   | 114.624 | 1968.951  |
| objectsInitEnd        | 1962.133 | 1940     | 1820   | 2459   | 115.251 | 2003.375  |
| fullyLoaded           | 3563.200 | 3545.500 | 3336   | 3905   | 125.718 | 3608.188  |
| uss                   | 19.649   | 19.563   | 18.988 | 21.508 | 0.464   | 19.815    |
| pss                   | 24.125   | 24.037   | 23.453 | 25.971 | 0.464   | 24.291    |
| rss                   | 40.109   | 40.022   | 39.449 | 41.961 | 0.463   | 40.275    |

1) master, memory-delay 10000
| Metric                | Mean     | Median   | Min    | Max    | StdDev  | 95% Bound |
| --------------------- | -------- | -------- | ------ | ------ | ------- | --------- |
| navigationLoaded      | 1013.267 | 1003     | 922    | 1202   | 56.836  | 1033.605  |
| willRenderThreads     | 1065.067 | 1056     | 971    | 1253   | 58.344  | 1085.945  |
| navigationInteractive | 1068.467 | 1059.500 | 974    | 1256   | 58.811  | 1089.512  |
| visuallyLoaded        | 1366.367 | 1363.500 | 1274   | 1524   | 57.182  | 1386.829  |
| contentInteractive    | 1960.900 | 1958     | 1851   | 2127   | 65.373  | 1984.293  |
| objectsInitEnd        | 1997.300 | 1991.500 | 1885   | 2161   | 65.325  | 2020.676  |
| fullyLoaded           | 3622.900 | 3597     | 3440   | 3809   | 106.969 | 3661.178  |
| uss                   | 19.270   | 19.285   | 19.008 | 21.062 | 0.360   | 19.398    |
| pss                   | 23.501   | 23.517   | 23.232 | 25.317 | 0.363   | 23.631    |
| rss                   | 39.722   | 39.732   | 39.461 | 41.523 | 0.360   | 39.851    |


2) master + enabled javascript.options.discardSystemSource, memory-delay 0
| Metric                | Mean     | Median   | Min    | Max    | StdDev  | 95% Bound |
| --------------------- | -------- | -------- | ------ | ------ | ------- | --------- |
| navigationLoaded      | 929.367  | 933      | 834    | 979    | 35.795  | 942.176   |
| willRenderThreads     | 976.267  | 979.500  | 877    | 1029   | 37.382  | 989.644   |
| navigationInteractive | 979.133  | 985      | 880    | 1032   | 37.517  | 992.559   |
| visuallyLoaded        | 1273.633 | 1274.500 | 1159   | 1356   | 42.758  | 1288.934  |
| contentInteractive    | 1849.567 | 1860     | 1724   | 2015   | 56.230  | 1869.688  |
| objectsInitEnd        | 1884.367 | 1894     | 1756   | 2057   | 56.654  | 1904.640  |
| fullyLoaded           | 3465.800 | 3472.500 | 3295   | 3921   | 118.029 | 3508.036  |
| uss                   | 19.559   | 19.451   | 18.566 | 21.461 | 0.707   | 19.812    |
| pss                   | 23.974   | 23.866   | 22.974 | 25.863 | 0.708   | 24.227    |
| rss                   | 39.691   | 39.584   | 38.703 | 41.586 | 0.705   | 39.943    |

3) master + enabled javascript.options.discardSystemSource, memory-delay 10000
| Metric                | Mean     | Median   | Min    | Max    | StdDev | 95% Bound |
| --------------------- | -------- | -------- | ------ | ------ | ------ | --------- |
| navigationLoaded      | 952.967  | 962      | 835    | 1016   | 44.552 | 968.909   |
| willRenderThreads     | 1001.733 | 1009     | 880    | 1067   | 45.694 | 1018.085  |
| navigationInteractive | 1004.700 | 1012.500 | 883    | 1071   | 45.851 | 1021.108  |
| visuallyLoaded        | 1296.067 | 1303.500 | 1201   | 1364   | 45.460 | 1312.334  |
| contentInteractive    | 1867.467 | 1870     | 1771   | 1937   | 46.830 | 1884.225  |
| objectsInitEnd        | 1903.767 | 1904     | 1805   | 1969   | 47.557 | 1920.785  |
| fullyLoaded           | 3475     | 3467     | 3314   | 3654   | 83.234 | 3504.785  |
| uss                   | 18.873   | 18.901   | 18.656 | 19.629 | 0.207  | 18.947    |
| pss                   | 23.042   | 23.071   | 22.819 | 23.822 | 0.209  | 23.117    |
| rss                   | 39.003   | 39.028   | 38.789 | 39.770 | 0.207  | 39.077    |

Raptor compare for base-2-3-4
sms.gaiamobile.org     base: mean  1: mean  1: delta  1: p-value  2: mean  2: delta  2: p-value  3: mean  3: delta  3: p-value
---------------------  ----------  -------  --------  ----------  -------  --------  ----------  -------  --------  ----------
navigationLoaded              969     1013        45      * 0.00      929       -39      * 0.00      953       -16        0.21
willRenderThreads            1018     1065        47      * 0.00      976       -41      * 0.00     1002       -16        0.21
navigationInteractive        1021     1068        48      * 0.00      979       -42      * 0.00     1005       -16        0.21
visuallyLoaded               1325     1366        42      * 0.04     1274       -51      * 0.01     1296       -29        0.13
contentInteractive           1928     1961        33        0.18     1850       -78      * 0.00     1867       -60      * 0.01
objectsInitEnd               1962     1997        35        0.16     1884       -78      * 0.00     1904       -58      * 0.01
fullyLoaded                  3563     3623        60        0.06     3466       -97      * 0.00     3475       -88      * 0.00
pss                        24.125   23.501    -0.624      * 0.00   23.974    -0.151        0.34   23.042    -1.083      * 0.00
uss                        19.649   19.270    -0.380      * 0.00   19.559    -0.090        0.57   18.873    -0.776      * 0.00
rss                        40.109   39.722    -0.387      * 0.00   39.691    -0.418      * 0.01   39.003    -1.106      * 0.00

Comment 4

[Oleg Zasypkin [:azasypkin]]

Attachment: metrics.ldjson (Messages app)

Ugh, raptor compare table is too long :) And is not markdown compatible to paste a Gist link, so attaching metrics.ldjson.

Comment 5

[Zibi Braniecki [:zbraniecki][:gandalf]]

Ugh, why memory delay still matters? I thought we are forcing GC before reading USS. That should make memory-delay not matter at all.

Comment 6

[Oleg Zasypkin [:azasypkin]]

Just noticed that we didn't use DEVICE_DEBUG=1 for "make raptor" in v2.2 - so does that mean Raptor v2.2 baseline was created with discardSystemSource enabled, but master had this pref disabled?

Comment 7

[:Eli Perelman]

No, the v2.2 baseline was built using [1] since the make command is outdated. See [2] for details.

[1] GAIA_OPTIMIZE=1 DEVICE_DEBUG=1 PERF_LOGGING=1 RAPTOR=1 NO_LOCK_SCREEN=1 NOFTU=1 SCREEN_TIMEOUT=0 make reset-gaia
[2] https://gist.github.com/eliperelman/7868c7144a6805ec4f2d

Comment 8

[Julien Wajsberg [:julienw] (PTO -> may 13)]

(In reply to :Eli Perelman from comment #2)
> Going this route we would need to remove DEVICE_DEBUG=1 from `make raptor`
> and replace any required preferences from it with what we specifically need.
> Here is the current preferences set through DEVICE_DEBUG:
> 
> dom.apps.developer_mode true
> network.disable.ipc.security true
> dom.webcomponents.enabled true
> devtools.debugger.prompt-connection false
> devtools.debugger.forbid-certified-apps false
> javascript.options.discardSystemSource false
> b2g.adb.timeout 0
> 
> I feel like right now the only things we really may need are the adb
> timeout, web components, and developer mode, even though I have no idea what
> developer mode is.

I don't think we need the global "web components" permission, they're enabled for certified apps with the right permission already.
I don't think we need "developer mode" either, which is basically a "god mode" that disables some checks in Gecko.

Comment 9

[Julien Wajsberg [:julienw] (PTO -> may 13)]

Did this move forward somewhere else ?

Comment 10

[Oleg Zasypkin [:azasypkin]]

(In reply to Julien Wajsberg [:julienw] from comment #9)
> Did this move forward somewhere else ?

Yep, please see bug 1219176 comment 9.

Comment 11

[Oleg Zasypkin [:azasypkin]]

We enabled javascript.options.discardSystemSource preference in PR for bug bug 1219176.