771745 - Compare startup(firstPaint, firstloaduri) for different values of STARTUP_USING_PRELOAD_TRIAL

Status: RESOLVED FIXED

(Mozilla Metrics :: Data/Backend Reports, defect)

Description

[(dormant account)]

+++ This bug was initially created as a clone of Bug #765850 +++

Saptashi, we need to redo this analysis filtering out warm startups.

I have one more theory, Saptashi, can you filter the data by SIMPLE_MEASURES_START?
According to https://metrics.mozilla.com/data/content/pentaho-cdf-dd/Render?solution=metrics2&path=%2Ftelemetry&file=telemetryHistogram.wcdf&bookmarkState={%22impl%22%3A%22client%22%2C%22params%22%3A{%22startDate%22%3A%222012-05-29%22%2C%22endDate%22%3A%222012-06-27%22%2C%22appVersion%22%3A%22%22%2C%22appName%22%3A%22Firefox%22%2C%22arch%22%3A%22%22%2C%22OS%22%3A%22%22%2C%22version%22%3A%22%22%2C%22channel%22%3A%22nightly%22%2C%22reason%22%3A%22idle-daily%22%2C%22appBuildID%22%3A%22%22%2C%22fromPlatformBuildID%22%3A%22%22%2C%22toPlatformBuildID%22%3A%22%22%2C%22excludeParam%22%3A%22%22%2C%22measure%22%3A%22SIMPLE_MEASURES_START%22%2C%22histogramCompareParam%22%3A%22appVersion%22%2C%22histogramVariablesParam%22%3A%22%22%2C%22platformBuildIDMode%22%3A%22LATEST%22%2C%22platformBuildIDTopCount%22%3A%2230%22%2C%22conditionsStatistic%22%3A%22%23conditionsStatistic%22%2C%22submissionsParameter%22%3A[[59459]]}}

23% of our users take >=1second before firefox code executes...Those could be 23% that do slow & cold startups. Could you redo the analysis, but filter on simpleMeasures.start > 0? I'm wondering if the fact that most startups are warm is skewing the data.

I'd also like to know the distribution of simpleMeasures.start for each value of  STARTUP_USING_PRELOAD_TRIAL.

Comment 1

[Annie Elliott]

https://metrics.mozilla.com/projects/browse/METRICS-844

Comment 2

["Saptarshi Guha[:joy]"]

Btw, there were 48,762 	records for the following conditions

json$info$appName == 'Firefox'
&& json$info$OS   == 'WINNT'
&& json$info$appUpdateChannel == "nightly
&& appBuildId in (20120617 ... 20120623)
&& dateofsubmission from 20120617 ... 20120623

The quartiles of START is (raw units , i guess ms)

    0%       25%       50%       75%      100% 
    2        31       127         742 418823980 

So all starts > 0.

Better someone explains what the role of start is in this analysis?

Comment 3

["Saptarshi Guha[:joy]"]

Erm, slight modification. 4% of the data have start == 0. I'll complete this soon.
Definition of start still pending.

Comment 4

["Saptarshi Guha[:joy]"]

So as mentioned, only 4% of records have start == 0. The next higher value is 2 (see Comment 2).
With a percentage so small and a delta (2ms) so small, i would expect little difference in the analysis.

Indeed, there isn't. 

As for distribution of start for prefetch X preload (where start is now in seconds)

$preloadOff.prefetchOff
     0%      5%     25%     50%     75%     98%    100% 
  0.003   0.015   0.035   0.123   0.452  11.306 750.999 

$preloadON.prefetchOff
     0%      5%     25%     50%     75%     98%    100% 
  0.003   0.015   0.039   0.125   0.484  13.245 360.605 

$preloadOff.prefetchON
     0%      5%     25%     50%     75%     98%    100% 
  0.002   0.010   0.031   0.140   0.991  15.938 881.878 

$preloadON.prefetchON
     0%      5%     25%     50%     75%     98%    100% 
  0.002   0.010   0.031   0.126   1.007  16.691 581.385

Comment 5

["Saptarshi Guha[:joy]"]

Attachment: Start Distribution

In case the table above is difficult to decipher the figure will help.

The figure has percentiles of Log(start,2) (2%,25%,50,75,and 98). As can be seen, Prefetch = On is associated with larger values of start as compared to prefetch=Off.

Also, no interaction with prefetch and preload.

Comment 6

[Brian R. Bondy [:bbondy]]

If I'm reading the data correctly, I think it says that cold startups are positively affected pretty significantly when prefetch is disabled.  

But there is no way to disable prefetch only for some startups, nor can we disable it only for a specific firefox install only.

Since most installs are hot startups, I think the backout process for disabling prefetch should continue.  Do you agree Taras?

Comment 7

[(dormant account)]

(In reply to Brian R. Bondy [:bbondy] from comment #6)
> Since most installs are hot startups, I think the backout process for
> disabling prefetch should continue.  Do you agree Taras?

I think we should turn off prefetch when START > 0.5s. For safety we can turn it back on on an interval

Comment 8

[(dormant account)]

Sorry, didn't mean to close this. 

Saptashi, the data in comment 4 is very handy, however we need a matching table for firstPaint/firstLoadURI to make a decision.

Comment 9

["Saptarshi Guha[:joy]"]

For firstpaint (seconds)

$prefetchOff.preloadOff
      2%      25%      50%      75%      98% 
 0.57900  2.07050  5.82000 14.50750 80.03664 

$prefetchON.preloadOff
      2%      25%      50%      75%      98% 
 0.53100  1.56450  3.79600 10.59325 67.91190 

$prefetchOff.preloadON
      2%      25%      50%      75%      98% 
 0.64000  2.24750  5.48300 13.21050 79.81156 

$prefetchON.preloadON
      2%      25%      50%      75%      98% 
 0.54924  1.60800  3.65200  9.09600 64.25872 




For *URI (seconds)

$prefetchOff.preloadOff
      2%      25%      50%      75%      98% 
 0.44100  1.91200  5.97600 14.79150 85.87704 

$prefetchON.preloadOff
      2%      25%      50%      75%      98% 
 0.39022  1.37575  3.96100 11.04075 72.41188 

$prefetchOff.preloadON
     2%     25%     50%     75%     98% 
 0.4860  2.0910  5.5940 13.6750 82.8168 

$prefetchON.preloadON
      2%      25%      50%      75%      98% 
 0.39100  1.45200  3.77600  9.86000 69.57224

Comment 10

[(dormant account)]

 According to the 48K number above populations for last 2 columns are ~12K, 2K...So these wouldn't be swayed by a few outliers. 

Looks like priming prefetch with preload is the best way to go(ie least risk of regressing people). Thanks a lot Saptashi!

Comment 11

[Brian R. Bondy [:bbondy]]

Great data :)

OK so the plan is to re-enable preload on Windows (always) and continue with reverting the disable prefetch task (There is only 1 patch left to land in 2 weeks). Correct?

Comment 12

[(dormant account)]

(In reply to Brian R. Bondy [:bbondy] from comment #11)

> OK so the plan is to re-enable preload on Windows (always) and continue with
> reverting the disable prefetch task (There is only 1 patch left to land in 2
> weeks). Correct?

I think that's the best short-term action based on this.