Last Comment Bug 771745 - Compare startup(firstPaint, firstloaduri) for different values of STARTUP_USING_PRELOAD_TRIAL
: Compare startup(firstPaint, firstloaduri) for different values of STARTUP_USI...
Status: RESOLVED FIXED
[Telemetry:P1]
:
Product: Mozilla Metrics
Classification: Other
Component: Data/Backend Reports (show other bugs)
: unspecified
: x86_64 Windows 7
: -- normal (vote)
: Moved to JIRA
Assigned To: Nobody; OK to take it and work on it
:
Mentors:
: 778123 (view as bug list)
Depends on: 764905 765850
Blocks: 774444 776720
  Show dependency treegraph
 
Reported: 2012-07-06 20:10 PDT by (dormant account)
Modified: 2012-11-29 13:47 PST (History)
9 users (show)
See Also:
Due Date:
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
Start Distribution (4.93 KB, application/pdf)
2012-07-11 15:25 PDT, "Saptarshi Guha[:joy]"
no flags Details

Description (dormant account) 2012-07-06 20:10:11 PDT
+++ This bug was initially created as a clone of Bug #765850 +++

Saptashi, we need to redo this analysis filtering out warm startups.

I have one more theory, Saptashi, can you filter the data by SIMPLE_MEASURES_START?
According to https://metrics.mozilla.com/data/content/pentaho-cdf-dd/Render?solution=metrics2&path=%2Ftelemetry&file=telemetryHistogram.wcdf&bookmarkState={%22impl%22%3A%22client%22%2C%22params%22%3A{%22startDate%22%3A%222012-05-29%22%2C%22endDate%22%3A%222012-06-27%22%2C%22appVersion%22%3A%22%22%2C%22appName%22%3A%22Firefox%22%2C%22arch%22%3A%22%22%2C%22OS%22%3A%22%22%2C%22version%22%3A%22%22%2C%22channel%22%3A%22nightly%22%2C%22reason%22%3A%22idle-daily%22%2C%22appBuildID%22%3A%22%22%2C%22fromPlatformBuildID%22%3A%22%22%2C%22toPlatformBuildID%22%3A%22%22%2C%22excludeParam%22%3A%22%22%2C%22measure%22%3A%22SIMPLE_MEASURES_START%22%2C%22histogramCompareParam%22%3A%22appVersion%22%2C%22histogramVariablesParam%22%3A%22%22%2C%22platformBuildIDMode%22%3A%22LATEST%22%2C%22platformBuildIDTopCount%22%3A%2230%22%2C%22conditionsStatistic%22%3A%22%23conditionsStatistic%22%2C%22submissionsParameter%22%3A[[59459]]}}

23% of our users take >=1second before firefox code executes...Those could be 23% that do slow & cold startups. Could you redo the analysis, but filter on simpleMeasures.start > 0? I'm wondering if the fact that most startups are warm is skewing the data.

I'd also like to know the distribution of simpleMeasures.start for each value of  STARTUP_USING_PRELOAD_TRIAL.
Comment 2 "Saptarshi Guha[:joy]" 2012-07-11 14:31:17 PDT
Btw, there were 48,762 	records for the following conditions

json$info$appName == 'Firefox'
&& json$info$OS   == 'WINNT'
&& json$info$appUpdateChannel == "nightly
&& appBuildId in (20120617 ... 20120623)
&& dateofsubmission from 20120617 ... 20120623

The quartiles of START is (raw units , i guess ms)

    0%       25%       50%       75%      100% 
    2        31       127         742 418823980 

So all starts > 0.

Better someone explains what the role of start is in this analysis?
Comment 3 "Saptarshi Guha[:joy]" 2012-07-11 14:44:30 PDT
Erm, slight modification. 4% of the data have start == 0. I'll complete this soon.
Definition of start still pending.
Comment 4 "Saptarshi Guha[:joy]" 2012-07-11 15:03:11 PDT
So as mentioned, only 4% of records have start == 0. The next higher value is 2 (see Comment 2).
With a percentage so small and a delta (2ms) so small, i would expect little difference in the analysis.

Indeed, there isn't. 

As for distribution of start for prefetch X preload (where start is now in seconds)

$preloadOff.prefetchOff
     0%      5%     25%     50%     75%     98%    100% 
  0.003   0.015   0.035   0.123   0.452  11.306 750.999 

$preloadON.prefetchOff
     0%      5%     25%     50%     75%     98%    100% 
  0.003   0.015   0.039   0.125   0.484  13.245 360.605 

$preloadOff.prefetchON
     0%      5%     25%     50%     75%     98%    100% 
  0.002   0.010   0.031   0.140   0.991  15.938 881.878 

$preloadON.prefetchON
     0%      5%     25%     50%     75%     98%    100% 
  0.002   0.010   0.031   0.126   1.007  16.691 581.385
Comment 5 "Saptarshi Guha[:joy]" 2012-07-11 15:25:45 PDT
Created attachment 641220 [details]
Start Distribution

In case the table above is difficult to decipher the figure will help.

The figure has percentiles of Log(start,2) (2%,25%,50,75,and 98). As can be seen, Prefetch = On is associated with larger values of start as compared to prefetch=Off.

Also, no interaction with prefetch and preload.
Comment 6 Brian R. Bondy [:bbondy] 2012-07-13 13:47:22 PDT
If I'm reading the data correctly, I think it says that cold startups are positively affected pretty significantly when prefetch is disabled.  

But there is no way to disable prefetch only for some startups, nor can we disable it only for a specific firefox install only.

Since most installs are hot startups, I think the backout process for disabling prefetch should continue.  Do you agree Taras?
Comment 7 (dormant account) 2012-07-16 09:40:45 PDT
(In reply to Brian R. Bondy [:bbondy] from comment #6)
> Since most installs are hot startups, I think the backout process for
> disabling prefetch should continue.  Do you agree Taras?

I think we should turn off prefetch when START > 0.5s. For safety we can turn it back on on an interval
Comment 8 (dormant account) 2012-07-16 09:50:03 PDT
Sorry, didn't mean to close this. 

Saptashi, the data in comment 4 is very handy, however we need a matching table for firstPaint/firstLoadURI to make a decision.
Comment 9 "Saptarshi Guha[:joy]" 2012-07-16 12:25:33 PDT
For firstpaint (seconds)

$prefetchOff.preloadOff
      2%      25%      50%      75%      98% 
 0.57900  2.07050  5.82000 14.50750 80.03664 

$prefetchON.preloadOff
      2%      25%      50%      75%      98% 
 0.53100  1.56450  3.79600 10.59325 67.91190 

$prefetchOff.preloadON
      2%      25%      50%      75%      98% 
 0.64000  2.24750  5.48300 13.21050 79.81156 

$prefetchON.preloadON
      2%      25%      50%      75%      98% 
 0.54924  1.60800  3.65200  9.09600 64.25872 




For *URI (seconds)

$prefetchOff.preloadOff
      2%      25%      50%      75%      98% 
 0.44100  1.91200  5.97600 14.79150 85.87704 

$prefetchON.preloadOff
      2%      25%      50%      75%      98% 
 0.39022  1.37575  3.96100 11.04075 72.41188 

$prefetchOff.preloadON
     2%     25%     50%     75%     98% 
 0.4860  2.0910  5.5940 13.6750 82.8168 

$prefetchON.preloadON
      2%      25%      50%      75%      98% 
 0.39100  1.45200  3.77600  9.86000 69.57224
Comment 10 (dormant account) 2012-07-16 12:36:22 PDT
 According to the 48K number above populations for last 2 columns are ~12K, 2K...So these wouldn't be swayed by a few outliers. 

Looks like priming prefetch with preload is the best way to go(ie least risk of regressing people). Thanks a lot Saptashi!
Comment 11 Brian R. Bondy [:bbondy] 2012-07-16 12:59:56 PDT
Great data :)

OK so the plan is to re-enable preload on Windows (always) and continue with reverting the disable prefetch task (There is only 1 patch left to land in 2 weeks). Correct?
Comment 12 (dormant account) 2012-07-16 13:29:34 PDT
(In reply to Brian R. Bondy [:bbondy] from comment #11)

> OK so the plan is to re-enable preload on Windows (always) and continue with
> reverting the disable prefetch task (There is only 1 patch left to land in 2
> weeks). Correct?

I think that's the best short-term action based on this.
Comment 13 (dormant account) 2012-11-29 13:47:05 PST
*** Bug 778123 has been marked as a duplicate of this bug. ***

Note You need to log in before you can comment on or make changes to this bug.