849269 - Accounting for the disk and memory cache for autophone devices during s1s2 tests?

Status: RESOLVED DUPLICATE of bug 850799

(Testing Graveyard :: Autophone, defect)

Description

[Bob Clary [:bc] (inactive)]

In Autophone's s1s2 tests, we are measuring performance data for a set of throbber start and stop measurements by loading the pages several times and then graphing the mean on phonedash.

I believe that with the cache enabled, the first measurement of the set will be effectively without cache while the remainder of the measurements of the set will be with the cached data of the first measurement.

Choices:

1. We could disable the disk and memory cache and only measure the uncached behavior.

2. We could measure only the cached response by discarding the initial data point and only recording the 2nd and subsequent data.

3. We could continue as now with the default cache settings and treat the data differently during  display on phonedash. We could provide the option (checkbox in the phonedash ui) to display either only the first data point or to display the average of the 2nd and later data points.

Although not directly related to the issue of the cache, I think it would be cool to be able to display the mean along with the stddev as an error bar for each data point.

jmaher, how does talos deal with the cache?

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

we don't do anything specifically with the cache on the OS or within Firefox other than setting specific preferences:
http://hg.mozilla.org/build/talos/file/880a267bfacc/talos/PerfConfigurator.py#l223

There are a few preferences that appear to help, and that is something we should revisit since those preferences were set a LONG time ago and I bet there are more things to set.

Regarding the data, we found that the First value is really noisy, and in many cases the first few values.  Rule of thumb is collect 25 data points and throw out the first 5, fine tuning can make that a tigher loop if needed.