Closed Bug 754302 Opened 12 years ago Closed 12 years ago

possible erroneous tsspider regression on talos.zip change

Categories

(Testing :: Talos, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: k0scist, Unassigned)

Details

Attachments

(4 files)

Subject: Talos Regression  SunSpider NoChrome increase 2.77% on
Android 2.2    (Native) mobile

Regression  SunSpider NoChrome increase 2.77% on Android 2.2 (Native)
mobile
------------------------------------------------------------------------------
    Previous: avg 73.062 stddev 0.560 of 30 runs up to revision
    f80568dba010
    New     : avg 75.088 stddev 0.435 of 5 runs since revision
    27aadc2c9951
    Change  : +2.026 (2.77% / z=3.617)
    Graph   : http://mzl.la/IXl9EA

Changeset range:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=f80568dba010&tochange=27aadc2c9951

Changesets:
  * http://hg.mozilla.org/mozilla-central/rev/27aadc2c9951
    : Joel Maher <jmaher@mozilla.com> - Bug 753822 - please upload a
    talos.zip to support tcheckerboard3 as well as talos
    updates. r=armenzg
    : http://bugzilla.mozilla.org/show_bug.cgi?id=753822

Bugs:
  * http://bugzilla.mozilla.org/show_bug.cgi?id=753822 - please upload
  a talos.zip to support tcheckerboard3 as well as talos updates

Log:
https://tbpl.mozilla.org/php/getParsedLog.php?id=11651657&tree=Firefox

TBPL:
https://tbpl.mozilla.org/?rev=27aadc2c9951&noignore=1
Amongst other changes, this includes bug 704654 which changes how configuration is setup.  I'll upload the differences in the .yml files and discuss them
This is the difference between the .yml data as produced using the buildbot invocation:

['-v',
                       '-e', 'org.mozilla.fennec',
                       '-t', 'tegra-284',
                       '--branchName', 'Try',
                       '--resultsServer', 'graphs.mozilla.org', '--resultsLink', '/server/collect.cgi',
                       '--browserWait', '60',
                       '--webServer', 'bm-remote.build.mozilla.org',
                       '--sampleConfig', 'remote.config',
'--activeTests', 'tsspider',
           '--noChrome', '--symbolsPath', '../symbols',
           '--nativeUI']
Taking these differences, the +s being pre 704654 values and the -s being after:

* tpmanifest, url, url_timestamp: these are all None in the old way of doing things.  With bug 704654, there is no need for these to be None explicitly.

* the environment variables, MOZ_CRASHREPORTER_NOREPORT, MOZ_CRASHREPORTER_SHUTDOWN, NO_EM_RESTART with bug 704654 are explicitly strings; before they are ints

* fennecIDs is not verbosely specified in the new way

* filters may be the problem;  I'll come back to that

*  -'browser.cache.disk.smart_size.first_run': False,
 +'browser.cache.disk.smart_size.firstrun': False,
  The old value is not actually the preference name; see https://bugzilla.mozilla.org/show_bug.cgi?id=704654#c46

* -'test_timeout': 3600,
+'test_timeout': 1200,

In the old way, a timeout of 3600 is specified in the remote.config file.  However, we overwrite this value with the CLI default of 1200.  While we may want to consider lowering this to 1200, this won't cause a regression
So this leaves the filters.  The old way does not specify filters in the .yml file unless they are specified on the command line, so that explains why they are missing.  The new value:

-'filters': [['ignore_first', []], ['median', []]],

However, while I thought that this is what we were using nowadays, looking at the code I believe I am mistaken.

In the case where filters are not specified in the .yml file, the old run_tests.py does this:
http://hg.mozilla.org/build/talos/file/c702ff8892be/talos/run_tests.py#l458

if not filters:
   # default filters
   if yaml_config.get('ignore_first'):
      filters = [['ignore_first', [1]], ['median']]
   else:
      filters = [['ignore_max'], ['median']]

So the default should stay this, I think.

:jmaher, you might want to double check whether I'm confusing myself or not.  I'll put up a patch that reverses this and will hopefully fix the regression.  We haven't actually done a talos.zip in awhile, so it is possible that it is something else, and I suppose it is possible that using the correct .first_run preference has somehow affected things.
Attachment #623178 - Flags: review?(jmaher)
Comment on attachment 623178 [details] [diff] [review]
make ignore_max, median the default filters

Review of attachment 623178 [details] [diff] [review]:
-----------------------------------------------------------------

can we prove this?  could we look at some of the raw data in datazilla and do hand calculations?
Attachment #623178 - Flags: review?(jmaher) → review+
Yes, if the data was available it would certainly be a good confirmation
requires attachment 623233 [details] in the same directory as tsspider.raw
Running attachment 623234 [details] on attachment 623233 [details] in the same directory i get:

ignore_first, median 75.14
ignore_max, median 73.92
Percent increase: 1.65043290043

The first number matches exactly what we see on perfomatic (as it should).

The increase is more than perfomatic reports (1.2%).  But as i understand it, this is versus a baseline and so should be expected to differ somewhat
pushed: http://hg.mozilla.org/build/talos/rev/7fd63f3ef011
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: