Closed
Bug 754302
Opened 12 years ago
Closed 12 years ago
possible erroneous tsspider regression on talos.zip change
Categories
(Testing :: Talos, defect)
Testing
Talos
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: k0scist, Unassigned)
Details
Attachments
(4 files)
Subject: Talos Regression SunSpider NoChrome increase 2.77% on Android 2.2 (Native) mobile Regression SunSpider NoChrome increase 2.77% on Android 2.2 (Native) mobile ------------------------------------------------------------------------------ Previous: avg 73.062 stddev 0.560 of 30 runs up to revision f80568dba010 New : avg 75.088 stddev 0.435 of 5 runs since revision 27aadc2c9951 Change : +2.026 (2.77% / z=3.617) Graph : http://mzl.la/IXl9EA Changeset range: http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=f80568dba010&tochange=27aadc2c9951 Changesets: * http://hg.mozilla.org/mozilla-central/rev/27aadc2c9951 : Joel Maher <jmaher@mozilla.com> - Bug 753822 - please upload a talos.zip to support tcheckerboard3 as well as talos updates. r=armenzg : http://bugzilla.mozilla.org/show_bug.cgi?id=753822 Bugs: * http://bugzilla.mozilla.org/show_bug.cgi?id=753822 - please upload a talos.zip to support tcheckerboard3 as well as talos updates Log: https://tbpl.mozilla.org/php/getParsedLog.php?id=11651657&tree=Firefox TBPL: https://tbpl.mozilla.org/?rev=27aadc2c9951&noignore=1
Reporter | ||
Comment 1•12 years ago
|
||
Amongst other changes, this includes bug 704654 which changes how configuration is setup. I'll upload the differences in the .yml files and discuss them
Reporter | ||
Comment 2•12 years ago
|
||
This is the difference between the .yml data as produced using the buildbot invocation: ['-v', '-e', 'org.mozilla.fennec', '-t', 'tegra-284', '--branchName', 'Try', '--resultsServer', 'graphs.mozilla.org', '--resultsLink', '/server/collect.cgi', '--browserWait', '60', '--webServer', 'bm-remote.build.mozilla.org', '--sampleConfig', 'remote.config', '--activeTests', 'tsspider', '--noChrome', '--symbolsPath', '../symbols', '--nativeUI']
Reporter | ||
Comment 3•12 years ago
|
||
Taking these differences, the +s being pre 704654 values and the -s being after: * tpmanifest, url, url_timestamp: these are all None in the old way of doing things. With bug 704654, there is no need for these to be None explicitly. * the environment variables, MOZ_CRASHREPORTER_NOREPORT, MOZ_CRASHREPORTER_SHUTDOWN, NO_EM_RESTART with bug 704654 are explicitly strings; before they are ints * fennecIDs is not verbosely specified in the new way * filters may be the problem; I'll come back to that * -'browser.cache.disk.smart_size.first_run': False, +'browser.cache.disk.smart_size.firstrun': False, The old value is not actually the preference name; see https://bugzilla.mozilla.org/show_bug.cgi?id=704654#c46 * -'test_timeout': 3600, +'test_timeout': 1200, In the old way, a timeout of 3600 is specified in the remote.config file. However, we overwrite this value with the CLI default of 1200. While we may want to consider lowering this to 1200, this won't cause a regression
Reporter | ||
Comment 4•12 years ago
|
||
So this leaves the filters. The old way does not specify filters in the .yml file unless they are specified on the command line, so that explains why they are missing. The new value: -'filters': [['ignore_first', []], ['median', []]], However, while I thought that this is what we were using nowadays, looking at the code I believe I am mistaken. In the case where filters are not specified in the .yml file, the old run_tests.py does this: http://hg.mozilla.org/build/talos/file/c702ff8892be/talos/run_tests.py#l458 if not filters: # default filters if yaml_config.get('ignore_first'): filters = [['ignore_first', [1]], ['median']] else: filters = [['ignore_max'], ['median']] So the default should stay this, I think. :jmaher, you might want to double check whether I'm confusing myself or not. I'll put up a patch that reverses this and will hopefully fix the regression. We haven't actually done a talos.zip in awhile, so it is possible that it is something else, and I suppose it is possible that using the correct .first_run preference has somehow affected things.
Reporter | ||
Comment 5•12 years ago
|
||
Attachment #623178 -
Flags: review?(jmaher)
Comment 6•12 years ago
|
||
Comment on attachment 623178 [details] [diff] [review] make ignore_max, median the default filters Review of attachment 623178 [details] [diff] [review]: ----------------------------------------------------------------- can we prove this? could we look at some of the raw data in datazilla and do hand calculations?
Attachment #623178 -
Flags: review?(jmaher) → review+
Reporter | ||
Comment 7•12 years ago
|
||
Yes, if the data was available it would certainly be a good confirmation
Reporter | ||
Comment 8•12 years ago
|
||
taken from https://tbpl.mozilla.org/php/getParsedLog.php?id=11651657&tree=Firefox
Reporter | ||
Comment 9•12 years ago
|
||
requires attachment 623233 [details] in the same directory as tsspider.raw
Reporter | ||
Comment 10•12 years ago
|
||
Running attachment 623234 [details] on attachment 623233 [details] in the same directory i get: ignore_first, median 75.14 ignore_max, median 73.92 Percent increase: 1.65043290043 The first number matches exactly what we see on perfomatic (as it should). The increase is more than perfomatic reports (1.2%). But as i understand it, this is versus a baseline and so should be expected to differ somewhat
Reporter | ||
Comment 11•12 years ago
|
||
pushed: http://hg.mozilla.org/build/talos/rev/7fd63f3ef011
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
You need to log in
before you can comment on or make changes to this bug.
Description
•