Closed
Bug 1222588
Opened 5 years ago
Closed 5 years ago
talos should output PERFHERDER_DATA instead of TALOS_DATA
Categories
(Testing :: Talos, defect)
Testing
Talos
Tracking
(firefox45 fixed, b2g-v2.5 fixed)
RESOLVED
FIXED
mozilla45
People
(Reporter: parkouss, Assigned: parkouss)
References
Details
Attachments
(5 files, 2 obsolete files)
6.89 KB,
patch
|
wlach
:
feedback+
|
Details | Diff | Splinter Review |
46.47 KB,
application/json
|
Details | |
MozReview Request: Bug 1222588 - talos should output PERFHERDER_DATA instead of TALOS_DATA. r=jmaher
40 bytes,
text/x-review-board-request
|
jmaher
:
review+
|
Details |
47 bytes,
text/x-github-pull-request
|
jmaher
:
review+
|
Details | Review |
1.49 KB,
patch
|
jmaher
:
review+
parkouss
:
feedback+
|
Details | Diff | Splinter Review |
So basically perferder now have a generic way to get perf data from jobs, the PERFHERDER_DATA line (format specification here: https://github.com/wlach/treeherder/tree/master/schemas, with an example here: https://bugzilla.mozilla.org/show_bug.cgi?id=1222549#c1). Currently talos do not use that and output his own TALOS_DATA line - with a talos specific format.We should get rid of that, i.e. make talos just output PERFHERDER_DATA instead.
Assignee | ||
Comment 1•5 years ago
|
||
Assignee | ||
Comment 2•5 years ago
|
||
Assignee | ||
Comment 3•5 years ago
|
||
I started to look at this, see attachment 8684559 [details] for a result, and attachment 8684560 [details] for a comparison with current data format. So basically we are loosing a lot of data: - each point value - machine name, os, platform - run date, talos options used Do we want that in the new data format ? What do we need exactly ? I took the choice to only export resulting values, with the addition of unit and lowerIsBetter information. But I am not sure what is needed exactly - and in which format. For example the PERFHERDER_DATA specification do not allow a list or raw data values (but I believe we can add a key like raw_points in subtest if we want that?). Also, I am not sure about the counters. They often have two values (mean and max) I believe mean must be the default one (ie the "value" in each subtest/suite). Do we want max to be exposed too ? Sorry, plenty of questions in only one comment. ;;
Flags: needinfo?(wlachance)
Comment 4•5 years ago
|
||
(In reply to Julien Pagès (:parkouss) from comment #3) > I started to look at this, see attachment 8684559 [details] for a result, > and attachment 8684560 [details] for a comparison with current data format. > > So basically we are loosing a lot of data: > > - each point value Yes, I think as I said earlier, we should provide an optional field in PERFHERDER_DATA called "replicates" for each subtest, where this can be included. We don't need to ingest it. > - machine name, os, platform This isn't needed, it's included in the job data ingested elsewhere by treeherder. > - run date, talos options used Likewise. > Also, I am not sure about the counters. They often have two values (mean and > max) I believe mean must be the default one (ie the "value" in each > subtest/suite). Do we want max to be exposed too ? So currently we're only ingesting "mean" in Perfherder: https://github.com/mozilla/treeherder/blob/master/treeherder/etl/perf.py#L263 So I think it makes sense to just put that into the PERFHERDER_DATA structure for now, to preserve the existing behaviour. > Sorry, plenty of questions in only one comment. ;; Np! Keep on asking. :)
Flags: needinfo?(wlachance)
Assignee | ||
Comment 5•5 years ago
|
||
Well, my first attempt to this. I made some decisions, tell me if anything needs a change. - framework name is always "talos" for counters (example log in [1]): - each counter is a suite - the name is prefixed with 'counter_' - the value is the mean - no subtests field for non counters (example in [2]) - each suite is a talos test (e.g, tp5o) - each page inside a talos test is a subtest (e.g, media.photobucket.com) - each suite has at least one subtest - suite and subtest hold the attribute "lowerIsBetter" when required - each subtest hold the unit field when required - subtest value is the previous "filtered" value - each suite has a "replicates" field which looks like the previously named "result" field (contains the raw list of values) Pushed to try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=75dcc769d8c4 Note that some counters are listed without any "value" field, this is fixed in the attached patch. Also note that the patch do NOT replace TALOSDATA - it just add a new PERFHERDER_DATA, as I believe we want to have both until we made the according changes to perfherder. Tell me if my assumption is wrong here (and what is the plan in such a case). [1]: http://archive.mozilla.org/pub/firefox/try-builds/j.parkouss@gmail.com-75dcc769d8c414dcedc191a90dd5ab61c05e3ed1/try-win32/try_win7-ix_test-xperf-bm126-tests1-windows-build3.txt.gz [2]: http://archive.mozilla.org/pub/firefox/try-builds/j.parkouss@gmail.com-75dcc769d8c414dcedc191a90dd5ab61c05e3ed1/try-win32/try_win7-ix_test-tp5o-bm126-tests1-windows-build2.txt.gz
Attachment #8684678 -
Flags: review?(jmaher)
Attachment #8684678 -
Flags: feedback?(wlachance)
Assignee | ||
Comment 6•5 years ago
|
||
For convenience, the data exported and formatted from tp5o build.
Attachment #8684559 -
Attachment is obsolete: true
Attachment #8684560 -
Attachment is obsolete: true
Assignee | ||
Comment 7•5 years ago
|
||
Oh, I forgot to say that the patch is basically a copy/paste of PerfherderOutput.__call__ method - with a few adaptations for the new format. I had some headache trying to understand all that code - finally I must say I gave up, some parts are here, I can't remember/understand why :(. Next great step would be to make that function understandable if you agree with me that it is hard to read.
Comment 8•5 years ago
|
||
(In reply to Julien Pagès (:parkouss) from comment #5) > Created attachment 8684678 [details] [diff] [review] > perfherder_data.patch > > Well, my first attempt to this. > > I made some decisions, tell me if anything needs a change. Thanks for looking into this so quickly. > - framework name is always "talos" Yup! This is right. > for counters (example log in [1]): > - each counter is a suite > - the name is prefixed with 'counter_' > - the value is the mean > - no subtests field No, this isn't the right organization IMO. We've made the counters subtests of the suite in which they're run, but defined seperately from the *actual* subtests, and not given a summary value (so they're displayed individually in the compare view). Here's an example from https://treeherder.mozilla.org/api/project/mozilla-inbound/performance/signatures/ "d22e336c3614c2b16d5e9b238ba67c087a6eeb85": { "test": "Main_RSS", "suite": "tp5o", "option_collection_hash": "102210fe594ee9b33d82058545b1ed14f4c8206e", "test_options": [ "e10s" ], "machine_platform": "linux64" } So for example, in tp5o, you'd have something like: suites: [ { 'name': 'tp5o', value: 123, subtests: [ <actual subtests of tp5o> ] }, { 'name': 'tp5o', subtests: [ {name: Main_RSS, value: 456 }, ... ] } ] Using the mean as the value for the counters is correct, however. > for non counters (example in [2]) > - each suite is a talos test (e.g, tp5o) > - each page inside a talos test is a subtest (e.g, media.photobucket.com) > - each suite has at least one subtest > - suite and subtest hold the attribute "lowerIsBetter" when required > - each subtest hold the unit field when required > - subtest value is the previous "filtered" value > - each suite has a "replicates" field which looks like the previously named > "result" field (contains the raw list of values) This all sounds right. > Pushed to try: > https://treeherder.mozilla.org/#/jobs?repo=try&revision=75dcc769d8c4 > > Note that some counters are listed without any "value" field, this is fixed > in the attached patch. > Also note that the patch do NOT replace TALOSDATA - it just add a new > PERFHERDER_DATA, as I believe we want to have both until we made the > according changes to perfherder. Tell me if my assumption is wrong here (and > what is the plan in such a case). No, we'll need to take out TALOSDATA otherwise we'll get double results for each test (since we're still parsing Talos by default). I don't see any reason to print both, at least once we've verified that we're getting the same values. :)
Comment 9•5 years ago
|
||
> (In reply to Julien Pagès (:parkouss) from comment #5)
> > - each suite has a "replicates" field which looks like the previously named
> > "result" field (contains the raw list of values)
Actually it would be good if you could associate the replicates with the corresponding subtests, instead of making them a property of the suite.
Comment 10•5 years ago
|
||
(In reply to Julien Pagès (:parkouss) from comment #7) > Oh, I forgot to say that the patch is basically a copy/paste of > PerfherderOutput.__call__ method - with a few adaptations for the new format. > > I had some headache trying to understand all that code - finally I must say > I gave up, some parts are here, I can't remember/understand why :(. Next > great step would be to make that function understandable if you agree with > me that it is hard to read. Yeah, I'd be totally fine with you completely rewriting this. :) As I said earlier, we can scrap TALOSDATA. Feel free to start with a blank slate.
Comment 11•5 years ago
|
||
Comment on attachment 8684678 [details] [diff] [review] perfherder_data.patch Review of attachment 8684678 [details] [diff] [review]: ----------------------------------------------------------------- Going to give this f+, but it needs some adjustment/refinement before it goes in.
Attachment #8684678 -
Flags: feedback?(wlachance) → feedback+
Comment 12•5 years ago
|
||
Comment on attachment 8684678 [details] [diff] [review] perfherder_data.patch will hold off until feedback is updated.
Attachment #8684678 -
Flags: review?(jmaher)
Assignee | ||
Comment 13•5 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=14e7966e7186
Assignee | ||
Comment 14•5 years ago
|
||
Oups, I forgot to update the mozharness part that still check for the TALOSDATA line...
Assignee | ||
Comment 15•5 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=0b54cfa0f885
Assignee | ||
Comment 16•5 years ago
|
||
Bug 1222588 - talos should output PERFHERDER_DATA instead of TALOS_DATA. r=jmaher
Attachment #8685046 -
Flags: review?(jmaher)
Updated•5 years ago
|
Attachment #8685046 -
Flags: review?(jmaher)
Comment 17•5 years ago
|
||
Comment on attachment 8685046 [details] MozReview Request: Bug 1222588 - talos should output PERFHERDER_DATA instead of TALOS_DATA. r=jmaher https://reviewboard.mozilla.org/r/24717/#review22255 overall I really like this patch, I see it cleaning up a lot. When we stop publishing to graph server we can remove some command line arguments to talos and stop tracking them (like branch, revision, machine, etc.)! A few questions below. ::: testing/talos/talos/output.py:483 (Diff revision 1) > + 'replicates': replicates[page], does this 'replicates' field work for non pageloader tests (like ts_paint)? ::: testing/talos/talos/output.py:514 (Diff revision 1) > + counter_subtests.append(subtest) can we do: counter_subtests.append(subtest) subtest.modify... and then have that subtest value updated inside of counter_subtests? If not, then we should append it after adding the value. ::: testing/talos/talos/output.py:519 (Diff revision 1) > # calculate mean and max value here we are only calculating the mean. I know this fits into our scenario well, but the max does have significant value as if we have new max values it could be a sign of something. Maybe a new bug to evaluate counters and determine if we want to duplicate the counters and track the max value as well- then we could insert as a unique counter and generate alerts on the max value.
Assignee | ||
Comment 18•5 years ago
|
||
For the replicates, looking at http://archive.mozilla.org/pub/firefox/try-builds/j.parkouss@gmail.com-a0aa3ce3b5d575132499fb385e12ef603591e66d/try-win32/try_win7-ix_test-other-bm119-tests1-windows-build7.txt.gz: > { > "name": "ts_paint", > "subtests": [ > { > "name": "ts_paint", > "replicates": [ > 1434.0, > 1077.0, > 1018.0, > 1030.0, > 1031.0, > 1064.0, > 1042.0, > 1035.0, > 1029.0, > 1117.0, > 1036.0, > 1049.0, > 1023.0, > 1053.0, > 1061.0, > 1058.0, > 1062.0, > 1092.0, > 1051.0, > 1042.0 > ], > "value": 1049.0 > } > ], > "value": 1049.0 > }, it looks ok to me. I will make the change for point 2, removing a variable not necessary seems good to me. For the last point, I asked before and :wlach answered in comment 4: for now, perfherder only digest "mean", this is why max is not taken into account. Maybe we could add that inside the subtest counter, ie: suites: [ { 'name': 'tp5o', value: 123, subtests: [ <actual subtests of tp5o> ] }, { 'name': 'tp5o', subtests: [ {name: Main_RSS, value: 456, max: 502 }, ... ] } ] Since we already add some extra keys (lowerIsBetter, unit) in subtests that are not counters.
Assignee | ||
Comment 19•5 years ago
|
||
hm, something strange though - ts_paint filters are "ignore first" + "mean" but if I calculate it manually it is slightly different: >>> l = [1077.0, 1018.0, 1030.0, 1031.0, 1064.0, 1042.0, 1035.0, 1029.0, 1117.0, 1036.0, 1049.0, 1023.0, 1053.0, 1061.0, 1058.0, 1062.0, 1092.0, 1051.0, 1042.0] >>> float(sum(l))/len(l) 1051.0526315789473 (Note I removed the first value) But this does not seem to be related to this patch - If I look into http://archive.mozilla.org/pub/firefox/try-builds/j.parkouss@gmail.com-75dcc769d8c414dcedc191a90dd5ab61c05e3ed1/try-win32/try_win7-ix_test-other-bm126-tests1-windows-build9.txt.gz, in the TALOSDATA section, I can see something similar: >>> l = [1058.0, 1048.0, 1023.0, 1032.0, 1050.0, 1007.0, 1052.0, 1055.0, 1107.0, 1073.0, 1024.0, 1044.0, 1057.0, 1049.0, 1062.0, 1052.0, 1100.0, 1043.0, 1040.0] >>> float(sum(l))/len(l) 1051.3684210526317 (Note I removed the first value) And the value/filtered are 1050.0.
Assignee | ||
Updated•5 years ago
|
Attachment #8685046 -
Flags: review?(jmaher)
Assignee | ||
Comment 20•5 years ago
|
||
Comment on attachment 8685046 [details] MozReview Request: Bug 1222588 - talos should output PERFHERDER_DATA instead of TALOS_DATA. r=jmaher Review request updated; see interdiff: https://reviewboard.mozilla.org/r/24717/diff/1-2/
Assignee | ||
Updated•5 years ago
|
Attachment #8685046 -
Flags: feedback?(wlachance)
Comment 21•5 years ago
|
||
Comment on attachment 8685046 [details] MozReview Request: Bug 1222588 - talos should output PERFHERDER_DATA instead of TALOS_DATA. r=jmaher This looks good to me. It looks like the logparser failed during the try run though, could you do another?
Attachment #8685046 -
Flags: feedback?(wlachance) → feedback+
Assignee | ||
Comment 22•5 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=284b3d771c61
Comment 23•5 years ago
|
||
the log parser is still failing and the talos tab of the job info pane has no data in it. I suspect this is a treeherder specific issue for the talos tab which could stem from the log parsing failure. I see this in the log file (http://archive.mozilla.org/pub/firefox/try-builds/j.parkouss@gmail.com-dcde84172c6d66b1f952ab0e6589d8a0ecea5185/try-linux64/try_ubuntu64_hw_test-chromez-bm103-tests1-linux-build1429.txt.gz): 15:28:59 INFO - 2015-11-10 15:28:59,525 INFO : PERFHERDER_DATA: {"framework": {"name": "talos"}, "suites": [{"subtests": [{"replicates": [18.316116666666687, 18.337366666666682, 18.90833333333333, 18.977333333333334, 18.72553333333333, 19.3645833333334, 17.87751666666667, 20.129883333333318, 19.62941666666671, 17.727200000000018, 17.79053333333334, 19.006166666666658, 19.046116666666677, 19.20368333333333, 18.742050000000003, 16.584133333333316, 16.829499999999985, 19.005433333333336, 16.419150000000034, 18.534716666666675], "name": "tresize", "value": 18.742050000000003}], "name": "tresize", "value": 18.742050000000003}, {"lowerIsBetter": false, "subtests": [{"lowerIsBetter": false, "replicates": [423.0, 401.0, 403.0, 418.0, 398.0], "unit": "score", "name": "Asteroids - Bitmaps", "value": 402.0}, {"lowerIsBetter": false, "replicates": [754.0, 711.0, 775.0, 760.0, 778.0], "unit": "score", "name": "Asteroids - Vectors", "value": 767.5}, {"lowerIsBetter": false, "replicates": [1103.0, 917.0, 979.0, 1017.0, 1071.0], "unit": "score", "name": "Asteroids - Bitmaps- shapes- text", "value": 998.0}, {"lowerIsBetter": false, "replicates": [747.0, 720.0, 731.0, 766.0, 750.0], "unit": "score", "name": "Asteroids - Shapes- shadows- blending", "value": 740.5}, {"lowerIsBetter": false, "replicates": [851.0, 799.0, 843.0, 908.0, 850.0], "unit": "score", "name": "Arena5 - Vectors- shadows- bitmaps- text", "value": 846.5}, {"lowerIsBetter": false, "replicates": [652.0, 632.0, 661.0, 660.0, 688.0], "unit": "score", "name": "Plasma - Maths- canvas shapes", "value": 660.5}, {"lowerIsBetter": false, "replicates": [413.0, 438.0, 425.0, 427.0, 424.0], "unit": "score", "name": "3D Rendering - Maths- polygons- image transforms", "value": 426.0}, {"lowerIsBetter": false, "replicates": [1420.0, 1495.0, 1474.0, 1471.0, 1362.0], "unit": "score", "name": "Pixel blur - Math- getImageData- putImageData", "value": 1472.5}], "name": "tcanvasmark", "value": 6313.5}]} nothing is jumping out to me as invalid here, :wlach do you see anything with this above output why it would cause these treeherder issues?
Flags: needinfo?(wlachance)
Comment 24•5 years ago
|
||
I don't see any results posted to graph server or treeherder, so something here must have removed the upload to graph server.
Comment 25•5 years ago
|
||
I also see that we remove the options from the list- in fact this probably means we can completely remove the options collection hash from perfherder code
Comment 26•5 years ago
|
||
Comment on attachment 8685046 [details] MozReview Request: Bug 1222588 - talos should output PERFHERDER_DATA instead of TALOS_DATA. r=jmaher sadly this is a r- until we can figure out why this is not uploading data to graph server and printing data out in treeherder (log parsing and test summary results)
Attachment #8685046 -
Flags: review?(jmaher) → review-
Assignee | ||
Comment 27•5 years ago
|
||
Yep, sadly that patch does not work. I even ran another no-op push to try to be sure: https://treeherder.mozilla.org/#/jobs?repo=try&revision=fc489ef4979d And here treeherder show results. :jmaher, does it also post to graphserver ? I can understand that data is not well formatted for perfherder (don't know why though), but it is hard to believe that this patch broke posting to graphserver.
Comment 28•5 years ago
|
||
the last data points on the graph: http://graphs.mozilla.org/graph.html#tests=[[223,23,43],[223,23,35]]&sel=1446657974318,1447262774318&displayrange=7&datatype=geo are for the no-op try push. So yes, we do post successfully to graph server without the patch. One thing to look at would be running a test locally with[out] the patch and looking at local.out- we could compare two versions of the file and determine what is different!
Comment 29•5 years ago
|
||
Oops, there's a bug in perfherder ingestion that would cause it to truncate lines, which means we wouldn't parse the json. I'll fix it, hang on!
Flags: needinfo?(wlachance)
Comment 30•5 years ago
|
||
Attachment #8686173 -
Flags: review?(jmaher)
Comment 31•5 years ago
|
||
Comment on attachment 8686173 [details] [review] PR oh, yeah- we should do this!
Attachment #8686173 -
Flags: review?(jmaher) → review+
Comment 32•5 years ago
|
||
Commit pushed to master at https://github.com/mozilla/treeherder https://github.com/mozilla/treeherder/commit/16c14a580f47075591c01412f9c40a41af4178c7 Bug 1222588 - Logparser shouldn't truncate lines with PERFHERDER_DATA in them I changed the magic string from PERFORMANCE_DATA to PERFHERDER_DATA before landing bug 1149164, but forgot to fix this particular part. Updated unit test so this hopefully won't happen again
Comment 33•5 years ago
|
||
ok, the retriggers worked for perfherder, as for the graph server we are still generating valid data, there must be something odd going on why we are not seeing the actual data points in the graphs.
Assignee | ||
Comment 34•5 years ago
|
||
Yes. Looking at the logs, we have everything graphserver related, including the response from the server, like: > 15:28:59 INFO - 2015-11-10 15:28:59,436 INFO : Posting result 0 of 2 to http://graphs.mozilla.org/server/collect.cgi, attempt 0 > 15:28:59 INFO - 2015-11-10 15:28:59,493 DEBUG : process_Request line: tresize graph.html#tests=[[254,23,35]] > 15:28:59 INFO - 2015-11-10 15:28:59,493 DEBUG : process_Request line: tresize 18.43 graph.html#tests=[[254,23,35]] > 15:28:59 INFO - 2015-11-10 15:28:59,493 INFO : Posting result 1 of 2 to http://graphs.mozilla.org/server/collect.cgi, attempt 0 > 15:28:59 INFO - 2015-11-10 15:28:59,524 DEBUG : process_Request line: tcanvasmark 6313.50 graph.html#tests=[[289,23,35]] ... > 15:28:59 INFO - RETURN: <a href='http://graphs.mozilla.org/graph.html#tests=[[254,23,35]]'>tresize: 18.43</a> > 15:28:59 INFO - RETURN: <a href='http://graphs.mozilla.org/graph.html#tests=[[289,23,35]]'>tcanvasmark: 6313.50</a> > 15:28:59 INFO - TinderboxPrint: TalosResult: {"graphserver": {"tcanvasmark": {"url": "http://graphs.mozilla.org/graph.html#tests=[[289,23,35]]", "result": "6313.50"}, "tresize": {"url": "http://graphs.mozilla.org/graph.html#tests=[[254,23,35]]", "result": "18.43"}}} I don't get it. I am going to add some debug to see the raw POST message that is sent, and compare that with the one working. Other ideas ?
Assignee | ||
Comment 35•5 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=46f03a25a15f
Assignee | ||
Comment 36•5 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=94836eea6be8
Assignee | ||
Comment 37•5 years ago
|
||
Oh! Looks like this time it appear on the Talos panel of treeherder ?! https://treeherder.mozilla.org/#/jobs?repo=try&revision=46f03a25a15f Ok, I'm going to push a more complete try, again. Without the debug commit, I don't see why that should change anything.
Assignee | ||
Comment 38•5 years ago
|
||
https://treeherder.mozilla.org/#/jobs?repo=try&revision=9d1687cbc212
Comment 40•5 years ago
|
||
the prolem is I don't see it on graph server :( take linux64 a11y: http://graphs.mozilla.org/graph.html#tests=[[223,23,43]]&sel=none&displayrange=7&datatype=geo the only data point for Sunday, November 15 is: https://treeherder.mozilla.org/#/jobs?repo=try&revision=914186cc867da8cb940d540b32aa9e1bb61cb1a5&filter-searchStr=talos I wonder if we are not posting the revision or some other important meta information?
Flags: needinfo?(jmaher)
Assignee | ||
Comment 41•5 years ago
|
||
Arg, sorry I thought if treeherder was displaying some data (it was not before) it would be ok. :( So I'm going to look at the raw post messages and try to find something meaningful in there.
Assignee | ||
Comment 42•5 years ago
|
||
The two post messages sent: https://pastebin.mozilla.org/8852364 I don't see any obvious difference. Maybe the test title, that is "talos-linux64-ix-068" for the broken version against "talos-linux64-ix-031" for the good one. I don't know if that is important.
Comment 43•5 years ago
|
||
in that pastebin, the two revisions, I just see a09b19fadb8590d072c84e8808d6e955be68e560 in the graph server, not 0768198546da3b2566e7a5031c45c21db9557e9b. what is even stranger is the data is in the database, lets look at tresize from the try pushes: http://graphs.mozilla.org/api/test/runs/?id=254&branchid=23&platformid=35&days=5 as you can easily find in there, the 2 revisions we care about: [54998114,[1244502,20151115022847,"0768198546da3b2566e7a5031c45c21db9557e9b"],1447585819,18.465299999999999,0,null,7637,18.555299999999999] [54998110,[1244500,20151115022655,"a09b19fadb8590d072c84e8808d6e955be68e560"],1447585820,18.483699999999999,0,null,7711,18.5639] the dates, revisions, and runids look fine, so do the runtimes and averages and the geomeans (geomeans are the last values, i.e. 18.5639). I do notice that we report 18.56 for both runs in the log files. this got me thinking and I looked at canvasmark- we have that on the graph: http://graphs.mozilla.org/graph.html#tests=[[289,23,35]]&sel=1447313076553.7705,1447763342701.6304,5818.803314469818,7494.016989683493&displayrange=7&datatype=geo canvasmark is posted as a type AVERAGE, whereas tresize is a type VALUE. This makes me wonder if we are just having issues with one type of test, or maybe hitting a random bug in graph server. oh crap- I just realized the data is on the graph, but it is 100% overlapping, in this case zooming way way way way way in, allows me to see the slight difference for each point. I vote for landing this!
Assignee | ||
Comment 44•5 years ago
|
||
Comment on attachment 8685046 [details] MozReview Request: Bug 1222588 - talos should output PERFHERDER_DATA instead of TALOS_DATA. r=jmaher Review request updated; see interdiff: https://reviewboard.mozilla.org/r/24717/diff/2-3/
Attachment #8685046 -
Flags: review?(jmaher)
Attachment #8685046 -
Flags: review-
Attachment #8685046 -
Flags: feedback+
Updated•5 years ago
|
Attachment #8685046 -
Flags: review?(jmaher) → review+
Comment 45•5 years ago
|
||
Comment on attachment 8685046 [details] MozReview Request: Bug 1222588 - talos should output PERFHERDER_DATA instead of TALOS_DATA. r=jmaher https://reviewboard.mozilla.org/r/24717/#review22845 cool, this is good.
Assignee | ||
Comment 46•5 years ago
|
||
https://hg.mozilla.org/integration/mozilla-inbound/rev/fe34d01fb2ecb2dd4cda82e788cf7b541d5cbdb4 Bug 1222588 - talos should output PERFHERDER_DATA instead of TALOS_DATA. r=jmaher
Comment 47•5 years ago
|
||
This is untested but should fix tp5o ingestion.
Attachment #8688519 -
Flags: review?(jmaher)
Attachment #8688519 -
Flags: feedback?(j.parkouss)
Comment 48•5 years ago
|
||
Comment on attachment 8688519 [details] [diff] [review] Fix counter ingestion in Perfherder Review of attachment 8688519 [details] [diff] [review]: ----------------------------------------------------------------- I am fine landing this as is.
Attachment #8688519 -
Flags: review?(jmaher) → review+
Comment 49•5 years ago
|
||
https://hg.mozilla.org/integration/mozilla-inbound/rev/06188c6531b5b06fc9bd2cdcd25eaf5c1b778882 Bug 1222588 - Fix counter ingestion in Perfherder;r=jmaher
Assignee | ||
Comment 50•5 years ago
|
||
Comment on attachment 8688519 [details] [diff] [review] Fix counter ingestion in Perfherder oups, a bit late here, but I said f+ on irc anyway. :)
Attachment #8688519 -
Flags: feedback?(j.parkouss) → feedback+
Comment 51•5 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/fe34d01fb2ec https://hg.mozilla.org/mozilla-central/rev/06188c6531b5
Status: NEW → RESOLVED
Closed: 5 years ago
status-firefox45:
--- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla45
Comment 52•5 years ago
|
||
bugherderuplift |
https://hg.mozilla.org/releases/mozilla-b2g44_v2_5/rev/2388bca84225 https://hg.mozilla.org/releases/mozilla-b2g44_v2_5/rev/70abb738e1f8
status-b2g-v2.5:
--- → fixed
You need to log in
before you can comment on or make changes to this bug.
Description
•