1222588 - talos should output PERFHERDER_DATA instead of TALOS_DATA

Status: RESOLVED FIXED

(Testing :: Talos, defect)

Description

[Julien Pagès (:parkouss)]

So basically perferder now have a generic way to get perf data from jobs, the PERFHERDER_DATA line (format specification here: https://github.com/wlach/treeherder/tree/master/schemas, with an example here: https://bugzilla.mozilla.org/show_bug.cgi?id=1222549#c1).

Currently talos do not use that and output his own TALOS_DATA line - with a talos specific format.We should get rid of that, i.e. make talos just output PERFHERDER_DATA instead.

Comment 1

[Julien Pagès (:parkouss)]

Attachment: talos data exported a la PERFHERDER_DATA

Comment 2

[Julien Pagès (:parkouss)]

Attachment: current talos data format

Comment 3

[Julien Pagès (:parkouss)]

I started to look at this, see attachment 8684559 [details] for a result, and attachment 8684560 [details] for a comparison with current data format.

So basically we are loosing a lot of data:

 - each point value
 - machine name, os, platform
 - run date, talos options used

Do we want that in the new data format ? What do we need exactly ? I took the choice to only export resulting values, with the addition of unit and lowerIsBetter information. But I am not sure what is needed exactly - and in which format. For example the PERFHERDER_DATA specification do not allow a list or raw data values (but I believe we can add a key like raw_points in subtest if we want that?).

Also, I am not sure about the counters. They often have two values (mean and max) I believe mean must be the default one (ie the "value" in each subtest/suite). Do we want max to be exposed too ?

Sorry, plenty of questions in only one comment. ;;

Comment 4

[William Lachance (:wlach) (use needinfo!)]

(In reply to Julien Pagès (:parkouss) from comment #3)
> I started to look at this, see attachment 8684559 [details] for a result,
> and attachment 8684560 [details] for a comparison with current data format.
> 
> So basically we are loosing a lot of data:
> 
>  - each point value

Yes, I think as I said earlier, we should provide an optional field in PERFHERDER_DATA called "replicates" for each subtest, where this can be included. We don't need to ingest it.

>  - machine name, os, platform

This isn't needed, it's included in the job data ingested elsewhere by treeherder.

>  - run date, talos options used

Likewise.

> Also, I am not sure about the counters. They often have two values (mean and
> max) I believe mean must be the default one (ie the "value" in each
> subtest/suite). Do we want max to be exposed too ?

So currently we're only ingesting "mean" in Perfherder:

https://github.com/mozilla/treeherder/blob/master/treeherder/etl/perf.py#L263

So I think it makes sense to just put that into the PERFHERDER_DATA structure for now, to preserve the existing behaviour.
 
> Sorry, plenty of questions in only one comment. ;;

Np! Keep on asking. :)

Comment 5

[Julien Pagès (:parkouss)]

Attachment: perfherder_data.patch

Well, my first attempt to this.

I made some decisions, tell me if anything needs a change.

 - framework name is always "talos"

for counters (example log in [1]):
 - each counter is a suite
 - the name is prefixed with 'counter_'
 - the value is the mean
 - no subtests field

for non counters (example in [2])
 - each suite is a talos test (e.g, tp5o)
 - each page inside a talos test is a subtest (e.g, media.photobucket.com)
 - each suite has at least one subtest
 - suite and subtest hold the attribute "lowerIsBetter" when required
 - each subtest hold the unit field when required
 - subtest value is the previous "filtered" value
 - each suite has a "replicates" field which looks like the previously named "result" field (contains the raw list of values)

Pushed to try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=75dcc769d8c4

Note that some counters are listed without any "value" field, this is fixed in the attached patch.
Also note that the patch do NOT replace TALOSDATA - it just add a new PERFHERDER_DATA, as I believe we want to have both until we made the according changes to perfherder. Tell me if my assumption is wrong here (and what is the plan in such a case).

[1]: http://archive.mozilla.org/pub/firefox/try-builds/[e-mail]-75dcc769d8c414dcedc191a90dd5ab61c05e3ed1/try-win32/try_win7-ix_test-xperf-bm126-tests1-windows-build3.txt.gz
[2]: http://archive.mozilla.org/pub/firefox/try-builds/[e-mail]-75dcc769d8c414dcedc191a90dd5ab61c05e3ed1/try-win32/try_win7-ix_test-tp5o-bm126-tests1-windows-build2.txt.gz

Comment 6

[Julien Pagès (:parkouss)]

Attachment: tp5o.thdata.json

For convenience, the data exported and formatted from tp5o build.

Comment 7

[Julien Pagès (:parkouss)]

Oh, I forgot to say that the patch is basically a copy/paste of PerfherderOutput.__call__ method - with a few adaptations for the new format.

I had some headache trying to understand all that code - finally I must say I gave up, some parts are here, I can't remember/understand why :(. Next great step would be to make that function understandable if you agree with me that it is hard to read.

Comment 8

[William Lachance (:wlach) (use needinfo!)]

(In reply to Julien Pagès (:parkouss) from comment #5)
> Created attachment 8684678 [details] [diff] [review]
> perfherder_data.patch
> 
> Well, my first attempt to this.
> 
> I made some decisions, tell me if anything needs a change.

Thanks for looking into this so quickly.

>  - framework name is always "talos"

Yup! This is right.

> for counters (example log in [1]):
>  - each counter is a suite
>  - the name is prefixed with 'counter_'
>  - the value is the mean
>  - no subtests field

No, this isn't the right organization IMO. We've made the counters subtests of the suite in which they're run, but defined seperately from the *actual* subtests, and not given a summary value (so they're displayed individually in the compare view). Here's an example from https://treeherder.mozilla.org/api/project/mozilla-inbound/performance/signatures/

    "d22e336c3614c2b16d5e9b238ba67c087a6eeb85": {
        "test": "Main_RSS",
        "suite": "tp5o",
        "option_collection_hash": "102210fe594ee9b33d82058545b1ed14f4c8206e",
        "test_options": [
            "e10s"
        ],
        "machine_platform": "linux64"
    }

So for example, in tp5o, you'd have something like:

suites: [ 
  { 'name': 'tp5o', value: 123, subtests: [ <actual subtests of tp5o> ] },
  { 'name': 'tp5o', subtests: [ {name: Main_RSS, value: 456 }, ... ] }
]

Using the mean as the value for the counters is correct, however.

> for non counters (example in [2])
>  - each suite is a talos test (e.g, tp5o)
>  - each page inside a talos test is a subtest (e.g, media.photobucket.com)
>  - each suite has at least one subtest
>  - suite and subtest hold the attribute "lowerIsBetter" when required
>  - each subtest hold the unit field when required
>  - subtest value is the previous "filtered" value
>  - each suite has a "replicates" field which looks like the previously named
> "result" field (contains the raw list of values)

This all sounds right.

> Pushed to try:
> https://treeherder.mozilla.org/#/jobs?repo=try&revision=75dcc769d8c4
> 
> Note that some counters are listed without any "value" field, this is fixed
> in the attached patch.
> Also note that the patch do NOT replace TALOSDATA - it just add a new
> PERFHERDER_DATA, as I believe we want to have both until we made the
> according changes to perfherder. Tell me if my assumption is wrong here (and
> what is the plan in such a case).

No, we'll need to take out TALOSDATA otherwise we'll get double results for each test (since we're still parsing Talos by default). I don't see any reason to print both, at least once we've verified that we're getting the same values. :)

Comment 9

[William Lachance (:wlach) (use needinfo!)]

> (In reply to Julien Pagès (:parkouss) from comment #5)
> >  - each suite has a "replicates" field which looks like the previously named
> > "result" field (contains the raw list of values)

Actually it would be good if you could associate the replicates with the corresponding subtests, instead of making them a property of the suite.

Comment 10

[William Lachance (:wlach) (use needinfo!)]

(In reply to Julien Pagès (:parkouss) from comment #7)
> Oh, I forgot to say that the patch is basically a copy/paste of
> PerfherderOutput.__call__ method - with a few adaptations for the new format.
> 
> I had some headache trying to understand all that code - finally I must say
> I gave up, some parts are here, I can't remember/understand why :(. Next
> great step would be to make that function understandable if you agree with
> me that it is hard to read.

Yeah, I'd be totally fine with you completely rewriting this. :) As I said earlier, we can scrap TALOSDATA. Feel free to start with a blank slate.

Comment 11

[William Lachance (:wlach) (use needinfo!)]

Comment on attachment 8684678 [details] [diff] [review]
perfherder_data.patch

Review of attachment 8684678 [details] [diff] [review]:
-----------------------------------------------------------------

Going to give this f+, but it needs some adjustment/refinement before it goes in.

Comment 12

[Joel Maher ( :jmaher ) (UTC -0800)]

Comment on attachment 8684678 [details] [diff] [review]
perfherder_data.patch

will hold off until feedback is updated.

Comment 13

[Julien Pagès (:parkouss)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=14e7966e7186

Comment 14

[Julien Pagès (:parkouss)]

Oups, I forgot to update the mozharness part that still check for the TALOSDATA line...

Comment 15

[Julien Pagès (:parkouss)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=0b54cfa0f885

Comment 16

[Julien Pagès (:parkouss)]

Attachment: MozReview Request: Bug 1222588 - talos should output PERFHERDER_DATA instead of TALOS_DATA. r=jmaher

Bug 1222588 - talos should output PERFHERDER_DATA instead of TALOS_DATA. r=jmaher

Comment 17

[Joel Maher ( :jmaher ) (UTC -0800)]

Comment on attachment 8685046 [details]
MozReview Request: Bug 1222588 - talos should output PERFHERDER_DATA instead of TALOS_DATA. r=jmaher

https://reviewboard.mozilla.org/r/24717/#review22255

overall I really like this patch, I see it cleaning up a lot.  When we stop publishing to graph server we can remove some command line arguments to talos and stop tracking them (like branch, revision, machine, etc.)!  A few questions below.

::: testing/talos/talos/output.py:483
(Diff revision 1)
> +                            'replicates': replicates[page],

does this 'replicates' field work for non pageloader tests (like ts_paint)?

::: testing/talos/talos/output.py:514
(Diff revision 1)
> +                    counter_subtests.append(subtest)

can we do:
counter_subtests.append(subtest)
subtest.modify...

and then have that subtest value updated inside of counter_subtests?  If not, then we should append it after adding the value.

::: testing/talos/talos/output.py:519
(Diff revision 1)
>                          # calculate mean and max value

here we are only calculating the mean.  I know this fits into our scenario well, but the max does have significant value as if we have new max values it could be a sign of something.

Maybe a new bug to evaluate counters and determine if we want to duplicate the counters and track the max value as well- then we could insert as a unique counter and generate alerts on the max value.

Comment 18

[Julien Pagès (:parkouss)]

For the replicates, looking at http://archive.mozilla.org/pub/firefox/try-builds/[e-mail]-a0aa3ce3b5d575132499fb385e12ef603591e66d/try-win32/try_win7-ix_test-other-bm119-tests1-windows-build7.txt.gz:

> {
>             "name": "ts_paint",
>             "subtests": [
>                 {
>                     "name": "ts_paint",
>                     "replicates": [
>                         1434.0,
>                         1077.0,
>                         1018.0,
>                         1030.0,
>                         1031.0,
>                         1064.0,
>                         1042.0,
>                         1035.0,
>                         1029.0,
>                         1117.0,
>                         1036.0,
>                         1049.0,
>                         1023.0,
>                         1053.0,
>                         1061.0,
>                         1058.0,
>                         1062.0,
>                         1092.0,
>                         1051.0,
>                         1042.0
>                     ],
>                     "value": 1049.0
>                 }
>             ],
>             "value": 1049.0
>         },

it looks ok to me.

I will make the change for point 2, removing a variable not necessary seems good to me.

For the last point, I asked before and :wlach answered in comment 4: for now, perfherder only digest "mean", this is why max is not taken into account. Maybe we could add that inside the subtest counter, ie:

suites: [ 
  { 'name': 'tp5o', value: 123, subtests: [ <actual subtests of tp5o> ] },
  { 'name': 'tp5o', subtests: [ {name: Main_RSS, value: 456, max: 502 }, ... ] }
]

Since we already add some extra keys (lowerIsBetter, unit) in subtests that are not counters.

Comment 19

[Julien Pagès (:parkouss)]

hm, something strange though - ts_paint filters are "ignore first" + "mean" but if I calculate it manually it is slightly different:

>>> l = [1077.0, 1018.0, 1030.0, 1031.0, 1064.0, 1042.0, 1035.0, 1029.0, 1117.0, 1036.0, 1049.0, 1023.0, 1053.0, 1061.0, 1058.0, 1062.0, 1092.0, 1051.0, 1042.0]
>>> float(sum(l))/len(l)
1051.0526315789473

(Note I removed the first value)

But this does not seem to be related to this patch - If I look into http://archive.mozilla.org/pub/firefox/try-builds/[e-mail]-75dcc769d8c414dcedc191a90dd5ab61c05e3ed1/try-win32/try_win7-ix_test-other-bm126-tests1-windows-build9.txt.gz, in the TALOSDATA section, I can see something similar:

>>> l = [1058.0, 1048.0, 1023.0, 1032.0, 1050.0, 1007.0, 1052.0, 1055.0, 1107.0, 1073.0, 1024.0, 1044.0, 1057.0, 1049.0, 1062.0, 1052.0, 1100.0, 1043.0, 1040.0]
>>> float(sum(l))/len(l)
1051.3684210526317

(Note I removed the first value)

And the value/filtered are 1050.0.

Comment 20

[Julien Pagès (:parkouss)]

Comment on attachment 8685046 [details]
MozReview Request: Bug 1222588 - talos should output PERFHERDER_DATA instead of TALOS_DATA. r=jmaher

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/24717/diff/1-2/

Comment 21

[William Lachance (:wlach) (use needinfo!)]

Comment on attachment 8685046 [details]
MozReview Request: Bug 1222588 - talos should output PERFHERDER_DATA instead of TALOS_DATA. r=jmaher

This looks good to me. It looks like the logparser failed during the try run though, could you do another?

Comment 22

[Julien Pagès (:parkouss)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=284b3d771c61

Comment 23

[Joel Maher ( :jmaher ) (UTC -0800)]

the log parser is still failing and the talos tab of the job info pane has no data in it.  I suspect this is a treeherder specific issue for the talos tab which could stem from the log parsing failure.

I see this in the log file (http://archive.mozilla.org/pub/firefox/try-builds/[e-mail]-dcde84172c6d66b1f952ab0e6589d8a0ecea5185/try-linux64/try_ubuntu64_hw_test-chromez-bm103-tests1-linux-build1429.txt.gz):
15:28:59     INFO -  2015-11-10 15:28:59,525 INFO : PERFHERDER_DATA: {"framework": {"name": "talos"}, "suites": [{"subtests": [{"replicates": [18.316116666666687, 18.337366666666682, 18.90833333333333, 18.977333333333334, 18.72553333333333, 19.3645833333334, 17.87751666666667, 20.129883333333318, 19.62941666666671, 17.727200000000018, 17.79053333333334, 19.006166666666658, 19.046116666666677, 19.20368333333333, 18.742050000000003, 16.584133333333316, 16.829499999999985, 19.005433333333336, 16.419150000000034, 18.534716666666675], "name": "tresize", "value": 18.742050000000003}], "name": "tresize", "value": 18.742050000000003}, {"lowerIsBetter": false, "subtests": [{"lowerIsBetter": false, "replicates": [423.0, 401.0, 403.0, 418.0, 398.0], "unit": "score", "name": "Asteroids - Bitmaps", "value": 402.0}, {"lowerIsBetter": false, "replicates": [754.0, 711.0, 775.0, 760.0, 778.0], "unit": "score", "name": "Asteroids - Vectors", "value": 767.5}, {"lowerIsBetter": false, "replicates": [1103.0, 917.0, 979.0, 1017.0, 1071.0], "unit": "score", "name": "Asteroids - Bitmaps- shapes- text", "value": 998.0}, {"lowerIsBetter": false, "replicates": [747.0, 720.0, 731.0, 766.0, 750.0], "unit": "score", "name": "Asteroids - Shapes- shadows- blending", "value": 740.5}, {"lowerIsBetter": false, "replicates": [851.0, 799.0, 843.0, 908.0, 850.0], "unit": "score", "name": "Arena5 - Vectors- shadows- bitmaps- text", "value": 846.5}, {"lowerIsBetter": false, "replicates": [652.0, 632.0, 661.0, 660.0, 688.0], "unit": "score", "name": "Plasma - Maths- canvas shapes", "value": 660.5}, {"lowerIsBetter": false, "replicates": [413.0, 438.0, 425.0, 427.0, 424.0], "unit": "score", "name": "3D Rendering - Maths- polygons- image transforms", "value": 426.0}, {"lowerIsBetter": false, "replicates": [1420.0, 1495.0, 1474.0, 1471.0, 1362.0], "unit": "score", "name": "Pixel blur - Math- getImageData- putImageData", "value": 1472.5}], "name": "tcanvasmark", "value": 6313.5}]}

nothing is jumping out to me as invalid here, :wlach do you see anything with this above output why it would cause these treeherder issues?

Comment 24

[Joel Maher ( :jmaher ) (UTC -0800)]

I don't see any results posted to graph server or treeherder, so something here must have removed the upload to graph server.

Comment 25

[Joel Maher ( :jmaher ) (UTC -0800)]

I also see that we remove the options from the list- in fact this probably means we can completely remove the options collection hash from perfherder code

Comment 26

[Joel Maher ( :jmaher ) (UTC -0800)]

Comment on attachment 8685046 [details]
MozReview Request: Bug 1222588 - talos should output PERFHERDER_DATA instead of TALOS_DATA. r=jmaher

sadly this is a r- until we can figure out why this is not uploading data to graph server and printing data out in treeherder (log parsing and test summary results)

Comment 27

[Julien Pagès (:parkouss)]

Yep, sadly that patch does not work. I even ran another no-op push to try to be sure:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=fc489ef4979d

And here treeherder show results. :jmaher, does it also post to graphserver ? I can understand that data is not well formatted for perfherder (don't know why though), but it is hard to believe that this patch broke posting to graphserver.

Comment 28

[Joel Maher ( :jmaher ) (UTC -0800)]

the last data points on the graph:
http://graphs.mozilla.org/graph.html#tests=[[223,23,43],[223,23,35]]&sel=1446657974318,1447262774318&displayrange=7&datatype=geo

are for the no-op try push. So yes, we do post successfully to graph server without the patch.

One thing to look at would be running a test locally with[out] the patch and looking at local.out- we could compare two versions of the file and determine what is different!

Comment 29

[William Lachance (:wlach) (use needinfo!)]

Oops, there's a bug in perfherder ingestion that would cause it to truncate lines, which means we wouldn't parse the json. I'll fix it, hang on!

Comment 30

[William Lachance (:wlach) (use needinfo!)]

Attachment: PR

Comment 31

[Joel Maher ( :jmaher ) (UTC -0800)]

Comment on attachment 8686173 [details] [review]
PR

oh, yeah- we should do this!

Comment 32

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/16c14a580f47075591c01412f9c40a41af4178c7
Bug 1222588 - Logparser shouldn't truncate lines with PERFHERDER_DATA in them

I changed the magic string from PERFORMANCE_DATA to PERFHERDER_DATA before
landing bug 1149164, but forgot to fix this particular part.

Updated unit test so this hopefully won't happen again

Comment 33

[Joel Maher ( :jmaher ) (UTC -0800)]

ok, the retriggers worked for perfherder, as for the graph server we are still generating valid data, there must be something odd going on why we are not seeing the actual data points in the graphs.

Comment 34

[Julien Pagès (:parkouss)]

Yes. Looking at the logs, we have everything graphserver related, including the response from the server, like:

> 15:28:59     INFO -  2015-11-10 15:28:59,436 INFO : Posting result 0 of 2 to http://graphs.mozilla.org/server/collect.cgi, attempt 0
> 15:28:59     INFO -  2015-11-10 15:28:59,493 DEBUG : process_Request line: tresize	graph.html#tests=[[254,23,35]]
> 15:28:59     INFO -  2015-11-10 15:28:59,493 DEBUG : process_Request line: tresize	18.43	graph.html#tests=[[254,23,35]]
> 15:28:59     INFO -  2015-11-10 15:28:59,493 INFO : Posting result 1 of 2 to http://graphs.mozilla.org/server/collect.cgi, attempt 0
> 15:28:59     INFO -  2015-11-10 15:28:59,524 DEBUG : process_Request line: tcanvasmark	6313.50	graph.html#tests=[[289,23,35]]
...
> 15:28:59     INFO -  RETURN: <a href='http://graphs.mozilla.org/graph.html#tests=[[254,23,35]]'>tresize: 18.43</a>
> 15:28:59     INFO -  RETURN: <a href='http://graphs.mozilla.org/graph.html#tests=[[289,23,35]]'>tcanvasmark: 6313.50</a>
> 15:28:59     INFO -  TinderboxPrint: TalosResult: {"graphserver": {"tcanvasmark": {"url": "http://graphs.mozilla.org/graph.html#tests=[[289,23,35]]", "result": "6313.50"}, "tresize": {"url": "http://graphs.mozilla.org/graph.html#tests=[[254,23,35]]", "result": "18.43"}}}

I don't get it. I am going to add some debug to see the raw POST message that is sent, and compare that with the one working. Other ideas ?

Comment 35

[Julien Pagès (:parkouss)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=46f03a25a15f

Comment 36

[Julien Pagès (:parkouss)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=94836eea6be8

Comment 37

[Julien Pagès (:parkouss)]

Oh! Looks like this time it appear on the Talos panel of treeherder ?!

https://treeherder.mozilla.org/#/jobs?repo=try&revision=46f03a25a15f

Ok, I'm going to push a more complete try, again. Without the debug commit, I don't see why that should change anything.

Comment 38

[Julien Pagès (:parkouss)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=9d1687cbc212

Comment 39

[Julien Pagès (:parkouss)]

Builds looks good to me. Joel ?

Comment 40

[Joel Maher ( :jmaher ) (UTC -0800)]

the prolem is I don't see it on graph server :(

take linux64 a11y:
http://graphs.mozilla.org/graph.html#tests=[[223,23,43]]&sel=none&displayrange=7&datatype=geo

the only data point for Sunday, November 15 is:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=914186cc867da8cb940d540b32aa9e1bb61cb1a5&filter-searchStr=talos

I wonder if we are not posting the revision or some other important meta information?

Comment 41

[Julien Pagès (:parkouss)]

Arg, sorry I thought if treeherder was displaying some data (it was not before) it would be ok. :(

So I'm going to look at the raw post messages and try to find something meaningful in there.

Comment 42

[Julien Pagès (:parkouss)]

The two post messages sent:

https://pastebin.mozilla.org/8852364

I don't see any obvious difference. Maybe the test title, that is "talos-linux64-ix-068" for the broken version against "talos-linux64-ix-031" for the good one.

I don't know if that is important.

Comment 43

[Joel Maher ( :jmaher ) (UTC -0800)]

in that pastebin, the two revisions, I just see a09b19fadb8590d072c84e8808d6e955be68e560 in the graph server, not 0768198546da3b2566e7a5031c45c21db9557e9b.

what is even stranger is the data is in the database, lets look at tresize from the try pushes:
http://graphs.mozilla.org/api/test/runs/?id=254&branchid=23&platformid=35&days=5

as you can easily find in there, the 2 revisions we care about:
[54998114,[1244502,20151115022847,"0768198546da3b2566e7a5031c45c21db9557e9b"],1447585819,18.465299999999999,0,null,7637,18.555299999999999]
[54998110,[1244500,20151115022655,"a09b19fadb8590d072c84e8808d6e955be68e560"],1447585820,18.483699999999999,0,null,7711,18.5639]

the dates, revisions, and runids look fine, so do the runtimes and averages and the geomeans (geomeans are the last values, i.e. 18.5639).  I do notice that we report 18.56 for both runs in the log files.

this got me thinking and I looked at canvasmark- we have that on the graph:
http://graphs.mozilla.org/graph.html#tests=[[289,23,35]]&sel=1447313076553.7705,1447763342701.6304,5818.803314469818,7494.016989683493&displayrange=7&datatype=geo

canvasmark is posted as a type AVERAGE, whereas tresize is a type VALUE.  This makes me wonder if we are just having issues with one type of test, or maybe hitting a random bug in graph server.

oh crap- I just realized the data is on the graph, but it is 100% overlapping, in this case zooming way way way way way in, allows me to see the slight difference for each point.

I vote for landing this!

Comment 44

[Julien Pagès (:parkouss)]

Comment on attachment 8685046 [details]
MozReview Request: Bug 1222588 - talos should output PERFHERDER_DATA instead of TALOS_DATA. r=jmaher

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/24717/diff/2-3/

Comment 45

[Joel Maher ( :jmaher ) (UTC -0800)]

Comment on attachment 8685046 [details]
MozReview Request: Bug 1222588 - talos should output PERFHERDER_DATA instead of TALOS_DATA. r=jmaher

https://reviewboard.mozilla.org/r/24717/#review22845

cool, this is good.

Comment 46

[Julien Pagès (:parkouss)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/fe34d01fb2ecb2dd4cda82e788cf7b541d5cbdb4
Bug 1222588 - talos should output PERFHERDER_DATA instead of TALOS_DATA. r=jmaher

Comment 47

[William Lachance (:wlach) (use needinfo!)]

Attachment: Fix counter ingestion in Perfherder

This is untested but should fix tp5o ingestion.

Comment 48

[Joel Maher ( :jmaher ) (UTC -0800)]

Comment on attachment 8688519 [details] [diff] [review]
Fix counter ingestion in Perfherder

Review of attachment 8688519 [details] [diff] [review]:
-----------------------------------------------------------------

I am fine landing this as is.

Comment 49

[William Lachance (:wlach) (use needinfo!)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/06188c6531b5b06fc9bd2cdcd25eaf5c1b778882
Bug 1222588 - Fix counter ingestion in Perfherder;r=jmaher

Comment 50

[Julien Pagès (:parkouss)]

Comment on attachment 8688519 [details] [diff] [review]
Fix counter ingestion in Perfherder

oups, a bit late here, but I said f+ on irc anyway. :)

Comment 51

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/fe34d01fb2ec
https://hg.mozilla.org/mozilla-central/rev/06188c6531b5

Comment 52

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/releases/mozilla-b2g44_v2_5/rev/2388bca84225
https://hg.mozilla.org/releases/mozilla-b2g44_v2_5/rev/70abb738e1f8