1527154 - Missing dromaeo_dom reports

Status: RESOLVED WORKSFORME

(Testing :: Talos, enhancement, P3)

Description

[Steve Fink [:sfink] [:s:]]

I was attempting to compare https://treeherder.mozilla.org/perf.html#/compare?originalProject=try&originalRevision=098c23e413e52bd0383cbf82e1fcc818ccd60a59&newProject=try&newRevision=13c2c6426ba6a76feba7ad5c98ae49c8284434b6&framework=1

perfherder claims that there are zero reports for the test in question, the subtest dom-attr of dromaeo_dom: see eg https://treeherder.mozilla.org/perf.html#/comparesubtest?originalProject=try&newProject=try&newRevision=13c2c6426ba6a76feba7ad5c98ae49c8284434b6&originalSignature=1656944&newSignature=1656944&framework=1&selectedTimeRange=172800

Treeherder does show results for both builds, eg:

https://treeherder.mozilla.org/#/jobs?repo=try&revision=098c23e413e52bd0383cbf82e1fcc818ccd60a59&selectedJob=227376031

and

https://treeherder.mozilla.org/#/jobs?repo=try&revision=13c2c6426ba6a76feba7ad5c98ae49c8284434b6&selectedJob=227374461

One potential issue is that I removed all the other dromaeo_dom subtests: https://hg.mozilla.org/try/rev/59776410b0bf000a7ce79e3046a26d44e93a1a13

but it still shows zero results for that subtest, which seems odd.

Comment 1

[Steve Fink [:sfink] [:s:]]

Also, in the log file for one of those runs:

PERFHERDER_DATA: {"framework": {"name": "talos"}, "suites": [{"lowerIsBetter": false, "subtests": [{"value": 3275.763762745078, "name": "", "lowerIsBetter": false, "alertThreshold": 5.0, "replicates": [16896.32414703412, 17115.657686846254, 17174.656506869862, 17198.0, 17245.3101875925, 8492.0, 8538.146185381467, 8545.145485451452, 8597.3122150228, 8612.138786121384, 867.7917299848039, 875.9299256059516, 876.4741155306813, 876.8597024476085, 885.4156256870467, 1442.192372271526, 1463.8536146385356, 1471.9705605887875, 1472.6760112775225, 1475.9409623615043, 2353.482233908537, 2358.8584684918874, 2373.860546937471, 2376.4296568823443, 2382.380581048931, 2723.891044358223, 2757.1177223288555, 2768.280247135749, 2794.2176190666546, 2798.8320700757918], "shouldAlert": false, "unit": "score"}], "extraOptions": ["e10s", "stylo"], "name": "dromaeo_dom", "alertThreshold": 5.0}]}

That's at https://taskcluster-artifacts.net/MYaWtCeOQdm2KVN-D0OF8w/0/public/test_info/perfherder-data.json too.

Uh, maybe that answers the question here -- suites[0].subtests[0].name is the empty string. I'm guessing it should be "dom-attr.html"? But I don't know why it would be empty.

Comment 2

[Steve Fink [:sfink] [:s:]]

Ok, https://taskcluster-artifacts.net/ab0JVFsRRgOIYAcvWdlwYw/0/public/test_info/perfherder-data.json (a randomly chosen run) says it should be "attr.html". Any idea why it wouldn't be filled in?

Comment 3

[Ionuț Goldan [:igoldan]]

(In reply to Steve Fink [:sfink] [:s:] from comment #2)

Ok, https://taskcluster-artifacts.net/ab0JVFsRRgOIYAcvWdlwYw/0/public/test_info/perfherder-data.json (a randomly chosen run) says it should be "attr.html". Any idea why it wouldn't be filled in?

Robert, could you provide some assistance regarding this Talos behavior?

Comment 4

[Robert Wood [:rwood]]

I don't know much about the dromeao tests but I think this will help clarify - from the Talos wiki [1]:

"subtest: Dromaeo is a custom benchmark which has a lot of micro tests inside each subtest, because of this we use a custom dromaeo filter to summarize the subtest. Each micro test produces 5 data points and for each 5 data points we take the mean, leaving 36 data points to represent the subtest (assuming 180 points). These 36 micro test means, are then run through a geometric_mean to produce a single number for the dromaeo subtest. source: filter.py"

[1] https://wiki.mozilla.org/Performance_sheriffing/Talos/Tests#Dromaeo_CSS

Note that :bz is the test owner, in case you have further questions. Hope that helps!

Comment 5

[Andrej (:aglavic)]

Hi Steve,
I just wanted to follow up with you and see if you have been experiencing this issue recently?

Comment 6

[Steve Fink [:sfink] [:s:]]

I have not, but I also haven't looked at dromaeo_dom since I filed this. I just checked a recent run of it, though, and the perfherder-data.json file does not seem to contain any empty names. So perhaps it was resolved?

Comment 7

[Greg Mierzwinski [:sparky]]

That's great, we haven't seen any issues related to empty names either. It may have been a rare intermittent.