751975 - Talos Tp5 Responsiveness results are bimodal

Status: RESOLVED WORKSFORME

(Testing :: Talos, defect)

Description

[Jim Blandy :jimb]

Check out this graph:

http://graphs-new.mozilla.org/graph.html#tests=[[196,64,21]]&sel=1333963788963.2634,1334587516545.5562&displayrange=30&datatype=running

On April 11th, the results go bimodal: from that point forward, each result seems to fall into one of two distinct ranges, one ~90 and one ~60, at random. This seems to be the guilty changeset:

http://hg.mozilla.org/integration/fx-team/rev/51fe735d19a9

I don't understand the test; I just got mail from Talos, whom I suspect of harassing every author of a changeset immediately after one of those 60->90 transitions.

Looking at the changeset, it does have a "catch {}"; those always worry me, because they lose information. But I don't have any real insight to offer.

Comment 1

[Jim Blandy :jimb]

It's worth noting that, when you click on the "explode" button (the little plus in the circle at the top left), it'll show each bot's results separately. Since the lines cross the 60 / 90 gap pretty regularly, it doesn't seem to be a bot-specific issue.

Comment 2

[Tim Taubert [:ttaubert] (inactive)]

The data for this chart seems gone and IMO it's quite unlikely that it was caused by the aforementioned changeset. Is this still actionable?

Comment 3

[Matt Brubeck (:mbrubeck)]

The new version of this test, Tp5 Optimized Responsiveness, is still bimodal on at least some platforms, though I don't know whether the cause is still the same:
http://graphs.mozilla.org/graph.html#tests=[[275,131,31],[275,1,25],[275,1,1]]&sel=1344870855544,1376406855544&displayrange=365&datatype=running

Without looking closer at the data (which is now expired from the graph server), it's hard to say if it had anything to do with the changeset in comment 0, or if it was something caused by a slightly earlier changeset (like the parent, a merge from m-c to fx-team) that didn't manifest until that one.

Comment 4

[(not currently active) Ted Mielczarek]

The underlying mechanism used for measurements here is the same one used for the profiler's idea of "unresponsiveness": a background thread is spawned that fires native toolkit events at the main thread's event loop, timing how long they take to get serviced. When they are above a certain threshold a data point is printed. The Talos responsiveness number is some mathematical combination of these data points taken while loading the Tp5 pageset (I've forgotten the exact combination, I suspect it's documented somewhere).

It seems pretty plausible that some code could be running on some runs but not others that makes the browser less responsive, which would increase the number. I wouldn't be surprised to find that the same thing impacts other benchmarks.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

Any chance that this will be fixed by bug 904624?  Basically making this a floating point value instead of an int could reduce some of the noise.  For these values I am not sure if it will help much.

This is a Talos bug, I am not sure what the action items for Talos are, possibly this could be fixed in the browser or whatever area responsiveness is defined?

Comment 6

[(not currently active) Ted Mielczarek]

If anything this is likely just a browser bug--we're probably running some code that makes us less responsive. Figuring that out would be a start. I don't think this is actually a Talos bug.

Comment 7

[Matt Brubeck (:mbrubeck)]

Attachment: re-enable regression analysis

This stopped being bimodal in October 2013, at the same time as Ts Paint (bug 859571 comment 49).

Comment 8

[Matt Brubeck (:mbrubeck)]

http://hg.mozilla.org/graphs/rev/4af2d2e7dc8d (requires an analysis.cfg change in production)

Comment 9

[Chris AtLee [:catlee]]

deployed