480467 - Tp increase on mozilla-central linux talos boxes on 2009/02/26

Status: RESOLVED WORKSFORME

(Release Engineering :: General, defect)

Description

[Daniel Holbert [:dholbert]]

There was a Tp increase on the Linux mozilla-central unittest boxes today -- previously the value was pretty stable at around 236, and now it's up to 243, on 2 boxes: qm-plinux-trunk02 and qm-plinux-trunk03.

That 236-243 jump is pretty significant, because it happened simultaneously on two boxes, and none of the boxes have passed 240 since January (except for a brief period where a single box had an apparently-sporadic huge spike on feb 16-17)

Here's the graph:
http://graphs.mozilla.org/#show=395125,395135,395166&sel=1235677337,1235692316

Here's the before/after logs for each box:

* qm-plinux-trunk02:
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1235679413.1235684789.24770.gz
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1235686243.1235692628.7935.gz

* qm-plinux-trunk03:
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1235682778.1235690788.4439.gz
http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1235688648.1235695012.12324.gz


Tight regression range (from last good qm-plinux-trunk03 build to first bad qm-plinux-trunk02 build):
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=cdfbe284c207&tochange=4bdc1778459e

Comment 1

[Daniel Holbert [:dholbert]]

These metrics appear to have returned to their baseline values -- both qm-plinux-trunk02 and qm-plinux-trunk03 had two consecutive significantly-high points, and then returned back to their prior levels.

http://graphs.mozilla.org/#show=395125,395135,395166&sel=1235656193,1235714974

The pushes in the fix range are:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=bd91a3c914fd&tochange=1e0d54adf926
... but at first glance, none of those commits look particularly significant, and they don't look particularly related to the checkins in the regression range.

Moreover, the third box (qm-plinux-trunk01) had one cycle while the other two were spiking, and that box's tp value was in its normal range of values. (not a spike)

So, I'm not sure what happened... maybe cosmic rays hit our tinderboxen? :) We should keep an eye on these boxes, in any case.

Comment 2

[[e-mail] (:bhearsum)]

51692b0a0e56 is just test changes, so we can rule that one out. That leaves http://hg.mozilla.org/mozilla-central/rev/1e0d54adf926 (Mano) and http://hg.mozilla.org/mozilla-central/rev/686d81117403 (Taras).

Neither really seems suspicious to me, but I'm not expert here. I would expect to see a backout fix this type of problem...

Comment 3

[(dormant account)]

My change adds an API that isn't called(it's for fennec), so it wouldn't make any difference.

Comment 4

[Daniel Holbert [:dholbert]]

and it looks like Mano's change only affects behavior when the user is dragging a tab, which AFAICT cannot happen while the automated Tp tests are running.

So, it doesn't seem possible that any of the three changesets in the fix range could have actually been "fixes".

Comment 5

[[e-mail] (:bhearsum)]

Given that only 2 out of 3 boxes spiked, and quickly returned to normal, and the fact that the checkins in the fix range almost certainly can't fix a Tp regression we're closing this WFM.