1277545 - linux64 tabpaint regression on may 25/26

Status: RESOLVED WONTFIX

(Testing :: Talos, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

not using the official bug filing template, I wanted to get this on file prior to the uplift.

There are 2 changes in the linux64 tabpaint numbers:
https://treeherder.allizom.org/perf.html#/graphs?timerange=2592000&series=%5Bmozilla-inbound,7f20e2504c722e15e70fde0c06c926f946d6022c,1,1%5D&highlightedRevisions=ec3ce347ebb2&highlightedRevisions=3dd0686489c6&zoom=1463911850043.214,1464462335726.0835,72.5348253893672,88.86793300258427

On May 25th, bug 1274992 landed, and the test went noisy.  In fact, on other platforms we saw a regression, this matched that regression (and we expected it), but for linux64 we just ended up with a very large noise range.

On May 26th, Bug 1264300 landed, and the test became less noisy (yay!) but on the higher end.  

I wanted to file a bug to have this documented for the change we see- I do believe this is just the change to focus introduced in bug 1274992, but possibly I am overlooking something.

Comment 1

[Avi Halachmi (:avih)]

Bug 1274992 reverted bug 1266181 because you said it reported worse numbers (talos only patch - not firefox patch hence not firefox regression regardless), and added code to talos powers which is not yet used in any test (the video test will use it).

So unless this reverts an "improvement" which bug 1266181 accidentally displayed (again neither of those bugs could have affected any performance since they only touch talos), then the regression we're seeing now is due to new code which is unused - i.e. just because the talos powers addon became bigger by few lines.

I don't believe there's anything actionable here, and if the numbers are indeed reported as slightly worse now due to some unused LOC in talos powers, then we should take it IMO.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

oh, I agree we should take it, my biggest concern is with the noise then the less noise the next day- it seemed quite random.

either way, any tabpaint regression from the talos-powers change is a test change for the better.

Comment 3

[Avi Halachmi (:avih)]

(In reply to Joel Maher (:jmaher) from comment #2)
> either way, any tabpaint regression from the talos-powers change is a test
> change for the better.

Not sure why you claim so. This adds code to talos powers which only the video test would use. I don't see how it should make the tabpaint test neither better nor worse.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

ok, so then tabpaint regressed oddly enough- possibly we should figure that out.

Comment 5

[Avi Halachmi (:avih)]

Other than the unused code the commit adds nothing. I'm guessing the tabpaint test is sensitive like that, so we should take it as the new baseline. Nothing to do here really.

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

my attempt at bisecting this on try server:
https://treeherder.mozilla.org/#/jobs?repo=try&author=[e-mail]&fromchange=b821a71bddbca2f6aee61d6740b6313ce1d7c6f1&tochange=906cfd7790827a825507851fffbf4e83fac74ac5

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

Looking at the 4 pushes:
https://treeherder.mozilla.org/perf.html#/graphs?timerange=86400&series=%5Btry,7f20e2504c722e15e70fde0c06c926f946d6022c,1,1%5D&zoom=1464890476642.915,1464890785164.3726,73.72818775176738,88.23828296880419&selected=%5Btry,7f20e2504c722e15e70fde0c06c926f946d6022c,109024,21897263,1%5D

you can see 1 that has a noticeably higher range, and this is the revision that accounts for the increase (Bug 1274992):
https://hg.mozilla.org/integration/mozilla-inbound/rev/5567b666d2c3

So if this is a backout of a patch (which had a small improvement originally), then there is nothing of concern with the regression except the noise level/magnitude.  I do suspect the noise level is mostly attributed to infrastructure (bug 1271948).  All random theory though.

Comment 8

[Avi Halachmi (:avih)]

(In reply to Joel Maher (:jmaher) from comment #7)
> you can see 1 that has a noticeably higher range, and this is the revision
> that accounts for the increase (Bug 1274992):
> https://hg.mozilla.org/integration/mozilla-inbound/rev/5567b666d2c3

Yup, this one is 100% backout code wise, but also includes repackaging pageloader-signed.xpi with a new version - though considering I also packaged the previous pageloader-signed.xpi which it replaced, I'm pretty sure I used the exact same procedure and the diff does not come from the new xpi packaging.

> So if this is a backout of a patch (which had a small improvement
> originally), then there is nothing of concern with the regression except the
> noise level/magnitude.  I do suspect the noise level is mostly attributed to
> infrastructure (bug 1271948).  All random theory though.

Interesting about the general infrastructure issue, but otherwise, yeah, we should totally accept it.

Thanks for the effort in pinpointing the exact revision.

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

no actions to take on this bug.