1088903 - 6% Linux32|64 CART regresion on fx-team (v.36) Oct 23 from push 5c8fc27aa8d4

Status: RESOLVED DUPLICATE of bug 1088771

(Hello (Loop) :: Client, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

A slight regression came in for linux64 cart:
http://graphs.mozilla.org/graph.html#tests=%5B%5B309,132,35%5D,%5B309,132,33%5D%5D&sel=1413575096201,1414179896201&displayrange=7&datatype=running

I did some retriggers to help narrow this down:
https://tbpl.mozilla.org/?tree=Fx-Team&startdate=2014-10-23&enddate=2014-10-24&jobname=Ubuntu%20HW%2012.04%20x64%20fx-team%20talos%20svgr

You can see that we were posting in a 34-37 range before and after a 36-38 range as a result of:
https://hg.mozilla.org/integration/fx-team/rev/5c8fc27aa8d4

here is some more information about cart:
https://wiki.mozilla.org/Buildbot/Talos/Tests#TART.2FCART

Comment 1

[Matthew N. [:MattN]]

Joel, are you sure this isn't bug 1088771 you're seeing? When I look at http://graphs.mozilla.org/graph.html#tests=%5B%5B309,132,35%5D,%5B309,132,33%5D%5D&sel=1413972993922.805,1414103910995.9756,31.03057504557877,44.38489159234136&displayrange=7&datatype=running it seems like a890f2283b74 is what started the increase.

Nothing in this patch should be running during the customize transition. The only possible reason it could affect any performance was through adding two JSMs and therefore compartments.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

I did a few more retriggers on different revisions, lets see how this looks.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

this is hard to tell exactly where the needle moved.  As you can see on the graph this is a noisy test:
http://graphs.mozilla.org/graph.html#tests=%5B%5B309,132,35%5D%5D&sel=none&displayrange=7&datatype=running

The retriggers are hard to tell what is the culprit, but looking at datazilla:https://datazilla.mozilla.org/?start=1413355190&stop=1414315146&product=Firefox&repository=Fx-Team-Non-PGO&os=linux&os_version=Ubuntu%2012.04&test=cart&graph_search=5c8fc27aa8d4&tr_id=7492392&x86=false&error_bars=false&project=talos

I see rev 3df00a905527 as moving the needle on a couple of tests.  Now we have a few other bugs to deal with:
https://hg.mozilla.org/integration/fx-team/pushloghtml?changeset=3df00a905527

The specific subtests that regressed are:
2-customize-exit.half.TART
2-customize-exit.error.TART
1-customize-enter.all.TART
1-customize-enter.error.TART (but over a few days we didn't regress much)

Unfortunately looking at graph server, 3df00a905527 doesn't appear to be a problem, so depending on graph server to fix this or back out wouldn't be useful.  Even using the geometric mean on graph server doesn't help:
http://graphs.mattn.ca/graph.html?geo#tests=%5B%5B309,132,35%5D%5D&sel=1413863447868,1414165847868,0,25&displayrange=7&datatype=running

I am not sure how to proceed here, maybe Avi has an idea

Comment 4

[Avi Halachmi (:avih)]

I _think_ it's 5c8fc27aa8d4 (looking at graph server). But if we can't be sure which revision caused it.. then I don't know how much we can do...

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

so I am not sure what to do here either.  This is so noisy.  there is an obvious shift in the graph that shows a regression.  I guess if any of the patches in a small window look offensive we could look at those.  unfortunately, it is near impossible to test on try server by backing it out and seeing the result.

Comment 6

[:shell escalante]

In triage - this looks like a dupe of bug 1088771.  the Loop team isn't likely to have caused the regression.  If this belongs somewhere else or isn't that bug - please relocate.