1060038 - CART regressions on nightly (v.34) need some attention

Status: RESOLVED WONTFIX

(Testing :: Talos, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

we have a variety of cart regressions:
http://graphs.mozilla.org/graph.html#tests=[[309,132,35],[309,132,33],[309,132,31],[309,132,25],[309,132,37]]&sel=1401481975835,1409257975835&displayrange=90&datatype=running

* On August 5th there is an improvement this is thanks to bug 1045864 (fixing a regression)
* On August 15th there is a regression, it appears to be related to bug 1014332 (which was backed out 6 days later). 
* The regression on the 15th seems partially affected by bug 862563 (mostly on winxp)
* The regression on the 21st (right after the backout) appears to be from bug 1007336.
* A regression on the 25th has yet to be defined.

I did a lot of retriggers around the 15/16th:
https://tbpl.mozilla.org/?tree=Fx-Team&startdate=2014-08-14&enddate=2014-08-17&jobname=fx-team%20talos%20svgr

The regression on the 25th could be a regression, it is hard to determine, but I have started some retriggers:
https://tbpl.mozilla.org/?tree=Fx-Team&startdate=2014-08-22&enddate=2014-08-27&jobname=fx-team%20talos%20svgr



Bug 1014332 affect all windows and linux results.

Comment 1

[:Gijs (he/him)]

(In reply to Joel Maher (:jmaher) from comment #0)
> we have a variety of cart regressions:
> http://graphs.mozilla.org/graph.html#tests=[[309,132,35],[309,132,33],[309,
> 132,31],[309,132,25],[309,132,37]]&sel=1401481975835,
> 1409257975835&displayrange=90&datatype=running

> * The regression on the 15th seems partially affected by bug 862563 (mostly
> on winxp)
> * The regression on the 21st (right after the backout) appears to be from
> bug 1007336.

The noise on the graph here is terrible. I can't see this regression on the 21st which you claim is there - in fact, it looks like things improved until the 23rd... Can you post compare-talos links for bug 1007336's impact, considering that you say you tracked it down, I'm guessing you have try pushes? 

> Bug 1014332 affect all windows and linux results.

This doesn't make any sense, because it was backed out. :-\

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

look at the graph closer (30 day view), you will see more of the regressions I am talking about.

discussing bug 1014332, look at a 30 day view on linux64, you can see that on the 16th we had a regression and an improvement on the 21st (although very briefly thanks to bug 1007336 landing).

There is more hunting to do here, but I wanted to get this on file as it is messy and it will continue to be so as this moves to Aurora next week.

Comment 3

[Shane Caraveo (:mixedpuppy)]

Bug 1014332 did cause a cart regression on linux, it's one thing I have been rewriting the patch to attempt to address.  Though that is a shot in the dark all the time since you cannot reproduce these problems locally.

Comment 4

[Avi Halachmi (:avih)]

(In reply to Shane Caraveo (:mixedpuppy) from comment #3)
>... .  Though that is a shot in the
> dark all the time since you cannot reproduce these problems locally.

Define "cannot"? Did you try running talos locally?

Also without running talos, you should get comparable results by installing the TART addon (link here: https://wiki.mozilla.org/Buildbot/Talos/Tests#TART.2FCART ), then clicking "Set ASAP" button, restarting the browser, deselecting all default tests and then select the last test "Customize" and run.

It would display the numbers on screen once the run is over.

Comment 5

[Shane Caraveo (:mixedpuppy)]

(In reply to Avi Halachmi (:avih) from comment #4)
> (In reply to Shane Caraveo (:mixedpuppy) from comment #3)
> >... .  Though that is a shot in the
> > dark all the time since you cannot reproduce these problems locally.
> 
> Define "cannot"? Did you try running talos locally?

Using the addon.

I don't get a regression locally, in fact it is reproducibly faster (<1%) with the patches in that bug than without, which is a big difference from the 6% regression I was getting via try.

Comment 6

[Avi Halachmi (:avih)]

6% is calculated over the average of all the sub-results, _after_ dropping the highest one (we hope to improve this formula at bug 1021842).

So if the highest value subtest has improved considerably with your patch, but other values have regressed slightly, it will still be reported as a regression.

jmaher and myself discussed in the past that talos will log an estimation of the graphserver's "final result" such that reported regressions could hopefully be reproduced locally, but neither of us recalls if we filed that bug.

Maybe I'll add such feature to the TART addon as well regardless of that talos mod.

Comment 7

[Shane Caraveo (:mixedpuppy)]

I got the 6% from compar-talos, but would have to find the old try builds to show that.  This report is the new patch I am working on:

http://compare-talos.mattn.ca/?oldRevs=88fc6b3f36a0&newRev=ed6b5b4dfa5f&server=graphs.mozilla.org&submit=true

It still shows 4.67% on linux, which is better, and not being flagged by that tool, but still seems high.  The strange thing is, the new patch has done away with [obvious] code paths that I believed to be run during customization entry.  I'll have to dig a bit further to ensure that.

Comment 8

[Avi Halachmi (:avih)]

(In reply to Shane Caraveo (:mixedpuppy) from comment #7)
> I got the 6% from compar-talos, but would have to find the old try builds to
> show that. 

No need (unless you want it yourself), we believe you :)

> This report is the new patch I am working on:
> 
> http://compare-talos.mattn.ca/
> ?oldRevs=88fc6b3f36a0&newRev=ed6b5b4dfa5f&server=graphs.mozilla.
> org&submit=true
> 
> It still shows 4.67% on linux, which is better, and not being flagged by
> that tool, but still seems high. ...

I guess "not being flagged" means that 4.67 doesn't have red background? If so, then it's for a good reason, compare-talos considers it within the noise level, or not conclusive enough when taking into account other results of this test.

Unless you know better (which is possible), I'd trust the un-flagged status, meaning that possibly there's no concrete regression there.

Comment 9

[:Gijs (he/him)]

It is possible that bug 1007336 caused a regression because the panel is not marked as hidden. We've had issues with tpaint before when having panels that weren't hidden by default in browser XUL. However, I'd like to know for sure before spending time architecturing around this...


baseline:
https://tbpl.mozilla.org/?tree=Try&rev=95e253f90502 (current fx-team tip)

backout:
https://tbpl.mozilla.org/?tree=Try&rev=1833f9c71576

Comment 10

[:Gijs (he/him)]

(In reply to :Gijs Kruitbosch from comment #9)
> It is possible that bug 1007336 caused a regression because the panel is not
> marked as hidden. We've had issues with tpaint before when having panels
> that weren't hidden by default in browser XUL. However, I'd like to know for
> sure before spending time architecturing around this...
> 
> 
> baseline:
> https://tbpl.mozilla.org/?tree=Try&rev=95e253f90502 (current fx-team tip)
> 
> backout:
> https://tbpl.mozilla.org/?tree=Try&rev=1833f9c71576

http://compare-talos.mattn.ca/?oldRevs=95e253f90502&newRev=1833f9c71576&server=graphs.mozilla.org&submit=true is fairly conclusive here - we did regress a bit with bug 1007336. Not sure how much of that we'll get back by adding hidden="true" to the panel, but it's worth a shot. I don't see anything else in the patch that'd cause any issues here.

Comment 11

[:Gijs (he/him)]

Attachment: hide themes panel initially to avoid cart hit,

Try push to see if this helps (will need some retriggers): https://tbpl.mozilla.org/?tree=Try&rev=4924316627f3 .

Comment 12

[:Gijs (he/him)]

Comment on attachment 8483072 [details] [diff] [review]
hide themes panel initially to avoid cart hit,

(In reply to :Gijs Kruitbosch from comment #11)
> Created attachment 8483072 [details] [diff] [review]
> hide themes panel initially to avoid cart hit,
> 
> Try push to see if this helps (will need some retriggers):
> https://tbpl.mozilla.org/?tree=Try&rev=4924316627f3 .

Comparing a backout to this ( http://compare-talos.mattn.ca/?oldRevs=1833f9c71576&newRev=4924316627f3&server=graphs.mozilla.org&submit=true ), this seems to not help - heck, it might be making things worse.

I got nothin'. Jared?

Comment 13

[:Gijs (he/him)]

(In reply to :Gijs Kruitbosch from comment #12)
> Comment on attachment 8483072 [details] [diff] [review]
> hide themes panel initially to avoid cart hit,
> 
> (In reply to :Gijs Kruitbosch from comment #11)
> > Created attachment 8483072 [details] [diff] [review]
> > hide themes panel initially to avoid cart hit,
> > 
> > Try push to see if this helps (will need some retriggers):
> > https://tbpl.mozilla.org/?tree=Try&rev=4924316627f3 .
> 
> Comparing a backout to this (
> http://compare-talos.mattn.ca/
> ?oldRevs=1833f9c71576&newRev=4924316627f3&server=graphs.mozilla.
> org&submit=true ), this seems to not help - heck, it might be making things
> worse.
> 
> I got nothin'. Jared?

Err, wait, that's wrong - if this is only slightly worse than a backout, but better than the original ( http://compare-talos.mattn.ca/?oldRevs=95e253f90502&newRev=4924316627f3&server=graphs.mozilla.org&submit=true ) then this might be OK? Need more Windows retriggers (done). Would still appreciate other ideas...

Comment 14

[Jared Wein [:jaws] (please needinfo? me)]

Comment on attachment 8483072 [details] [diff] [review]
hide themes panel initially to avoid cart hit,

Review of attachment 8483072 [details] [diff] [review]:
-----------------------------------------------------------------

rs=me

Comment 15

[:Gijs (he/him)]

(In reply to Jared Wein [:jaws] (please needinfo? me) from comment #14)
> Comment on attachment 8483072 [details] [diff] [review]
> hide themes panel initially to avoid cart hit,
> 
> Review of attachment 8483072 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> rs=me

Yeah, but this is a wash on Win7 and Win8, where a backout very much isn't. Need to figure out why we're doing worse because of bug 1007336. :-(

Comment 16

[Joel Maher ( :jmaher ) (UTC -8)]

this issue is on aurora after the uplift:
win7: 9.01% regression
win8: 12.5% regression

to be honest we have some big wins on inbound for win8 cart:
http://graphs.mozilla.org/graph.html#tests=[[309,52,31],[309,63,31],[309,131,31]]&sel=none&displayrange=30&datatype=running

Comment 17

[:Gijs (he/him)]

(In reply to Joel Maher (:jmaher) from comment #16)
> this issue is on aurora after the uplift:
> win7: 9.01% regression
> win8: 12.5% regression
> 
> to be honest we have some big wins on inbound for win8 cart:
> http://graphs.mozilla.org/graph.html#tests=[[309,52,31],[309,63,31],[309,131,
> 31]]&sel=none&displayrange=30&datatype=running

That's good, but the regressions here still puzzle me... it'd be good to understand why they are happening.

Comment 18

[:Gijs (he/him)]

This shipped, and realistically, I don't think we're going to go back and do more archaeology than we already did (without much result :-( ).