1069450 - Regression accross all gaia apps on Sep 15 2014

Status: RESOLVED WORKSFORME

(Firefox OS Graveyard :: Performance, defect)

Description

[Zibi Braniecki [:zbraniecki][:gandalf]]

There's a regression in range of 100ms on Sept 15th 2014.

Read more here: https://groups.google.com/d/msg/mozilla.dev.gaia/vstguNLgDek/bDPqJVZrkucJ

Comment 1

[Zibi Braniecki [:zbraniecki][:gandalf]]

Eli, is there anything else I can do to help debug this? The regression is still present.

Comment 2

[:Eli Perelman]

Hey gandalf, sorry I haven't had a chance to work towards this one as I've been swamped with other priorities. Anything we can do to narrow down a regression window would help to get this triaged.

Comment 3

[Aus Lacroix [:aus]]

[Blocking Requested - why for this release]: 100 ms cold start perf regression is quite noticeable to the user!

:gandalf, do you happen to know the changeset for that one as well? is it in the system app bits? If so, I can take a look at that one too this week.

Comment 4

[Gregor Wagner [:gwagner]]

Perf regression

Comment 5

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

Gandalf, I see 2.1 performance is back up as of yesterday, presumably from bug 1069452 landing.

Can we close this out?

On another note:

I don't think we could have efficiently gotten a tighter regression range anyway, and we should temper expectations there. 

100ms differential over several days on a noisy branch confounds the type of clear yes/no conclusions you need to do to bisect. If it doesn't look like a clear step function on the graph, it won't work well.

Commit inspection and profiling to look for obvious issues should be first/second-line responses. Bisection/ranging past what Datazilla already shows via the push tests should be a distant third. It takes a long time and it's not a high-confidence procedure.

FWIW, if it's something obvious enough to be reasonably bisected, looking at the gaia/gecko revision changing between the discontinuous data points in Datazilla is pretty informative, and further bisection might not be necessary. If that can't be done, bisection probably can't be either.

Comment 6

[Zibi Braniecki [:zbraniecki][:gandalf]]

(In reply to Geo Mealer [:geo] from comment #5)
> Gandalf, I see 2.1 performance is back up as of yesterday, presumably from
> bug 1069452 landing.
> 
> Can we close this out?

I'll try to bisect it today and will close if I won't be able to find it.

> I don't think we could have efficiently gotten a tighter regression range
> anyway, and we should temper expectations there. 
> 
> 100ms differential over several days on a noisy branch confounds the type of
> clear yes/no conclusions you need to do to bisect. If it doesn't look like a
> clear step function on the graph, it won't work well.

If you zoom in on 12th-17th September in datazilla you will very clearly see two jumps. One on 15th, one on 16th.

Both were clear 100ms jumps that look way beyond noise, and if you remove extremes I believe you would get a clear jumps. (it would be nice for datazilla to have an option to average daily results, to reduce noise and see the jump patterns)

Comment 7

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

(In reply to Zibi Braniecki [:gandalf] from comment #6)
> (In reply to Geo Mealer [:geo] from comment #5)
> > Gandalf, I see 2.1 performance is back up as of yesterday, presumably from
> > bug 1069452 landing.
> > 
> > Can we close this out?
> 
> I'll try to bisect it today and will close if I won't be able to find it.
> 
> > I don't think we could have efficiently gotten a tighter regression range
> > anyway, and we should temper expectations there. 
> > 
> > 100ms differential over several days on a noisy branch confounds the type of
> > clear yes/no conclusions you need to do to bisect. If it doesn't look like a
> > clear step function on the graph, it won't work well.
> 
> If you zoom in on 12th-17th September in datazilla you will very clearly see
> two jumps. One on 15th, one on 16th.
> 
> Both were clear 100ms jumps that look way beyond noise, and if you remove
> extremes I believe you would get a clear jumps. (it would be nice for
> datazilla to have an option to average daily results, to reduce noise and
> see the jump patterns)

I'll have more to say about it when I send out the acceptance comparisons, but averaging is the wrong way to go for these, even within a single data point. Load time turns out to be a positive-skewed normal distribution due to the hard lower-bound, so average, std deviation, CIs, etc, all invalid. Median is better, perhaps geometric mean.

The better option, IMO, is a once-daily scheduled run with more replicates. I'm going to try this on the side day over day to see if it's clearer, but I'm guessing it will be.

Re: the results, which branch are you looking at? I'm not seeing anything that's as clear as what you're seeing, which tells me I might be looking in the wrong place.

I strongly recommend screenshotting any Datazilla results a bug is based on. It's pretty hard to duplicate a view later in it, especially as more results come in.

Comment 8

[Zibi Braniecki [:zbraniecki][:gandalf]]

(In reply to Geo Mealer [:geo] from comment #7)
> 
> I'll have more to say about it when I send out the acceptance comparisons,
> but averaging is the wrong way to go for these, even within a single data
> point. Load time turns out to be a positive-skewed normal distribution due
> to the hard lower-bound, so average, std deviation, CIs, etc, all invalid.
> Median is better, perhaps geometric mean.


Good point. I was thinking about repetitions vs replications.
 
> The better option, IMO, is a once-daily scheduled run with more replicates.
> I'm going to try this on the side day over day to see if it's clearer, but
> I'm guessing it will be.

That would be even better. With an option to only show those results that would make spotting substantial regressions trivial.
 
> Re: the results, which branch are you looking at? I'm not seeing anything
> that's as clear as what you're seeing, which tells me I might be looking in
> the wrong place.

master
 
> I strongly recommend screenshotting any Datazilla results a bug is based on.
> It's pretty hard to duplicate a view later in it, especially as more results
> come in.

Good idea. I will use it next time when I see it :)

Comment 9

[Joshua Mitchell (Inactive)]

removing regression-window request based on comment 5

Comment 10

[Gregor Wagner [:gwagner]]

(In reply to Zibi Braniecki [:gandalf] from comment #6)
> (In reply to Geo Mealer [:geo] from comment #5)
> > Gandalf, I see 2.1 performance is back up as of yesterday, presumably from
> > bug 1069452 landing.
> > 
> > Can we close this out?
> 
> I'll try to bisect it today and will close if I won't be able to find it.
> 

Any luck here?

Comment 11

[Kevin Grandon :kgrandon]

Nothing actionable about this bug in the current state so going to close it out. Based on 2.1 performance testing, we will be addressing performance concerns in bug 1079585.