1321373 - Determine cause for increase in number of Nightly users seeing spinners between Sept 16th and Sept 17th

Status: NEW

(Firefox :: General, defect)

Description

[Mike Conley (:mconley) (:⚙️)]

I've written some visualizations for our Telemetry data that let's us get a sense of how much our Nightly population is plagued by tab switch spinners, and the severity of those spinners.

I've added those visualizations to the bottom of http://mikeconley.github.io/bug1310250 ("Affected client" and "Severity").

There was an increase in the number of Nightly clients affected by tab switch spinners between September 16th and September 17th. That increase mostly falls into the "0ms - 999ms" bucket, which went from 24% of Nightly clients being in that bucket, to about 28% of Nightly clients being in that bucket. Because there was an increase in the number of affected clients, this implies that a new set of clients were being affected by spinners.

Here's the changeset range for the builds in question:

https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=edfff7a9e4c43f6d516dfbf69a64e205e5cdb699&tochange=b401cb17167b34c362eb819259effbb3c0979f59

Comment 1

[Mike Conley (:mconley) (:⚙️)]

Attachment: Screenshot of graph

Comment 2

[Ben Kelly [:bkelly, not reviewing]]

Mike, doesn't your blog post github page show that after an initial spike the total number of affected clients reduced?  And the increase in 0-999ms spinner remained elevated while the longer duration spinners remained reduced.  Doesn't this suggest we largely just reduced the duration of the spinners so more show up in the 0-999ms bucket instead of the longer duration buckets?

Comment 3

[Mike Conley (:mconley) (:⚙️)]

A lot of things appear to happen in that short window of time (Sept 16 - 18). Here's my read on it. Let me know if I've missed something:

September 17th:
* A tab spinner regression causes affected users to go from ~50% to ~55%. There's a corresponding increase in the 0 - 999ms bucket from ~24% to ~27%. Smaller increases are seen in some of the other buckets, while the others remain steady. I conclude from this that more users were being affected by spinners in this build than the day previous, as opposed to users being shuffled into different buckets.

September 18th:
* A drop occurs in almost all buckets, and the number of affected users goes from 55% to 46%. That's a difference of 9%.
* There is a drop in the 0 - 999ms bucket from 28% - 27% (1% drop)
* There is a drop in the 1000ms - 2296ms bucket from 10% - 7% (3% drop)
* There is a drop in the 2297ms - 5276ms bucket from 8% - 5% (3% drop)
* There is a drop in the 5277ms - 12123ms bucket from 5% - 4% (1% drop)
* There is a drop in the 12124ms - 27855ms bucket from 3% - 2% (1% drop)
* The rest of the buckets don't change significantly.
* The drops account for the total 9% difference in users affected.

What this suggests to me is that we made things worse on the 17th build, but on the 18th, a lot of the really long spinners went away, which put more users into the 0 - 999ms bucket. And yeah, the drop in affected users seems to support this.

Which is to say, bkelly, that I think you're right. :)

Still would be interesting to know what caused things to go sour, and then what caused things to improve. FWIW, here's the commits that landed in the 18th build:

https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=b401cb17167b34c362eb819259effbb3c0979f59&tochange=eaf5eb6f8fa0d8e7a09f3774c0da53c0dd6dadd7

Comment 4

[Mike Conley (:mconley) (:⚙️)]

Ah, well, bug 1279086 landed on the 18th, so I think that accounts for the big win. I'll add a marker to the graph.

Comment 5

[Mike Conley (:mconley) (:⚙️)]

(But I'd still like to understand what happened in the Sept 17th build.)

Comment 6

[Ben Kelly [:bkelly, not reviewing]]

The number of tab switches spiked by 50% (1M) on Sep 17. Maybe some test or something was run on that day and getting included in the results.  Bill working on his patches?

Comment 7

[Ben Kelly [:bkelly, not reviewing]]

(In reply to Ben Kelly [:bkelly] from comment #6)
> The number of tab switches spiked by 50% (1M) on Sep 17. Maybe some test or
> something was run on that day and getting included in the results.  Bill
> working on his patches?

Oh, that was a different day.  The timescales confused me.  Sorry.

Comment 8

[Ben Kelly [:bkelly, not reviewing]]

I have to say, though, its nice to see hard data backing up the feeling of improved responsiveness from bug 1279086.