1128635 - BHR add-on analysis

Status: RESOLVED FIXED

(Toolkit :: Telemetry, defect)

Description

[Roberto Agostino Vitillo (:rvitillo)]

Attachment: BHR add-ons correlation analysis.ipynb

The analysis was run on a single day for now. Vladan, could you please have a quick look at it?

Comment 1

[Vladan Djeric (:vladan)]

Comment on attachment 8558050 [details]
BHR add-ons correlation analysis.ipynb

- I read over the analysis code, but I'll have to do a second pass later

Notes from first pass:

- Good idea to use Aurora.. Telemetry is in bad shape on Nightly right now. We'll have to use Nightly for e10s comparison eventually though.
- Didn't the line "return m.group(1)" in get_name() throw an exception when an addon didn't match your regex?
- Your js_prog regex would miss *.jsm addon source files
- You'd also miss out on hang stacks consisting entirely of C++ labels and "(chrome script)" frames, although i'm not 100% if those could be extension code, and obviously it would be hard to associate them with an extension. Might be interesting to know how common these are though.
- Did the "RAM Restart" addon (jid0-uqa46zynuvsnaydawtvivgmogls) cause 35k hangs in a single session?
- I'm also interested in seeing the "hangs/hour" rate for a profile with no extensions, and also how much an addon contributes to that value on average. Of course we'd have to filter the outliers. We should also break it down by Windows/Mac and HDD/SSD. I think this approach would allow us to do comparisons of e10s vs non-e10s, e.g. addon Foo causes X hangs per hour on non-e10s and an additional +50% hangs/hour with e10s. What do you think?


Some ideas for follow up investigations:

- How often is threadhangstats missing (different from bug 1106058 and bug 1106040)
- What do the machines that report no hangs in sessions lasting at least an hour have in common? SSDs? i7?
- for the problem addons we find, we'll want to figure out if it got worse or better at some version

Comment 2

[Roberto Agostino Vitillo (:rvitillo)]

(In reply to Vladan Djeric (:vladan) from comment #1)
> Comment on attachment 8558050 [details]
> BHR add-ons correlation analysis.ipynb
> 
> - I read over the analysis code, but I'll have to do a second pass later
> 
> Notes from first pass:
> 
> - Good idea to use Aurora.. Telemetry is in bad shape on Nightly right now.
> We'll have to use Nightly for e10s comparison eventually though.
> - Didn't the line "return m.group(1)" in get_name() throw an exception when
> an addon didn't match your regex?

It always matches, at least in the data I have seen.

> - Your js_prog regex would miss *.jsm addon source files

Good catch.

> - You'd also miss out on hang stacks consisting entirely of C++ labels and
> "(chrome script)" frames, although i'm not 100% if those could be extension
> code, and obviously it would be hard to associate them with an extension.
> Might be interesting to know how common these are though.

In my preliminary analysis, I couldn't spot any C++ stack that I could associate easily with an add-on. I wonder how common those are, do we have some stats on the the kind of code our add-ons use?

> - Did the "RAM Restart" addon (jid0-uqa46zynuvsnaydawtvivgmogls) cause 35k
> hangs in a single session?

Yes. Outliers can be particulary bad, I am going to use robust statistics in the final report.

> - I'm also interested in seeing the "hangs/hour" rate for a profile with no
> extensions, and also how much an addon contributes to that value on average.
> Of course we'd have to filter the outliers. We should also break it down by
> Windows/Mac and HDD/SSD. I think this approach would allow us to do
> comparisons of e10s vs non-e10s, e.g. addon Foo causes X hangs per hour on
> non-e10s and an additional +50% hangs/hour with e10s. What do you think?

Breaking down by HDD and SSD might not be that helpful. Machines with SSDs are usually newer than those with HDDs so we might end up believing that having a SSD is causing the improvement while it may not be the only factor.

If we want to test the hypothesis that hangs are reduced by having a SSD we could use propensity matching to account for the covariates or simply make sure to compare groups with similar HW/SW configurations.

Comment 3

[Roberto Agostino Vitillo (:rvitillo)]

Forgot to mention, the biggest obstacle here is to match the prefix of a frame, which usually contains a hint of the add-on name, to the ID of the add-on in the info field. Any thought on how we could automate the process is welcome.

Comment 4

[Vladan Djeric (:vladan)]

(In reply to [e-mail] from comment #2)
> > - You'd also miss out on hang stacks consisting entirely of C++ labels and
> > "(chrome script)" frames, although i'm not 100% if those could be extension
> > code, and obviously it would be hard to associate them with an extension.
> > Might be interesting to know how common these are though.
> 
> In my preliminary analysis, I couldn't spot any C++ stack that I could
> associate easily with an add-on. I wonder how common those are, do we have
> some stats on the the kind of code our add-ons use?

I didn't mean addons with C++ code, I was wondering how common are BHR stacks which consist only of labels from Firefox C++ code and "(chrome script)" labels

> Breaking down by HDD and SSD might not be that helpful. Machines with SSDs
> are usually newer than those with HDDs so we might end up believing that
> having a SSD is causing the improvement while it may not be the only factor.

I agree, but I think the presence of an SSD is a decent proxy for "a more powerful machine". My concern is that if we don't control for hardware characteristics, we'll end up comparing hang rates of two populations with very different hardware.

> If we want to test the hypothesis that hangs are reduced by having a SSD we
> could use propensity matching to account for the covariates or simply make
> sure to compare groups with similar HW/SW configurations.

Agreed, and I definitely would like to see the contribution of SSDs to hang-rates although that's not top priority right now.

Comment 5

[Roberto Agostino Vitillo (:rvitillo)]

Attachment: BHR add-ons correlation analysis.ipynb

Comment 6

[Vladan Djeric (:vladan)]

A few follow-ups from the last review:
- I filed bug 1132558 to report OS install date as a rough proxy for machine age
- Can you look into the origin of the "(chrome script)" frames
- What are your thoughts on comparing e10s vs non-e10s hanginess?

Comment 7

[Vladan Djeric (:vladan)]

- I'm not familar with these data analysis libraries, so I'm a bit confused.. it looks like you're calculating an addon's contribution to browser jankiness, i.e. ratio %, by dividing (total number of hangs attributable to the addon) by (total number of browser hangs + this addon's hangs). Is that right?
- For the top stacks analysis: Try to generate better "signatures" from the BHR stacks.. e.g. trim all frames before Timer::Fire, coalesce adjacent "(content script)" frames into a single frame (same for (chrome script) frames), etc. Look at what the chromehangs signature generation script does for other ideas

Comment 8

[Roberto Agostino Vitillo (:rvitillo)]

(In reply to Vladan Djeric (:vladan) -- please needinfo! from comment #6)
> - What are your thoughts on comparing e10s vs non-e10s hanginess?

We should open another bug for that discussion. If you are referring to comparing e10s vs non-e10s wrt add-on hanginess, I would start by comparing the ratios and go from there. 

Probably in that case it would make sense to have a common denominator, like session time, so that we could compare the e.g. the median number of hangs per hour for add-on A with e10 vs the median number of hangs per hour for add-on A without e10s.

Comment 9

[Roberto Agostino Vitillo (:rvitillo)]

(In reply to Vladan Djeric (:vladan) -- please needinfo! from comment #7)
> - I'm not familar with these data analysis libraries, so I'm a bit
> confused.. it looks like you're calculating an addon's contribution to
> browser jankiness, i.e. ratio %, by dividing (total number of hangs
> attributable to the addon) by (total number of browser hangs + this addon's
> hangs). Is that right?

That's correct.