1224518 - [e10s][telemetry] Cross match whitelisted e10s add-ons with most popular add-ons reported by telemetry

Status: RESOLVED INVALID

(Toolkit :: Telemetry, defect)

Description

[Roberto Agostino Vitillo (:rvitillo)]

See Comment 4 and 5 of Bug 1223780.

Comment 1

[Biru [:poiru]]

Submitted PR: https://github.com/vitillo/e10s_analyses/pull/6

Comment 2

[Biru [:poiru]]

Results: http://nbviewer.ipython.org/github/poiru/e10s_analyses/blob/top-addons/aurora/addons/top_extensions.ipynb

jimm, does this look reasonable?

Comment 3

[Jim Mathies [:jimm]]

(In reply to Birunthan Mohanathas [:poiru] from comment #2)
> Results:
> http://nbviewer.ipython.org/github/poiru/e10s_analyses/blob/top-addons/
> aurora/addons/top_extensions.ipynb
> 
> jimm, does this look reasonable?

Currently discussing this in a background email thread.

Is there some way to get a number for the percentage of users who have one or more non-whitelisted addon installed?

Comment 4

[Biru [:poiru]]

(In reply to Jim Mathies [:jimm] from comment #3)
> Is there some way to get a number for the percentage of users who have one
> or more non-whitelisted addon installed?

I'll submit a PR to include that in the analysis, but here is the information for now:

- 54.53% had at least one addon
- 8.43% had only whitelisted addons
- 46.10% had at least one unwhitelisted addon

Comment 5

[Biru [:poiru]]

Here is the updated analysis: http://nbviewer.ipython.org/github/vitillo/e10s_analyses/blob/master/aurora/addons/top_extensions.ipynb

Comment 6

[Jim Mathies [:jimm]]

(In reply to Jim Mathies [:jimm] from comment #3)
> (In reply to Birunthan Mohanathas [:poiru] from comment #2)
> > Results:
> > http://nbviewer.ipython.org/github/poiru/e10s_analyses/blob/top-addons/
> > aurora/addons/top_extensions.ipynb
> > 
> > jimm, does this look reasonable?
> 
> Currently discussing this in a background email thread.
> 
> Is there some way to get a number for the percentage of users who have one
> or more non-whitelisted addon installed?

So we do want things broken down:

1) users with no addons
2) users with whitelisted addons
3) everyone else

Sorry, we can't use the most popular for now. Also it's anticipated that the whitelist will change over time, so we probably want to get the infra in place to filter based on an arbitrary set.

Comment 7

[Biru [:poiru]]

(In reply to Jim Mathies [:jimm] from comment #6)
> So we do want things broken down:
> 
> 1) users with no addons
> 2) users with whitelisted addons
> 3) everyone else

I'm not sure what you want broken down. Could you clarify?

Comment 8

[Jim Mathies [:jimm]]

(In reply to Birunthan Mohanathas [:poiru] from comment #7)
> (In reply to Jim Mathies [:jimm] from comment #6)
> > So we do want things broken down:
> > 
> > 1) users with no addons
> > 2) users with whitelisted addons
> > 3) everyone else
> 
> I'm not sure what you want broken down. Could you clarify?

When the perf team generates regression numbers for e10s, we should break the numbers down as such:

1) regression numbers for user who do not have addons installed
2) regression numbers for user who have one or more whitelisted addons installed.
3) users who do not fit in buckets #1 and #2.

We will be rolling out initially to users in the first group. Then we will be rolling out to user in the second and third groups. It is expected that we will fix regressions for group #2 before rolling e10s out to them. It is expected that we will identify misbehaving addons in group #3 and disable them.

So when you generate stats, we're interested in seeing the reports broken down according to the three groups of users listed above.

Comment 9

[Jim Mathies [:jimm]]

This bug was about seeing how closely the most popular list matched the white list. I think it's pretty clear that it doesn't so we'll need to be able to filter on a specific list.

Comment 10

[Biru [:poiru]]

Do you have any thoughts on comment 9? In particular, how what would be the correct way to allow filtering on an arbitrary list? Should this be e.g. a dashboard of some sort?

Comment 11

[Roberto Agostino Vitillo (:rvitillo)]

(In reply to Birunthan Mohanathas [:poiru] from comment #10)
> Do you have any thoughts on comment 9? In particular, how what would be the
> correct way to allow filtering on an arbitrary list? Should this be e.g. a
> dashboard of some sort?

We can't filter on an arbitrary list in a dashboard as that would require way too many batch jobs to pre-compute the datasets for all possible combinations. The best we can do is parametrizing that notebook on an input file that contains a set of whitelisted add-ons. That would allow us to re-run the notebook without changing the code.

Jim, is that what you had in mind?

Comment 12

[Jim Mathies [:jimm]]

(In reply to Roberto Agostino Vitillo (:rvitillo) from comment #11)
> (In reply to Birunthan Mohanathas [:poiru] from comment #10)
> > Do you have any thoughts on comment 9? In particular, how what would be the
> > correct way to allow filtering on an arbitrary list? Should this be e.g. a
> > dashboard of some sort?
> 
> We can't filter on an arbitrary list in a dashboard as that would require
> way too many batch jobs to pre-compute the datasets for all possible
> combinations. The best we can do is parametrizing that notebook on an input
> file that contains a set of whitelisted add-ons. That would allow us to
> re-run the notebook without changing the code.
> 
> Jim, is that what you had in mind?

Yes this is fine. The e10s team won't change the list often (if ever) from what it is currently.

Comment 13

[:Gijs (he/him)]

Is this still relevant, or should we resolve wfm / wontfix?