1296282 - Run an experiment on Beta to generate crash rate data for e10s+XP users

Status: RESOLVED WORKSFORME

(Firefox :: General, defect, P1)

Description

[Jim Mathies [:jimm]]

We'd like to run an experiment for XP users who do not have addons in beta 50 to find out what the crash rates are under e10s. We specifically want to be able to compare crash rates of non-e10s to e10s users.

It would be good to know first how many XP users we have on Beta. chutten, any change you might have those numbers?

Comment 1

[Jim Mathies [:jimm]]

felipe, any chance you could help get this set up (or give me a little guidance on doing so)?

Comment 2

[Chris H-C :chutten]

Proportionally-speaking (ie, using Longitudinal[1]):

Windows XP is about 15% of Beta

Using a sophisticated (not really) query [2] to figure out how many WinXP Beta users have addons that aren't just hello, pocket, or otherwise irrelevant gets us:

61.5% of Windows XP users on Beta have exclusionary addons.

So, 38.5% of Windows XP users don't. So about 5.7% of Beta users are WinXP users without problematic addons.

Now, this is over no particular span in time, so these are rough figures.

For comparison: the Beta49 addons-e10s experiment is presently running on a population of about 5% of beta (Halved to 2.5% for experimenting. i.e. Roughly 1.25% per branch)

From a client-count perspective (ie, using client_count[3]):
Beta Windows XP was at about 425k clients last Wednesday.

We have no data on addons in client_count (and likely never will, unless it's just a count), but taking the prior proportion (38.5% of Windows XP has 0 addons) we're looking at a full population of about 164k. Halved (for beta): 82k.

Hopefully this answers your questions?

[1]: https://sql.telemetry.mozilla.org/queries/1020/source
[2]: https://sql.telemetry.mozilla.org/queries/1021/source
[3]: https://sql.telemetry.mozilla.org/queries/1019/source#1766

Comment 3

[:Felipe Gomes (needinfo for replies!)]

So, while XP users are disqualified from e10s on the release channel, they were left qualified on Beta for exactly this reason. So this is done and the data already exists :)  You just need to filter by WinXP users only and compare the test and controls cohorts.

Note:
 - cohorts "test" and "control" only contain users with no add-ons
 - Please don't forget that bug 1237769 still applies, so only non-HW-accelerated XP users are running e10s

Comment 4

[Jim Mathies [:jimm]]

Thanks! So we have data on beta for users with unaccelerated gfx. I'll file a bug on reverting a3f8a4e83cdd.