Closed Bug 1198383 Opened 5 years ago Closed 4 years ago

[e10s] figure out what the crash rates are for a11y users on Nightly

Categories

(Firefox :: Disability Access, defect)

37 Branch
Unspecified
Linux
defect
Not set
normal

Tracking

()

RESOLVED WONTFIX
Tracking Status
e10s + ---

People

(Reporter: jimm, Unassigned)

References

Details

(Whiteboard: aes+)

To assess the stability of a11y + e10s, we need a metric similar to our standard crashstats # of crashes / 100 ADI. Except I'd like to limit this to the pool of users who have a11y turned on.

https://crash-stats.mozilla.com/search/?product=Firefox&version=43.0a1&accessibility=__true__&_facets=signature&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#facet-signature
Robert - do you have any suggestions here? I can do queries but I'm not sure how we go about normalizing using ADI value.
Flags: needinfo?(kairo)
(In reply to Jim Mathies [:jimm] from comment #0)
> To assess the stability of a11y + e10s, we need a metric similar to our
> standard crashstats # of crashes / 100 ADI. Except I'd like to limit this to
> the pool of users who have a11y turned on.

As we do not have ADI numbers for those people, we cannot provide such stats.

> Robert - do you have any suggestions here? I can do queries but I'm not sure
> how we go about normalizing using ADI value.

See above. Also, Nightly is so noisy due to people checking in patches all the time that cause crashes that it's really hard to do good assessment of any data there.


That said, e10s is on by default on Nightly, so the amount of people who have it off is probably very small.
Flags: needinfo?(kairo)
Does the A11Y_INSTANTIATED_FLAG (telemetry) help here?
(In reply to David Bolter [:davidb] from comment #2)
> Does the A11Y_INSTANTIATED_FLAG (telemetry) help here?

I don't think we can correlate the two since they are different data sets collected via different methods.
(In reply to Jim Mathies [:jimm] from comment #3)
> (In reply to David Bolter [:davidb] from comment #2)
> > Does the A11Y_INSTANTIATED_FLAG (telemetry) help here?
> 
> I don't think we can correlate the two since they are different data sets
> collected via different methods.

Well, by comparing the percent of the telemetry population using a11y to the percent of crashes where a11y is running you could get some kind of guess how much more a11y crashes.  It wouldn't be perfectly accurate because the telemetry number isn't a perfect sample of the population, but my statistics fu isn't strong enough to be able to quantify that.
Assignee: nobody → birunthan
Depends on: 1224334
Assignee: birunthan → nobody
Blocks: e10sa11y2
Hi how can you test for this wen if you have a11y on e10s is turned off at start-up. If i force e10s on then Nvda can not read anything in a firefox window.
Flags: needinfo?(jmathies)
Whiteboard: aes-linux, aes-win
Flags: needinfo?(jmathies)
OS: Unspecified → Linux
Whiteboard: aes-linux, aes-win → aes+
Blocks: 1277670
Hey chris, can we calculate a crash rate comparison linux users who use accessibility?

https://telemetry.mozilla.org/new-pipeline/dist.html#!compare=e10sEnabled&cumulative=0&end_date=2016-05-27&keys=__none__!__none__!__none__&max_channel_version=beta%252F47&measure=A11Y_INSTANTIATED_FLAG&min_channel_version=null&os=Linux&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2016-04-25&table=0&trim=1&use_submission_date=0

The a11y team wants to figure out if they can safely roll out with e10s to release, and to do that, we need some way of knowing how stable the combination is.
Flags: needinfo?(chutten)
So, to clarify, you would like to measure something in a population:

1) On beta
2) Linux
3) A11y enabled

And then we want to compare the subpopulations based on E10s enabled?

Well, the size of the population of just 1 and 2 (Linux users on Beta) is 83,067 [1]

As you've linked [2] there is at most a 3.4% a11y enabled rate in that population, which translates to 2,833. 

This is beta, so 50% of that population is sacrosanct and we can't ask it for anything. But assuming we can use it as the control population, we're comparing the crashes of one population of about 1400 users to another population of about 1400 users.

I'm not sure at what rate a particular crash that only shows on beta+linux+a11y+e10s would have to occur in order to be visible in that sort of sample size and not just be counted as noise.

[1]: https://sql.telemetry.mozilla.org/queries/465
[2]: http://mzl.la/1RRSWg1
Flags: needinfo?(chutten)
David, sounds like we don't have enough users to do a good comparison with.
Flags: needinfo?(dbolter)
Wow so few Linux users on beta. I guess we'll watch crash-stats as always and revisit this if needed.
Status: NEW → RESOLVED
Closed: 4 years ago
Flags: needinfo?(dbolter)
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.