1224334 - Add accessibility status to Telemetry Environment

Status: RESOLVED WONTFIX

(Toolkit :: Telemetry, defect)

Description

[Vladan Djeric (:vladan)]

We want to determine a11y's impact on e10s crash rates. This requires accessibility information in crash pings (in the Environment block).

You can just query the accessibilityService, e.g. http://hg.mozilla.org/mozilla-central/annotate/3cc3b1968524/toolkit/modules/Troubleshoot.jsm#l449

Comment 1

[Vladan Djeric (:vladan)]

Possible dupes maybe?
bug 648121
bug 754416

Comment 2

[David Bolter [:davidb] (NeedInfo me for attention)]

This is the flag we use for regular telemetry for accessibility instantiated: A11Y_INSTANTIATED_FLAG

https://dxr.mozilla.org/mozilla-central/search?q=A11Y_INSTANTIATED_FLAG&redirect=true&case=true

Comment 3

[Vladan Djeric (:vladan)]

(In reply to David Bolter [:davidb] from comment #2)
> This is the flag we use for regular telemetry for accessibility
> instantiated: A11Y_INSTANTIATED_FLAG

Yeah I'm familiar with this histogram, it's one of the first histograms in the alphabetically-sorted histogram list on telemetry.mozilla.org :) But we need to access this information from situations where recording in *histograms* is disabled. So Telemetry can't just read back from this histogram.

Comment 4

[Benjamin Smedberg]

I'm concerned about putting this in the environment block. Whether accessibility is active is a dynamic property, not the state of the system. Would you for instance split subsessions if somebody ran for a while without a11y and then it got turned on for some reason?

Obviously we have problems with these "flip-once" flag histograms and subsession splits: can we move flip-once flags into a separate section?

Separately, what does "recording in histograms is disabled" mean? I'm perfectly fine with maaking A11Y_INSTANTIATED an opt-out metric, since it's a key correlating factor for other metrics.

Comment 5

[Trevor Saunders (:tbsaunde)]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #4)
> I'm concerned about putting this in the environment block. Whether
> accessibility is active is a dynamic property, not the state of the system.

its not really clear to me why we need to add anything.  We already annotate if a11y was on in crash reports, so I'd think you could just look at the number of reports that have that annotation vs not, and if they have e10s on.

Comment 6

[Vladan Djeric (:vladan)]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #4)
> I'm concerned about putting this in the environment block. Whether
> accessibility is active is a dynamic property, not the state of the system.
> Would you for instance split subsessions if somebody ran for a while without
> a11y and then it got turned on for some reason?

The existing A11Y_INSTANTIATED_FLAG reports whether accessibility service was *ever* started during a session.
I'd like an environment-block flag for whether the session was *started* with accessibility enabled. 

Trevor: does Firefox not decide at startup whether to enable accessibility features?

> Obviously we have problems with these "flip-once" flag histograms and
> subsession splits: can we move flip-once flags into a separate section?

Sure, that's an option

> Separately, what does "recording in histograms is disabled" mean? 

I was trying to explain at a high level the situation where a user has not opted into extended Telemetry (but base Telemetry is not opted out either) and some code submits a crash ping as an external ping for sending. The crash ping doesn't have histograms so the accessibility information would have to be in the environment block. Additionally, the TelemetryEnvironment.jsm code wouldn't be able to get the a11y status from the A11Y_INSTANTIATED_FLAG histogram because extended Telemetry is disabled.

> I'm perfectly fine with making A11Y_INSTANTIATED an opt-out metric, since it's
> a key correlating factor for other metrics.

Ok. Crash pings still wouldn't have this histogram, but it might make it easier to get the a11y status in TelemetryEnvironment.jsm

(In reply to Trevor Saunders (:tbsaunde) from comment #5)
> its not really clear to me why we need to add anything.  We already annotate
> if a11y was on in crash reports, so I'd think you could just look at the
> number of reports that have that annotation vs not, and if they have e10s on.

We'd like to correlate users' crashes with their Telemetry submissions. It's a lot easier to do these analyses in one system

Comment 7

[Vladan Djeric (:vladan)]

Hmm, on second thought, we could accomplish the same thing by look at aborted-session pings and the value of their A11Y_INSTANTIATED_FLAG histogram. For the time-being our experiments are confined to Aurora & Beta, so nothing needs to be done on the client after all.

Comment 8

[Trevor Saunders (:tbsaunde)]

(In reply to Vladan Djeric (:vladan) -- please needinfo! from comment #6)
> (In reply to Benjamin Smedberg  [:bsmedberg] from comment #4)
> > I'm concerned about putting this in the environment block. Whether
> > accessibility is active is a dynamic property, not the state of the system.
> > Would you for instance split subsessions if somebody ran for a while without
> > a11y and then it got turned on for some reason?
> 
> The existing A11Y_INSTANTIATED_FLAG reports whether accessibility service
> was *ever* started during a session.
> I'd like an environment-block flag for whether the session was *started*
> with accessibility enabled. 
> 
> Trevor: does Firefox not decide at startup whether to enable accessibility
> features?

not really, on windows an mac iirc the OS sends a message saying please enable accessibility.  In the vast majority of cases I imagine that happens shortly after startup.  So other than setting an arbitrary cut off I don't think there is a distinction between started at startup, and started later.

> > its not really clear to me why we need to add anything.  We already annotate
> > if a11y was on in crash reports, so I'd think you could just look at the
> > number of reports that have that annotation vs not, and if they have e10s on.
> 
> We'd like to correlate users' crashes with their Telemetry submissions. It's
> a lot easier to do these analyses in one system

ok fine, but I don't see why you are involving telemetry at all. Or is your goal something other than the first sentence of the bug description?

Comment 9

[Chris H-C :chutten]

From discussion, the goal is to enable further analyses of telemetry data once we get a conclusion on the initial question. For that, it makes sense to do everything in telemetry.

So I guess we're good.