1299290 - Do we need to change anything in addon/e10s experiment telemetry to account for RTL or A11y

Status: RESOLVED FIXED

(Firefox :: Extension Compatibility, defect, P2)

Description

[:shell escalante]

with RTL in Beta 50 and Accessibility in Nightly 51 - is there anything we need to add to our metric analysis to be able to see if either of those is causing issues.

particularly for beta 50 - where RTL is added.  will there be a way to isolate those?

Comment 1

[Dave Zeber [:dzeber]]

Shell, do you want to just exclude profiles with RTL/A11y from consideration, or do you want to actually flag them and compare between groups?

IIRC profiles with RTL/A11y are currently tagged as "disqualified" when assigning e10sCohort (cf bug 1255013), and so are currently excluded from the e10s/addons analysis of test vs. control. Felipe, can you confirm this is the case?

Comment 2

[:shell escalante]

hi dave - i added you to Friday's e10s meeting.  everyone is there and likely this is something we can talk about with chutten, bds, and felipe at once to resolve how we should be looking at it.

also want to ask about release data for Fx49.

Comment 3

[Dave Zeber [:dzeber]]

Update after speaking with Felipe:

- Currently (49 or lower), having either RTL or A11y leads to an e10s cohort of "disqualified", so these profiles are excluded from the experiment
- Starting in 50 (RTL) and 51 (A11y), having these will no longer cause disqualification, so these profiles can enter any cohort.

Comment 4

[:shell escalante]

from that meeting it sounds like there is a way to isolate if needed.  If you are comfortable with what that is Dave, and being able to do it quickly if the data looks bad and we want to remove those populations (to limit variable - and see if they are the cause or if concerning results are still present) - feel free to close this bugs as fixed.

meeting notes: "Telemetry prefs can show if RTL or A11y…. Crash stats annotated with information as well"

Comment 5

[Dave Zeber [:dzeber]]

Before we close this I'd like to nail down how we can tell whether we have either A11y or RTL for a given profile. Chris has confirmed the following:

- For A11y, we can check the Telemetry flag A11Y_INSTANTIATED_FLAG (extended Telemetry only, which is fine for pre-release).

- For RTL, we can look up the profile's locale (environment.settings.locale) and check whether it is an RTL locale. This would require knowing which locales are RTL. Kev, do you know if there is such a list somewhere?

These fields are available in the Telemetry pings (accessible to the notebook-based E10s experiment analysis we've been doing), but not in the crash_aggregates derived dataset that powers the re:dash e10s stability dashboard.

Comment 6

[Dave Zeber [:dzeber]]

The RTL locales are  "ar", "he", "fa", "ug", "ur".

Comment 7

[Dave Zeber [:dzeber]]

Resolving this bug, since this answers the question of how to identify A11y/RTL profiles. This doesn't require any changes to data collection itself.

However, the crash_aggregates dataset that powers the Beta 49 Stability dashboard does not have these fields, so looking at crash counts will require a custom job.