Closed Bug 1282136 Opened 8 years ago Closed 8 years ago

how to measure add-ons/e10s experiment in Release

Categories

(Firefox :: Extension Compatibility, defect)

defect
Not set
normal

Tracking

()

RESOLVED FIXED
Tracking Status
firefox50 --- affected

People

(Reporter: shell, Assigned: dzeber)

References

Details

(Whiteboard: telemetry, triaged)

BDS mentioned that crash stats were not in the telemetry spreadsheet - where are they?  can we get that info for e10s-addons experiments?
Whiteboard: telemetry → telemetry, triaged
needing info to Kev who is leading data analysis interactions with various teams
Flags: needinfo?(kev)
Flags: needinfo?(kev)
Summary: how to measure e10s-addons crash stats release criteria measurements → how to measure e10s-addons crash stats in Release
Depends on: 1281283
bringing up at e10s meeting sept 2nd to get details for fx 49 release
Assignee: nobody → dzeber
Flags: needinfo?(sescalante)
Summary: how to measure e10s-addons crash stats in Release → how to measure add-ons/e10s experiment in Release
for release we need david to provide add-on/e10s data similar to e10s:

crash stats: https://sql.telemetry.mozilla.org/queries/613/source
eng-ratio: https://sql.telemetry.mozilla.org/dashboard/client-count-firefox-e10s-release-
ADI board: (ask chutten - not sure if there is a specific one for our cohorts) https://crash-stats.mozilla.com/crashes-per-day/?p=Firefox&v=48.0.2&v=48.0.1&v=48.0&v=47.0.1&hang_type=any&os=Windows&os=Mac+OS+X&os=Linux&date_start=2016-08-18&date_end=2016-09-02&submit=Generate

i didn't hear that there were any other sources of visibility into release

ddurst sent an eng ratio query by ID - should we need to look at one in specific:
SELECT count(client_id)
FROM longitudinal
WHERE
    cardinality(array_intersect(map_keys(active_addons[1]), ARRAY[
    '{d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d}',
    '{b9db16a4-6edc-47ec-a1f4-b86292ed211d}'
    ])) = cardinality(map_keys(active_addons[1]))
Flags: needinfo?(sescalante)
The most actionable source of data we have going into release will probably be crash stats. Extended telemetry is rarely submitted, and the engagement ratio has a long burn-in period due to its use of MAU.

That said, the best way to make use of the crash data is still to split by e10s cohort. Felipe, can you confirm whether we will still see the e10s-with-addons group as a separate cohort in release 49 (and what the tag will be, if it changes)? That way, if we see increased crashiness, we can see whether it depends on the presence of add-ons.
Flags: needinfo?(felipc)
Yep, the cohort name will continue to be present on crash annotations under the "E10SCohort" annotation.

The tag name on release will be "addons-set49a-test" and "addons-set49a-control". Actually this will also be the name on the beta RCs, changed in bug 1297753. Since we added one extra addon from "set2a" (PersonasPlus), it needed to change name.
Flags: needinfo?(felipc)
We shouldn't use crash-stats for rate comparisons; it should definitely be the crashes/1000 hours metrics we use for everything. All of that data is in base telemetry.
(In reply to Benjamin Smedberg [:bsmedberg] from comment #6)
> We shouldn't use crash-stats for rate comparisons; it should definitely be
> the crashes/1000 hours metrics we use for everything. All of that data is in
> base telemetry.

Yes, sorry - this is what I meant. Basically the same as the stability metrics we've been looking at for Beta 49 (https://sql.telemetry.mozilla.org/queries/939/source).
A dashboard for tracking stability metrics in release 49 is at https://sql.telemetry.mozilla.org/dashboard/stability-metrics-for-e10s-add-ons-experiment-release-49-

It is very similar to the dashboard we used for Beta, showing daily crash rates by cohort. Since E10s switched on for all qualified profiles, we are only showing stats for:
- E10s on and has whitelisted add-ons enabled
- E10s on and has no add-ons
As discussed over email, we expect the dashboard in Comment 8 to satisfy top-level needs, and if necessary we can do post-hoc analysis to group profiles in the add-ons cohort according to which specific add-ons they have (eg. Test Pilot).

Closing this, as I think all our questions about metrics for e10s/add-ons in release 49 have been answered.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.