[Shield] Data Review for Unified URL Bar v2

RESOLVED FIXED

Status

Shield
Shield Study
RESOLVED FIXED
6 months ago
5 months ago

People

(Reporter: shgupta@mozilla.com, Assigned: dzeber)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

6 months ago
v1 study data bug here: Bug 1312952

+++ This bug was initially created as a clone of Bug #1312952 +++

Please supply a sample payload that your study will submit via Unified Telemetry.  This must reside in the /docs/metrics.md of your repository or some similar location (core to the add-on code).

We also need answers to the following questions:
* If you have extended Telemetry or custom instrumentation, why are you taking these additional measurements?  (E.g. these measurements are outcome measures of the study)

* How long will the study run?

* Who is the owner of the data analysis for this study?

* Does your data analyst have an analysis plan?  (An outline of a strategy for analysis is sufficient)

* Will your study require third-party data collection?  E.g. Google analytics, loggly, any Apache server logs, etc.
(Reporter)

Updated

6 months ago
Blocks: 1360440
(Assignee)

Comment 1

6 months ago
(In reply to shgupta@mozilla.com from comment #0)
> Please supply a sample payload that your study will submit via Unified
> Telemetry.  This must reside in the /docs/metrics.md of your repository or
> some similar location (core to the add-on code).

This study does not add any new Telemetry probes beyond what was used in the v1 study (bug 1312952). The metrics.md (https://github.com/mak77/unified-urlbar-shield-study/blob/master/docs/metrics.md) for that study still applies.

Some new fields have been added to the `shield-study` payloads to describe the working of the study in granularity than in the v1 study. These include a number of diagnostic measurements that check whether expected experiment conditions have been applied.

The other major difference is that we are reporting subsets of extended Telemetry in the `shield-study` payloads themselves. This is to provide easier access to this data (prior to the Experiment data pipeline being fully operational).

There is a doc (https://gist.github.com/dzeber/f2ee1e03d72b9cd8fad09d1e2056dc75) detailing data collection and design for this study, which includes a sample payload and description of the fields.

> * If you have extended Telemetry or custom instrumentation, why are you
> taking these additional measurements?  (E.g. these measurements are outcome
> measures of the study)

Yes, these are outcome measures of the study.

> * How long will the study run?

As stated in bug 1359118, comment 2, the study will run for a maximum of 45 days.

> * Who is the owner of the data analysis for this study?

Dzeber.

> * Does your data analyst have an analysis plan?  (An outline of a strategy
> for analysis is sufficient)

Yes, it will roughly follow the same plan as the analysis for the v1 study (https://metrics.mozilla.com/protected/dzeber/reports/unified-search-analysis.html).

> * Will your study require third-party data collection?  E.g. Google
> analytics, loggly, any Apache server logs, etc.

No.
(Assignee)

Comment 2

6 months ago
Rebecca, is anything else needed to sign off?
Flags: needinfo?(rweiss)
(Reporter)

Comment 3

6 months ago
Rebecca, Dave - Can you sign off on the data review? We are lined up to launch the shield tomorrow morning.

Thanks
Shilpi

Comment 4

6 months ago
r+
Flags: needinfo?(rweiss)
Review done, the study is live as of a few weeks ago. Resolving.
Status: NEW → RESOLVED
Last Resolved: 5 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.