1359900 - [Shield] Data Review for Unified URL Bar v2

Status: RESOLVED FIXED

(Shield :: Shield Study, enhancement)

Description

[Shilpi Gupta [:shilpi]]

v1 study data bug here: Bug 1312952

+++ This bug was initially created as a clone of Bug #1312952 +++

Please supply a sample payload that your study will submit via Unified Telemetry.  This must reside in the /docs/metrics.md of your repository or some similar location (core to the add-on code).

We also need answers to the following questions:
* If you have extended Telemetry or custom instrumentation, why are you taking these additional measurements?  (E.g. these measurements are outcome measures of the study)

* How long will the study run?

* Who is the owner of the data analysis for this study?

* Does your data analyst have an analysis plan?  (An outline of a strategy for analysis is sufficient)

* Will your study require third-party data collection?  E.g. Google analytics, loggly, any Apache server logs, etc.

Comment 1

[Dave Zeber [:dzeber]]

(In reply to [e-mail] from comment #0)
> Please supply a sample payload that your study will submit via Unified
> Telemetry.  This must reside in the /docs/metrics.md of your repository or
> some similar location (core to the add-on code).

This study does not add any new Telemetry probes beyond what was used in the v1 study (bug 1312952). The metrics.md (https://github.com/mak77/unified-urlbar-shield-study/blob/master/docs/metrics.md) for that study still applies.

Some new fields have been added to the `shield-study` payloads to describe the working of the study in granularity than in the v1 study. These include a number of diagnostic measurements that check whether expected experiment conditions have been applied.

The other major difference is that we are reporting subsets of extended Telemetry in the `shield-study` payloads themselves. This is to provide easier access to this data (prior to the Experiment data pipeline being fully operational).

There is a doc (https://gist.github.com/dzeber/f2ee1e03d72b9cd8fad09d1e2056dc75) detailing data collection and design for this study, which includes a sample payload and description of the fields.

> * If you have extended Telemetry or custom instrumentation, why are you
> taking these additional measurements?  (E.g. these measurements are outcome
> measures of the study)

Yes, these are outcome measures of the study.

> * How long will the study run?

As stated in bug 1359118, comment 2, the study will run for a maximum of 45 days.

> * Who is the owner of the data analysis for this study?

Dzeber.

> * Does your data analyst have an analysis plan?  (An outline of a strategy
> for analysis is sufficient)

Yes, it will roughly follow the same plan as the analysis for the v1 study (https://metrics.mozilla.com/protected/dzeber/reports/unified-search-analysis.html).

> * Will your study require third-party data collection?  E.g. Google
> analytics, loggly, any Apache server logs, etc.

No.

Comment 2

[Dave Zeber [:dzeber]]

Rebecca, is anything else needed to sign off?

Comment 3

[Shilpi Gupta [:shilpi]]

Rebecca, Dave - Can you sign off on the data review? We are lined up to launch the shield tomorrow morning.

Thanks
Shilpi

Comment 4

[[e-mail]]

r+

Comment 5

[Panos Astithas (he/him) [:past] (please ni?)]

Review done, the study is live as of a few weeks ago. Resolving.