1312952 - [Shield] Data Review for Unified URL Bar

Status: RESOLVED FIXED

(Shield :: Shield Study, defect)

Description

[Matt Grimes [:Matt_G]]

Please supply a sample payload that your study will submit via Unified Telemetry.  This must reside in the /docs/metrics.md of your repository or some similar location (core to the add-on code).

We also need answers to the following questions:
* If you have extended Telemetry or custom instrumentation, why are you taking these additional measurements?  (E.g. these measurements are outcome measures of the study)

* How long will the study run?

* Who is the owner of the data analysis for this study?

* Does your data analyst have an analysis plan?  (An outline of a strategy for analysis is sufficient)

* Will your study require third-party data collection?  E.g. Google analytics, loggly, any Apache server logs, etc.

Comment 1

[Javaun Moradi [:javaun]]

(In reply to Matt Grimes [:Matt_G] from comment #0)
> Please supply a sample payload that your study will submit via Unified
> Telemetry.  This must reside in the /docs/metrics.md of your repository or
> some similar location (core to the add-on code).

mak, can you supply a payload? 


> 
> We also need answers to the following questions:
> * If you have extended Telemetry or custom instrumentation, why are you
> taking these additional measurements?  (E.g. these measurements are outcome
> measures of the study)
> 
> * How long will the study run?

I assume this one is blocked on the experiment design review. 


> * Who is the owner of the data analysis for this study?

I need to find an analyst who can take this. Suggestions? Spenrose and Amit are working on search right now. Dzeber could at least advise. 

> 
> * Does your data analyst have an analysis plan?  (An outline of a strategy
> for analysis is sufficient)
> 
> * Will your study require third-party data collection?  E.g. Google
> analytics, loggly, any Apache server logs, etc.
No.

Comment 2

[Marco Bonardo [:mak]]

The fact is we don't have a specific "payload", cause we just use extendedTelemetry and I don't think anyone wants to really look into an extended telemetry payload (just open about:telemetry, raw view). I have a list of the probes we'll most likely use in the analysis that I will attach.

Indeed I'm not sure what to put in the metrics.md that I should add to the add-on itself.

Comment 3

[Marco Bonardo [:mak]]

Attachment: telemetry.txt

Comment 4

[Panos Astithas (he/him) [:past] (please ni?)]

(In reply to Matt Grimes [:Matt_G] from comment #0)
> Please supply a sample payload that your study will submit via Unified
> Telemetry.  This must reside in the /docs/metrics.md of your repository or
> some similar location (core to the add-on code).

To be clear, the document Marco attached is also present in the add-on repo:
https://github.com/mak77/unified-urlbar-shield-study/blob/master/telemetry.txt

> We also need answers to the following questions:
> * If you have extended Telemetry or custom instrumentation, why are you
> taking these additional measurements?  (E.g. these measurements are outcome
> measures of the study)

Yes, the additional measurements are outcome measures of the study.

Comment 5

[Javaun Moradi [:javaun]]

Hi Rebecca, I don't see your approval in the bug. What do we need to do?

Comment 6

[Javaun Moradi [:javaun]]

(In reply to Matt Grimes [:Matt_G] from comment #0)
> Please supply a sample payload that your study will submit via Unified
> Telemetry.  This must reside in the /docs/metrics.md of your repository or
> some similar location (core to the add-on code).
> 
> We also need answers to the following questions:
> * If you have extended Telemetry or custom instrumentation, why are you
> taking these additional measurements?  (E.g. these measurements are outcome
> measures of the study)
> 
> * How long will the study run?

Approximately 2 weeks. We may extend to 3 if data is not sufficient
> 
> * Who is the owner of the data analysis for this study?

DZeber is overseeing analysis.

> 
> * Does your data analyst have an analysis plan?  (An outline of a strategy
> for analysis is sufficient)

Yes. We're measuring user invocation of searches and Awesome Bar navigation (numeric counts only -- we will not capture content, nor URLs, nor search terms) for a combined search field vs. a separate Awesome Bar. Our goal is to understand how combining the bars impacts user navigation behavior.

> 
> * Will your study require third-party data collection?  E.g. Google
> analytics, loggly, any Apache server logs, etc.
No.

Comment 7

[[e-mail]]

Okay, I sign off!

Comment 8

[Marco Bonardo [:mak]]

I added a metrics.md to the repository, reporting the part we use of the telemetry payload. This contains the same identical information of the previously attached telemetry.txt, just in a payload shape.
https://raw.githubusercontent.com/mak77/unified-urlbar-shield-study/master/docs/metrics.md

Comment 9

[Panos Astithas (he/him) [:past] (please ni?)]

This study has shipped and the results were already collected and analyzed.