1381816 - Pref study: Race Cache with Network iteration #2

Status: RESOLVED FIXED

(Shield :: Shield Study, enhancement)

Description

[Valentin Gosu [:valentin] (he/him)]

+++ This bug was initially created as a clone of Bug #1367951 +++

We would like to do another experiment for RCWN. The first experiment (bug 1367951), has completed successfully, but several bugs that we discovered during the run may have skewed the results.

This iterations should also run on Nightly, with a 50/50 split of the population.
The pref controlling the feature is: network.http.rcwn.enabled

Apart from tracking crash-rates and regressions, we are interested in the following telemetry probes:
 NETWORK_RACE_CACHE_WITH_NETWORK_*
   - these probes are only gathered when the feature is active.
 
 HTTP_PAGE_COMPLETE_LOAD_V2
 HTTP_PAGE_COMPLETE_LOAD_NET_V2
 HTTP_PAGE_COMPLETE_LOAD_CACHED_V2
 HTTP_SUB_COMPLETE_LOAD_V2
 HTTP_SUB_COMPLETE_LOAD_CACHED_V2
 HTTP_SUB_COMPLETE_LOAD_NET_V2
 TOTAL_CONTENT_PAGE_LOAD_TIME
   - these probes are useful when comparing the target population with the control, so we can evaluate improvement in page load speed, if any.

We could also use some help from the data wizards to evaluate the telemetry we get back.

Comment 1

[Michal Novotny [:michal]]

Please, wait for bug 1377223 before starting this study. We're still waiting for the data review in that bug.

Comment 2

[Matt Grimes [:Matt_G]]

Sounds like we are still holding off on launching this experiment. Is that the case?

Comment 3

[Valentin Gosu [:valentin] (he/him)]

I don't think there are any other blockers to launching this experiment. We can do it as soon as possible. We should run it for one week (until next Wednesday, before the merge?)

Comment 4

[Matt Grimes [:Matt_G]]

Just to confirm, you want 100% of Nightly split equally between branches. Is that correct? If that is the case please bump your intent to ship email on Release Drivers and let them know we're launching today due to some blockers and give them the new end date.

Comment 5

[Valentin Gosu [:valentin] (he/him)]

(In reply to Matt Grimes [:Matt_G] from comment #4)
> Just to confirm, you want 100% of Nightly split equally between branches. Is
> that correct? If that is the case please bump your intent to ship email on
> Release Drivers and let them know we're launching today due to some blockers
> and give them the new end date.

That's correct. I'll send the email in a couple of minutes.

Comment 6

[Matt Grimes [:Matt_G]]

Study has been pushed live. We will end the study on August 2nd as requested.

Comment 7

[Matt Grimes [:Matt_G]]

Hey Valentin. According to the Shield calendar your study will end today. Please confirm and I will make the changes on our end.

Comment 8

[Valentin Gosu [:valentin] (he/him)]

(In reply to Matt Grimes [:Matt_G] from comment #7)
> Hey Valentin. According to the Shield calendar your study will end today.
> Please confirm and I will make the changes on our end.

Hi Matt, indeed the study is supposed to end today. Thanks!

Comment 9

[Matt Grimes [:Matt_G]]

Ok. The study has been terminated. If you have interesting results from the experiment we'd be interested in seeing them.

Comment 10

[Valentin Gosu [:valentin] (he/him)]

Hi Josh, could you help us pull the data from this experiment similar to bug 1367951 comment 15?
Also, some instructions on how to analyze the data from both experiments would be very helpful. Thanks!

Comment 11

[Josh Gaunt [:jgaunt]]

Valentin - I would be glad to with the caveat that I'd be able to offer guidance with analysis but not perform analysis myself. However, we should have a modular analysis solution that produces plots by the end of the week. Would you rather I pull the pings for you and hand them over, or hold out until the modular solution is complete?

Comment 12

[Valentin Gosu [:valentin] (he/him)]

(In reply to Josh Gaunt [:jgaunt] from comment #11)
> Valentin - I would be glad to with the caveat that I'd be able to offer
> guidance with analysis but not perform analysis myself. However, we should
> have a modular analysis solution that produces plots by the end of the week.
> Would you rather I pull the pings for you and hand them over, or hold out
> until the modular solution is complete?

I think we can wait until the end of the week. Please let us know when it's ready. Thanks!

Comment 13

[Michal Novotny [:michal]]

(In reply to Josh Gaunt [:jgaunt] from comment #11)
> Valentin - I would be glad to with the caveat that I'd be able to offer
> guidance with analysis but not perform analysis myself. However, we should
> have a modular analysis solution that produces plots by the end of the week.
> Would you rather I pull the pings for you and hand them over, or hold out
> until the modular solution is complete?

Is the solution already complete? Where can we get the data?

Comment 14

[Michal Novotny [:michal]]

(In reply to Valentin Gosu [:valentin] from comment #0)
>  HTTP_PAGE_COMPLETE_LOAD_V2
>  HTTP_SUB_COMPLETE_LOAD_V2

Due to bug 1390683, we're also interested in:

HTTP_PAGE_COMPLETE_LOAD
HTTP_SUB_COMPLETE_LOAD

Comment 15

[Michal Novotny [:michal]]

Could anybody help us to obtain the data from this experiment? Data in csv format (just like what can be downloaded from telemetry.mozilla.org) for the mentioned probes would be ok.

Comment 16

[Josh Gaunt [:jgaunt]]

Michal,

I used the following pyspark gist to produce output on s3 at s3://telemetry-private-analysis-2/jgaunt/pref-flip-rcwn2-1381816.csv

https://gist.github.com/jtg567/9c048993a2710b5e6df73b2244de066e

if you have question feel free to email or ping me on irc/slack, same handle everywhere

Comment 17

[Josh Gaunt [:jgaunt]]

https://gist.github.com/jtg567/3a69308c67724484369fc3650012b4a3

Michal - there are interim results at the end of the notebook linked above. The current approach is to collapse all histograms across pings over clients, but this leads to more active clients being over-represented in the data. Please bear that caveat in mind when interpreting. The method of analysis is still evolving while we determine what's optimal for these data.