Pref study: Race Cache with Network iteration #2

RESOLVED FIXED

Status

Shield
Shield Study
RESOLVED FIXED
7 months ago
6 months ago

People

(Reporter: valentin, Unassigned)

Tracking

(Blocks: 1 bug)

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [necko-active])

(Reporter)

Description

7 months ago
+++ This bug was initially created as a clone of Bug #1367951 +++

We would like to do another experiment for RCWN. The first experiment (bug 1367951), has completed successfully, but several bugs that we discovered during the run may have skewed the results.

This iterations should also run on Nightly, with a 50/50 split of the population.
The pref controlling the feature is: network.http.rcwn.enabled

Apart from tracking crash-rates and regressions, we are interested in the following telemetry probes:
 NETWORK_RACE_CACHE_WITH_NETWORK_*
   - these probes are only gathered when the feature is active.
 
 HTTP_PAGE_COMPLETE_LOAD_V2
 HTTP_PAGE_COMPLETE_LOAD_NET_V2
 HTTP_PAGE_COMPLETE_LOAD_CACHED_V2
 HTTP_SUB_COMPLETE_LOAD_V2
 HTTP_SUB_COMPLETE_LOAD_CACHED_V2
 HTTP_SUB_COMPLETE_LOAD_NET_V2
 TOTAL_CONTENT_PAGE_LOAD_TIME
   - these probes are useful when comparing the target population with the control, so we can evaluate improvement in page load speed, if any.

We could also use some help from the data wizards to evaluate the telemetry we get back.
Depends on: 1382852
Depends on: 1377223
Depends on: 1382845
Please, wait for bug 1377223 before starting this study. We're still waiting for the data review in that bug.

Comment 2

7 months ago
Sounds like we are still holding off on launching this experiment. Is that the case?
(Reporter)

Comment 3

7 months ago
I don't think there are any other blockers to launching this experiment. We can do it as soon as possible. We should run it for one week (until next Wednesday, before the merge?)

Comment 4

7 months ago
Just to confirm, you want 100% of Nightly split equally between branches. Is that correct? If that is the case please bump your intent to ship email on Release Drivers and let them know we're launching today due to some blockers and give them the new end date.
(Reporter)

Comment 5

7 months ago
(In reply to Matt Grimes [:Matt_G] from comment #4)
> Just to confirm, you want 100% of Nightly split equally between branches. Is
> that correct? If that is the case please bump your intent to ship email on
> Release Drivers and let them know we're launching today due to some blockers
> and give them the new end date.

That's correct. I'll send the email in a couple of minutes.

Comment 6

7 months ago
Study has been pushed live. We will end the study on August 2nd as requested.

Comment 7

7 months ago
Hey Valentin. According to the Shield calendar your study will end today. Please confirm and I will make the changes on our end.
(Reporter)

Comment 8

7 months ago
(In reply to Matt Grimes [:Matt_G] from comment #7)
> Hey Valentin. According to the Shield calendar your study will end today.
> Please confirm and I will make the changes on our end.

Hi Matt, indeed the study is supposed to end today. Thanks!

Comment 9

7 months ago
Ok. The study has been terminated. If you have interesting results from the experiment we'd be interested in seeing them.

Updated

7 months ago
Status: NEW → RESOLVED
Last Resolved: 7 months ago
Resolution: --- → FIXED
(Reporter)

Comment 10

7 months ago
Hi Josh, could you help us pull the data from this experiment similar to bug 1367951 comment 15?
Also, some instructions on how to analyze the data from both experiments would be very helpful. Thanks!
Flags: needinfo?(jgaunt)

Comment 11

7 months ago
Valentin - I would be glad to with the caveat that I'd be able to offer guidance with analysis but not perform analysis myself. However, we should have a modular analysis solution that produces plots by the end of the week. Would you rather I pull the pings for you and hand them over, or hold out until the modular solution is complete?
Flags: needinfo?(jgaunt)
(Reporter)

Comment 12

7 months ago
(In reply to Josh Gaunt [:jgaunt] from comment #11)
> Valentin - I would be glad to with the caveat that I'd be able to offer
> guidance with analysis but not perform analysis myself. However, we should
> have a modular analysis solution that produces plots by the end of the week.
> Would you rather I pull the pings for you and hand them over, or hold out
> until the modular solution is complete?

I think we can wait until the end of the week. Please let us know when it's ready. Thanks!
(In reply to Josh Gaunt [:jgaunt] from comment #11)
> Valentin - I would be glad to with the caveat that I'd be able to offer
> guidance with analysis but not perform analysis myself. However, we should
> have a modular analysis solution that produces plots by the end of the week.
> Would you rather I pull the pings for you and hand them over, or hold out
> until the modular solution is complete?

Is the solution already complete? Where can we get the data?
Flags: needinfo?(jgaunt)
(In reply to Valentin Gosu [:valentin] from comment #0)
>  HTTP_PAGE_COMPLETE_LOAD_V2
>  HTTP_SUB_COMPLETE_LOAD_V2

Due to bug 1390683, we're also interested in:

HTTP_PAGE_COMPLETE_LOAD
HTTP_SUB_COMPLETE_LOAD
Could anybody help us to obtain the data from this experiment? Data in csv format (just like what can be downloaded from telemetry.mozilla.org) for the mentioned probes would be ok.
Flags: needinfo?(mgrimes)
Flags: needinfo?(isegall)

Comment 16

6 months ago
Michal,

I used the following pyspark gist to produce output on s3 at s3://telemetry-private-analysis-2/jgaunt/pref-flip-rcwn2-1381816.csv

https://gist.github.com/jtg567/9c048993a2710b5e6df73b2244de066e

if you have question feel free to email or ping me on irc/slack, same handle everywhere
Flags: needinfo?(jgaunt)
Flags: needinfo?(mgrimes)
Flags: needinfo?(isegall)

Comment 17

6 months ago
https://gist.github.com/jtg567/3a69308c67724484369fc3650012b4a3

Michal - there are interim results at the end of the notebook linked above. The current approach is to collapse all histograms across pings over clients, but this leads to more active clients being over-represented in the data. Please bear that caveat in mind when interpreting. The method of analysis is still evolving while we determine what's optimal for these data.
You need to log in before you can comment on or make changes to this bug.