1514541 - [Experimentation Support] Document Running Funnelcake Experiment Process

Status: RESOLVED INACTIVE

(Data Science :: Documentation, task)

Description

[Su-Young Hong]

Brief description of the request:

* general idea is land something in DTMO with info on what funnelcake experiments are, what the process for doing them looks like (get the builds done, request distribution through marketing), what it is in real life (redirecting users who come to mozilla.org), distinction between stub and full installers, what targeting we can do, etc. 

* include what teams to reach  out to for each of these
* include when funnelcake is appropriate and when shield/normandy experiment is appropriate

Link to any assets:

* reference: pin in Win10 experiment

Is there a specific data scientist you would like or someone who has helped to triage this request:

* reported by self, assigned to self

Comment 1

[Su-Young Hong]

* include 64 vs 32 OS targeting (and build making) 

* notes on QA and monitoring the traffic cop configs, i.e. the targeting and sampling (should be done on the marketing / website side but something to be aware) 

* install pings from funnelcakes

* maybe related, but traffic cop targeting accuracy vs telemetry reporting info

Comment 2

[Su-Young Hong]

Update:

This is documentation. Not sure where it should live... (DTMO, team docs, team github, mana, ???) but this should describe explicitly with resources how to launch a Funnelcake experiment.

• Written or PMs but more geared towards Data Scientists
• Information it needs to include:
  • What a funnelcake is, and how it differs from other experiments
  • How funnelcakes are distributed, and implications (the websites)
  • Who to contact to get a funnelcake built
  • Who to contact to get the funnelcake distributed
  • How Traffic cop works and what this means
  • All the specs in a funnelcake (non-optional specs, i.e. OS bits are important to think through, can't be ignored)
  • Stubinstaller vs full installer
  • (optional): getting install data from funnelcake installers
  • (optional): getting download data from website
  • any other considerations (QA, what to check in QA like paveover behavior, notes on download to install to profile mapping)
  • how to identify your funnelcake in telemetry (you should look at the data starting at downloads, since the experience differs at that point)

strategy: Write in gdoc first, and then find a "home" for it.

Comment 3

[Su-Young Hong]

unassigning from self due to upcoming PTO / leave.

If another team member wants to pick it up for Doc Days, feel free, otherwise, I'll re-take it once I'm back / have bandwidth.

Comment 5

[Su-Young Hong]

combining this bug with bug 1514549

this documentation should also include:

---

It should describe:

What a funnelcake is
How they are identified
Use cases:
    Experiments (with a link to "launching funnelcake experiments" documentation - this documentation should be forthcoming)
    Partner distributions

It should have a details section which describes

what happens / is supposed to happen to the distribution_id field on paverover installs (see QA documentation)
how partner distributions get distributed
etc.

Comment 6

[Jess Mokrzecki [:jmok]]

Work for the DS team is now tracked in Jira. You can search with the Data Science Jira project for the corresponding ticket.