1029799 - [Search experiment] Define experiment parameters

Status: VERIFIED FIXED

(Firefox :: Search, defect)

Description

[:Felipe Gomes (needinfo for replies!)]

To publish a telemetry experiment, we need to define some configuration parameters. The most important are:

- start date
- end date
- max running time
- target locale: en-US
- target channel: beta
- version: probably 32 or 33, or both if the experiment window overlaps an uplift
- sampling size: a sample that will give us roughly 30k users per cell

Detailed explanation of all these fields, and other optional ones: https://ci.mozilla.org/job/mozilla-central-docs/Tree_Documentation/experiments/manifest.html

Comment 1

[:Felipe Gomes (needinfo for replies!)]

Note: if the target audience is meant to target geo US, and not simply the en-US locale, we will also need to define a jsfilter for this.

Comment 2

[Benjamin Smedberg]

AIUI we plan on continuing the experiment for 6 weeks, even thought we expect to get most data after a couple of weeks.

startTime: 1-July
maxStartTime: 15-July
endTime: 12-August
maxActiveSeconds: 42 days
locale: ["en-US"]
channel: ["beta"]
minVersion: "31.0"
minBuildID: TBD, either this Thursday's or next Monday's beta

Still working on the sample value: it will be something between 0.6 and 0.04.

Comment 3

[Kamil Jozwiak [:kjozwiak]]

Marking this as qa+, we need to make sure that the experiments parameters are working. For example, MinVersion, MinBuildID, make sure the experiment expires at the correct time etc..

Comment 4

[Benjamin Smedberg]

Attachment: manifest.json

I believe these are the final parameters, attached as a manifest.json (I'll attach a patch against the repo you sent me also).

In English:

Search Experiment

Start Date
    01-Jul-2014
Start Until
    20-Jul-2014
End Date
    30-Sep-2014
Maximum runtime
    28.0 days
Update Channels
    beta
Sample Rate
    50%

Test alternate default search settings for users of Firefox beta.

This experiment divides users into five groups: a control and four test groups with various default search parameters and notification settings.

Comment 5

[Benjamin Smedberg]

Attachment: manifest-fix

Same thing, as a patch

Comment 6

[:Felipe Gomes (needinfo for replies!)]

Should we set minBuildID to today's build which included the data.set fix? I verified that it made it on time for today's Aurora, but I haven't checked beta.

Comment 7

[Benjamin Smedberg]

Ah yes, I forgot. We won't know the buildid of today's beta until later today.

Comment 8

[:Felipe Gomes (needinfo for replies!)]

Comment on attachment 8448157 [details]
manifest.json

r+ with that included

Comment 9

[Benjamin Smedberg]

    "minBuildID": "20140630185627",

The parameters are being included in the server push from bug 1029810. I believe *this* bug is QA- because the experiment QA is being done in bug 1029818.

Comment 10

[Kamil Jozwiak [:kjozwiak]]

As mentioned in comment #9, this was tested in bug #1029818. I quickly went through the manifest and made sure that all the parameters matched what was described in this bug: (just double checking)
* https://telemetry-experiment.cdn.mozilla.net/manifest/v1/firefox/31.0/beta