build search experiments

RESOLVED WONTFIX

Status

()

RESOLVED WONTFIX
4 years ago
3 years ago

People

(Reporter: Gavin, Assigned: Felipe)

Tracking

Trunk
Points:
13
Bug Flags:
firefox-backlog +
qe-verify -

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(4 attachments, 1 obsolete attachment)

We're going to do another round of search experiments, details TBD.
Flags: qe-verify-
Flags: firefox-backlog+
(Assignee)

Updated

4 years ago
Assignee: nobody → felipc
Status: NEW → ASSIGNED

Updated

4 years ago
Iteration: --- → 36.3
Group: mozilla-employee-confidential
(Assignee)

Comment 1

4 years ago
Created attachment 8521175 [details]
bootstrap.js

This is the bootstrap.js file based on the previous search experiment code.

The removal of the notification requirements made it possible to simplify a lot of the previous code which had to handle a manual experiment termination (3 days before expiry date) in order to display the "finished" notification.

I also added a bunch of logging using TelemetryLog

It doesn't include the UI code from Florian's bug 1088660
Attachment #8521175 - Flags: review?(benjamin)
(Assignee)

Updated

4 years ago
Attachment #8521175 - Attachment mime type: application/x-javascript → text/plain
(Assignee)

Comment 2

4 years ago
Created attachment 8521178 [details] [diff] [review]
Patch

Same thing, but in patch form, if you prefer to review it this way. It includes the whole project folder. You probably don't need to review the Makefile.
The install.rdf is not finished yet: we need the strings for the experiment name, description, etc. Who can provide it?
Attachment #8521178 - Flags: review?(benjamin)
(Assignee)

Comment 3

4 years ago
Created attachment 8521205 [details] [diff] [review]
Full project, with UI code

If anyone wants to build and test

Updated

4 years ago
Attachment #8521175 - Flags: review?(benjamin) → review+

Comment 4

4 years ago
Comment on attachment 8521178 [details] [diff] [review]
Patch

Kev should provide the name/description. Kev, those will be displayed to users in the experiments pane of the addon manager.

Please remove em:homepageURL.
Attachment #8521178 - Flags: review?(benjamin)
Attachment #8521178 - Flags: review+
Attachment #8521178 - Flags: feedback?(kev)

Comment 5

4 years ago
Created attachment 8522463 [details]
previous-experiment-filter.js

Felipe this is the filter function that I think needs to end up as filter.js in the experiment. With this function, the experiment should be set to a 100% sample rate.
Attachment #8522463 - Flags: review?(felipc)
(Assignee)

Comment 6

4 years ago
>function filter(c) {
>  let days = c.healthReportPayload.data.days;
>  for (let day of days) {
>    let experiments = day["org.mozilla.experiments.info"];
>    if (experiments.lastActive == "fx-searchtest-en-beta31@mozilla.org") {
>      throw new Error("fx-searchtest-en-beta31 previously installed");
>    }
>  }
>}

don't you need `if (experiments && ...` in case it's not defined for one of the days? or is it guaranteed to be?

Comment 7

4 years ago
Created attachment 8522511 [details]
previous-experiment-filter.js

Yes I do.
Attachment #8522463 - Attachment is obsolete: true
Attachment #8522463 - Flags: review?(felipc)
Attachment #8522511 - Flags: review?(felipc)
(Assignee)

Updated

4 years ago
Attachment #8522511 - Flags: review?(felipc) → review+
(Assignee)

Comment 8

4 years ago
I pushed the experiment (without the filtering function) for testing on people.mozilla.org. Please don't post the URL publicly.

To test:

experiments.manifest.uri                    https://people.mozilla.org/~fgomes/searchtest2/firefox-manifest.json
experiments.force-sample-value              string "0.1"
experiments.logging.dump                    true
experiments.manifest.cert.checkAttributes   false
experiments.logging.level                   0



set the env MOZ_SEARCH_EXPERIMENT_BRANCH=3 when running FF to choose which branch to take


what to test:
  branch 1 = control group, no change
  branch 2 = search engine changed to yahoo
  branch 3 = search engine changed to yahoo, URL when searching changed to www.example.com/search

branches 1 - 3 should be picked at random with equal chance

branch 0 will be assigned to users who don't have Google as their search engine

if the searchbar is customized away from the nav-bar, any branch will be marked with "a"


This experiment only applies to beta, en-US, and the minBuildID is 20140630004007 (it will change on the final version)

Comment 9

4 years ago
Tomorrow can you push a new version with the filter function and with no sample (or a 1.0 sample)? That's what we're going to ship anyway, and I think that it's going to be important to QA that the filter works correctly.
(Assignee)

Comment 10

4 years ago
I removed the sample from the no-filter version, and published a version with the filter function at
-> https://people.mozilla.org/~fgomes/searchtest2-withfilter/firefox-manifest.json

I also republished a version with the first experiment, which needs to be installed first in order to test the filter. It's at
-> https://people.mozilla.org/~fgomes/searchtest1/firefox-manifest.json
I adjusted the endTime and maxVersion to allow it to be installed on the current version.

Updated

4 years ago
Iteration: 36.3 → 37.1
I think we're no longer pursuing these experiments, right?
Status: ASSIGNED → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → WONTFIX
Group: mozilla-employee-confidential

Updated

4 years ago
Iteration: 37.1 → ---

Updated

3 years ago
Attachment #8521178 - Flags: feedback?(kev)
You need to log in before you can comment on or make changes to this bug.