[Shield] Unified Search Bar Study: Add-on version 3.0

RESOLVED FIXED

Status

Shield
Shield Study
RESOLVED FIXED
4 months ago
22 days ago

People

(Reporter: shilpi, Assigned: Paolo)

Tracking

(Blocks: 1 bug)

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(4 attachments, 5 obsolete attachments)

(Reporter)

Description

4 months ago
1. Testing result composition: We don't yet know the optimal result composition in a unified Awesome Bar. We're inclined to believe it's not what we have now (10 results, history first). We need ASAP to test that in Shield.  Our ENG plan all along was to land pref'able options prior to 57 and that we could use off-the-trains tech (i.e. Shield) to experiment with composition and even work right up to November to deploy the final best option via Shield, if that's needed.

Panos team landed new pref-able options in 56 to allow us to put search suggestions on top (right now they're only bottom) and also to allow us to vary the number of Awesome Bar results (such as 5 total results) where a user doesn't just see 5 history results, but sees at least a few search too. 

    
Branches for Testing Result Composition and Unified vs. 2 Bar preferences:

    Control: Normal Fx (two separate bars, as they exist now)
    Unified Bar with 10 results, history on top (currently default under unification)
    Unified Bar with 10 results, search on top
    Unified Bar with 5-6 results, history on top
    Unified Bar with 5-6 results, search on top
    Unified Bar with 10 results, history on top (currently default under unification) + top hit on top
    Unified Bar with 10 results, search on top + top hit on top
    Unified Bar with 5-6 results, history on top + top hit on top
    Unified Bar with 5-6 results, search on top + top hit on top


2. Testing to understand who gets unified vs. keeps search bar.
(Reporter)

Comment 1

4 months ago
Current plan and next steps:

1. Experiment design document is WIP - Javaun ETA: Aug 7
2. Coding add on for 8+ branches with 3 prefs - Paolo ETA: Aug 9
3. Generate survey URL: Matt G
4. File shield + legal bug: Shilpi & Javaun
5. Write up survey questions and translate for top tier markets: Javaun
6. QA on Add on - Softvision/ PI (Shilpi to figure QA resource)
7. Intent to ship email - Shilpi
8. Launch shield in 56 beta: tentative ETA Aug 15

9. Collect and Analyse Data
10. Relaunch version 2 in 56 Release

Updated

4 months ago
Depends on: 1387323
(Reporter)

Comment 2

3 months ago
Latest status:

* Add on development in progress. ETA end of week.  
* PI request for QA filed.
* Experiment design here: https://docs.google.com/document/d/1AstlOsrwDXQFR2ruLEy4PpeqPiJA-36XtZUZyr3e2bc/edit 
* Legal bug for the study approved
(Reporter)

Comment 3

3 months ago
Matt, Can you please update on the status of opt out uplifting to beta? Is it still on track for this week? 
Secondly, can you send us a blank survey URL?

Thanks
Shilpi
Flags: needinfo?(mgrimes)

Comment 4

3 months ago
+ The Normandy engineering team for status on landing the action
Flags: needinfo?(mgrimes) → needinfo?(mcooper)
(Assignee)

Comment 5

3 months ago
To have search suggestions always present at the top:

"browser.search.widget.inNavBar": false
"browser.urlbar.maxRichResults": 10
"browser.urlbar.matchBuckets": "suggestion:5,general:4"

To have history always present at the top and less results:

"browser.search.widget.inNavBar": false
"browser.urlbar.maxRichResults": 6
"browser.urlbar.matchBuckets": "general:3,suggestion:2"

To get a second top result that is the opposite type of the first:

"browser.search.widget.inNavBar": false
"browser.urlbar.maxRichResults": 10
"browser.urlbar.matchBuckets": "suggestion:1,general:4,suggestion:4"
"browser.urlbar.matchBucketsSearch": "general:1,suggestion:4,general:4"

This requires Firefox 56 and should work in Firefox 57 Nightly as well.
> Can you please update on the status of opt out uplifting to beta? Is it still on track for this week?

We have never planned to uplift to Beta this week. We are planning to land on Nightly and begin QA this week. We need to complete QA before we can uplift to Beta. Our current target is the end of August for having opt-out studies on Beta.
Flags: needinfo?(mcooper)
(Reporter)

Comment 7

3 months ago
Matt and Mike, 

We were told that opt out was being uplifted to beta this week. From c6 above, looks like it wasn't planned.
There seems to be some disconnect on this. Can someone throw some light?

Thanks
Shilpi
Flags: needinfo?(mgrimes)
(Assignee)

Comment 8

3 months ago
We have discussed the creation of a rolling study that uses the same deployment method and data collection infrastructure as the previous version from bug 1359118. This will be in parallel to the investigation of other study venues. I'm creating a new version of the study add-on with the following changes:

1. The add-on can be installed in Firefox 55 and Firefox 56. The study period will not start before the profile is updated to Firefox 56. For 25% of the participants there will still be a guaranteed delay of 15 days before the study starts, but will be absorbed if the experiment is installed in Firefox 55 first.

2. The study will now run for 60 days from the date of first installation. The study can also be deployed as an update to a previous version, in which case the date of first installation is the original one. Data from these updated profiles can be used or discarded by looking at the first-run revision that is already sent in the telemetry ping.

3. Since one-off search buttons and search suggestions are present by default in Firefox 56, they are not explicitly enabled, and there is no special code to test variations of the onboarding tip anymore.

4. The study branches now test the "result composition" by setting the new preferences available in Firefox 56. The code to restore the original preferences when the study has ended was updated to handle these special hidden preferences correctly.

5. All the existing diagnostic measurements are kept, with the addition of the result composition preferences.

6. It is now possible to test a new study branch by changing the "variation" preference in about:config and restarting.
(Assignee)

Comment 9

3 months ago
Created attachment 8896975 [details]
Signed study add-on version 3.0.0b

We may still have to edit the study branches to reflect what we need more precisely, and we still have to update the survey URL, but we can start some preliminary testing on this version for the changes mentioned above.
(Assignee)

Comment 10

3 months ago
TEST PLAN - INSTALLATION
------------------------

1. Create a new profile in Firefox 55, and start the browser with that profile.

2. For convenience, pin the tabs for "about:addons", "about:telemetry", and "about:config", unchecking the checkbox on the latter.

3. Download the add-on attached to this bug and install it, for example by dragging the XPI file to the "about:addons" tab.

4. Verify that the installed version in "about:addons" matches the latest attachment, at the time of this comment it is "3.0.0b".

5. Go to "about:config" and search for "study.shield". Keep this filter, because you'll have to look at these preferences often.

6. Verify that the preferences for "@unified-urlbar-shield-study" are present. The rest of the steps will refer to these preferences using only the last part of their name.

7. Restart the browser. The add-on should still be installed and the preferences present.
(Assignee)

Comment 11

3 months ago
TEST PLAN - DATA COLLECTION
---------------------------

1. Go to "Options", "Advanced", "Data Choices". Verify that "Share additional data (i.e., Telemetry)" is already checked.

2. Go to "about:temeletry", select "Archived ping data", and verify that some of the entries in the "Ping" dropdown have the type "shield-study".

3. Select one of the "shield-study" pings, expand the "General Data" section below, and look for the "clientId" value at the end.

4. Paste the client ID in this bug, so we can use it to verify the pings have been received on the server side.
(Assignee)

Comment 12

3 months ago
TEST PLAN - DAILY DATA COLLECTION
---------------------------------

1. Ensure you are on the same profile as all the other tests, so the client ID is the same.

2. Perform the following actions a variable number of times:

- Start a search by typing in the address bar and pressing ENTER.
- Navigate to a search engine by typing "yahoo.com" or "google.com" and pressing ENTER, then start a search from the search engine's home page instead of the Firefox interface.
- If the search bar is visible, start a search from it.

3. With the study installed and the browser open, move the computer clock forward by one day.

4. Send the computer to sleep and resume.

5. Go to "about:telemetry", refresh the page, and check that the most recent "shield-study" ping is from the new day.

6. Repeat step 2, but with a different number of actions of each type. Fore example, if you did two searches from the address bar, do only one search now.

7. Restart the browser, this time without moving the computer clock.

8. Repeat steps 3 to 6 two more times without restarting the browser in-between, then close the browser.

9. Now you can restore the computer clock.
(Assignee)

Comment 13

3 months ago
TEST PLAN - ACTIVATION ON BROWSER UPDATE
----------------------------------------

1. In Firefox 55, ensure that:

- The "variation" preference does *not* have the value "control"
- The "phasedVariation" preference has the value "false"

If this is not the case, uninstall the add-on from "about:addons" and reinstall it until the values become the expected ones. Uninstalling the add-on displays a survey, but you can close the tab as you don't have to complete the survey.

2. Verify that the search bar is still visible in the toolbar.

3. Close Firefox 55, then start Firefox 56 with the same profile where the experiment was installed.

4. Verify that the search bar is not visible in the toolbar anymore.
(Assignee)

Comment 14

3 months ago
TEST PLAN - END OF STUDY
------------------------

1. In Firefox 56, check that the preference "browser.urlbar.matchBuckets" exists and has been assigned a custom value.

2. Move the computer clock forward by 50 days and restart the browser.

3. Verify that the preference is still present and the search bar is not visible in the toolbar.

4. Move the computer clock forward by 20 more days, for a total of 70 days, and restart the browser.

5. The end of study survey should be displayed, and the URL should include "reason=end-of-study".

6. Check that the preference "browser.urlbar.matchBuckets" does not exist anymore.

7. Verify that the search bar has been restored in the toolbar.

8. In "about:telemetry", verify that the state is "Telemetry is enabled and extended telemetry is disabled".

9. Now you can restore the computer clock.
I've started to test here but I ran into some issues with Ubuntu (suspend mode) and Windows 10 (changing date), I'll have to try on Mac once we are back at the office. 
I've tested until step 3 from comment 12 where I got into problems, but so far everything looks good. Will be back Wednesday (since we have a holiday tomorrow and we will not be at the office) and conclude the testing.
Also here are the clientIds I ran into during testing:
"clientId": "bf7b1a4c-889e-4c56-b643-7cd71574683d"
"clientId": "1d14474a-e08d-4544-a2e2-54b98519e423"
"clientId": "1dc500da-9f14-4890-8080-ace8c5c6fc4a"
(Assignee)

Comment 17

3 months ago
After the study add-on is installed in Firefox 56, the branches can be tested by changing this preference in about:config and restarting:

  extensions.@unified-urlbar-shield-study.shield.variation

In version 3.0.0b of the add-on, there are 10 branches defined, plus the control branch. I'm providing a summary of the current settings without describing the implementation details at this time, because we're still defining the exact behavior we want to test.

The branch names are:

  control                10 results: search bar still present
  unified-topfirst       10 results: 1 heuristic, 1 "top" opposite, 4 same, 4 opposite
  unified-searchfirst     9 results: 1 heuristic, 4 suggestions, 4 history
  unified-historyfirst    9 results: 1 heuristic, 4 history, 4 suggestions
  unified-mostsearch      9 results: 1 heuristic, 8 suggestions
  unified-mosthistory     9 results: 1 heuristic, 8 history
  minimal-topfirst        8 results: 1 heuristic, 1 "top" opposite, 3 same, 3 opposite
  minimal-searchfirst     7 results: 1 heuristic, 3 suggestions, 3 history
  minimal-historyfirst    7 results: 1 heuristic, 3 history, 3 suggestions
  minimal-mostsearch      7 results: 1 heuristic, 6 suggestions
  minimal-mosthistory     7 results: 1 heuristic, 6 history

The "mostsearch" and "mosthistory" are control branches that will likely not be implemented, but are interesting for getting comparative data. We may want to remove some of them from the mix but still keep others.
(Assignee)

Comment 18

3 months ago
Since we can push updates to the add-on, we don't necessarily need the final survey URL to start onboarding new participants.

We can also adjust study branches, push in-flight surveys before the end of the study, or choose to terminate the experiment earlier or later based on arbitrary criteria.
Flags: needinfo?(mgrimes)
Link sent via email. Sorry for the delay.
(Assignee)

Comment 20

3 months ago
Javaun, you mentioned you were working on the definition of the study branches. Any progress there, or any updates for the branches in comment 17?
Flags: needinfo?(jmoradi)
(Reporter)

Updated

3 months ago
Blocks: 1390592
I finished testing version 3.0.0b of the add-on across platforms (Windows 10 64bit, macOS 10.12.6 and Ubuntu 16.04 64bit), and the only issue I found is that after the ending of the study the pref "browser.urlbar.matchBuckets" is still visible in about:config and has the same value as before (comment 14 step 6).
(Assignee)

Comment 22

3 months ago
Created attachment 8897856 [details]
Signed study add-on version 3.0.2b

This fixes the bug from comment 21, loads on Firefox 56, and updates the branches.
Attachment #8896975 - Attachment is obsolete: true
(Assignee)

Comment 23

3 months ago
Bogdan, I've also found a new bug related to direct installation instead of upgrade. Can you try the steps from comment 10 on a new profile directly in Firefox 56 and see if they work?

I'll post some new tests for the branches that are currently in the add-on.
Flags: needinfo?(bogdan.maris)
(Assignee)

Comment 24

3 months ago
TEST PLAN - STUDY BRANCHES
--------------------------

1. Ensure that the installed version of the experiment is 3.0.2b and it is running on Firefox 56.

2. Ensure that the "phasedVariation" preference is set to "false", changing it if necessary.

3. Change the "variation" preference to one of the values listed below.
   IMPORTANT: The value should be typed in lowercase and without extra spaces before or after.

4. Restart the browser.

5. Verify that the preferences under "browser.urlbar." are either set to the expected values or absent, depending on the branch.


Variation                maxRichResults / matchBuckets / matchBucketsSearch

  control                10 (search bar still present)
  unified-topfirst       11 / suggestion:1,general:5,suggestion:4 / general:1,suggestion:4,general:5
  unified-historyfirst   10 / general:5,suggestion:4
  unified-searchfirst    10 / suggestion:4,general:5
  unified-samefirst      10 / general:5,suggestion:4 / suggestion:4,general:5
  unified-mosthistory    10 / general:9
  unified-mostsearch     10 / suggestion:9
  minimal-topfirst        8 / suggestion:1,general:3,suggestion:3 / general:1,suggestion:3,general:3
  minimal-historyfirst    7 / general:3,suggestion:3
  minimal-searchfirst     7 / suggestion:3,general:3
  minimal-samefirst       7 / general:3,suggestion:3 / suggestion:3,general:3
  minimal-mosthistory     7 / general:6
  minimal-mostsearch      7 / suggestion:6
I verified the steps from comment 24, I only tried 2 variations (unified-historyfirst and minimal-mosthistory) on Ubuntu (due to lack of time) and they worked as expected (meaning the browser.urlbar preferences had the expected values), I will try a few more tomorrow, across all platforms and also the request from comment 23. It's good to point out that I used the new version of the add-on 3.0.2b.
I added some (the total branches swelled to 13) and then I proposed cuts. I kept all the smaller result sets (8 or less) and cut the bigger ones. In addition to the dynamic "top opposite" you have, I created a "top same". We also have a dynamic smallset, where the first bucket of results is always the same as the autocomplete. 

Here's the branches broken down by prefs. I need to QA this. 
https://docs.google.com/document/d/1AstlOsrwDXQFR2ruLEy4PpeqPiJA-36XtZUZyr3e2bc/edit#heading=h.93rt7oafqxmz


================

  control                10 results: search bar still present



  minimal-historyfirst    7 results: 1 heuristic, 3 history, 3 suggestions
  minimal-searchfirst     7 results: 1 heuristic, 3 suggestions, 3 history

  fixed-tophit		   8 results: 1 heuristic, 1 history, 3 suggestions, 3 history


  dynamic-smallset        7 results: 1 heuristic: 3 same, 3 opposite

  dynamic-top-opposite    8 results: 1 heuristic, 1 "top" opposite, 3 same, 3 opposite
  dynamic-top-same        8 results: 1 heuristic, 1 "top" same, 3 opposite, 3 same



//------------------------------------------------

//Consider DROPPING...

  unified-historyfirst    9 results: 1 heuristic, 4 history, 4 suggestions
  unified-searchfirst     9 results: 1 heuristic, 4 suggestions, 4 history

  minimal-mostsearch      7 results: 1 heuristic, 6 suggestions
  minimal-mosthistory     7 results: 1 heuristic, 6 history

  dynamic-bigset         10 results: 1 heuristic: 4 same, 5 opposite

  fixed-tophit		  10 results: 1 heuristic, 1 history, 4 suggestions, 4 history
Flags: needinfo?(jmoradi)
(In reply to Bogdan Maris, QA [:bogdan_maris] from comment #25)
> I verified the steps from comment 24, I only tried 2 variations
> (unified-historyfirst and minimal-mosthistory) on Ubuntu (due to lack of
> time) and they worked as expected (meaning the browser.urlbar preferences
> had the expected values), I will try a few more tomorrow, across all
> platforms and also the request from comment 23. It's good to point out that
> I used the new version of the add-on 3.0.2b.

Just finished testing using add-on version 3.0.2b using steps from comment 24 and request from comment 23 across platforms. Everything worked as expected.

Here are the clientIds used for today testing:

macOS: "clientId": "65f4f536-3f97-7548-b417-43f7a5538841"
Windows 10: "clientId": "9f322061-7fe4-4456-9403-7dd1db33eec9"
Linuxx 64: "clientId": "334105a3-ef0c-44d0-bf4d-11e36cc86ec8"

I also verified across platforms using 3.0.2b that "browser.urlbar.matchBuckets" is successfully removed (along with the Search Shield Study add-on from about:addons / Extensions) from about:config once the study has ended (comment 14 step 6).
Flags: needinfo?(bogdan.maris)
(Assignee)

Comment 28

3 months ago
Updating title per bug 1390592 comment 3.
Summary: [Shield] Opt-out Study: Unified Search 3 → [Shield] Unified Search Bar Study: Add-on version 3.0
(Assignee)

Comment 29

3 months ago
This is a new full matrix that cross-references the branches from the opt-out pref-flipping study in bug 1390584 comment 12.

It is close to the branches listed in comment 26 with some small variations to match the pref-flipping study.


Variation                  maxRichResults / matchBuckets / matchBucketsSearch

=1  minimal-top-history     7 / general:1,suggestion:3,general:2
    minimal-top-same        7 / general:1,suggestion:3,general:2 / suggestion:1,general:2,suggestion:3
    minimal-top-opposite    7 / suggestion:1,general:2,suggestion:3 / general:1,suggestion:3,general:2
=2  minimal-history         7 / general:3,suggestion:3
=3  minimal-search          7 / suggestion:3,general:3
    minimal-same            7 / general:3,suggestion:3 / suggestion:3,general:3
~4  minimal-mosthistory     7 / general:6
    minimal-mostsearch      7 / suggestion:6
=5  unified-top-history    10 / general:1,suggestion:3,general:5
//  unified-top-same       10 / general:1,suggestion:3,general:5 / suggestion:1,general:5,suggestion:3
//  unified-top-opposite   10 / suggestion:1,general:5,suggestion:3 / general:1,suggestion:3,general:5
=8  unified-history        10 / general:5,suggestion:4
    unified-search         10 / suggestion:4,general:5
//  unified-same           10 / general:5,suggestion:4 / suggestion:4,general:5
//  unified-mosthistory    10 / general:9
//  unified-mostsearch     10 / suggestion:9
    control                10 (search bar still present)


Branches marked with "//" can be excluded. For the "mostsearch" and "mosthistory" branches I propose to keep the "minimal" variation of them, given the similarity of the "minimal-mosthistory" branch with what we're doing in the pref-flipping study.

Note that the "unified-history" branch is a control branch that should match what we've already tested in the previous study. It makes sense to me to keep "unified-search" as it is symmetric to both "unified-history" and "minimal-search".
(Assignee)

Comment 30

3 months ago
Created attachment 8898348 [details]
Signed study add-on version 3.0.4b
Assignee: shilpi → paolo.mozmail
Attachment #8897856 - Attachment is obsolete: true
Status: NEW → ASSIGNED
(Assignee)

Comment 31

3 months ago
Version 3.0.4b of the add-on implements the branches from comment 29. To test them, you can install the add-on on Firefox 56 and set the variation preferences in about:config.

extensions.@unified-urlbar-shield-study.shield.phasedVariation
  Ensure this is set to false.

extensions.@unified-urlbar-shield-study.shield.variation
  Set this to the branch name, lowercase and without extra spaces before or after.
  Restart the browser for the changes to take effect.

The valid branch names are:

minimal-top-history
minimal-top-same
minimal-top-opposite
minimal-history
minimal-search
minimal-same
minimal-mosthistory
minimal-mostsearch
unified-top-history
unified-history
unified-search
control

Comment 32

3 months ago
On the data side, I ran a quick verification of the data coming back from the test client running 3.0.3, and everything looks as expected.

I created a table of the reported payload values at https://docs.google.com/spreadsheets/d/1C_b8LLF1GIqX_a3HnMNMa9zVfVBa_SAPtr49Nd-onZQ/edit#gid=759867014
(Assignee)

Comment 33

3 months ago
Created attachment 8900379 [details]
Signed study add-on version 3.0.5b

This version has the following changes:

1. For the duration of the study, a new entry is added to the "experiments" list in the Telemetry environment.

2. A new "installMode" field is set to "opt-in", "opt-out", or rarely "undetermined" based on how the add-on was installed.

3. Extended Telemetry is not enabled if the add-on is installed in opt-out mode.

Opt-out mode is assumed if the recipeId of any installed add-on-based study contains the string "urlbar" case-insensitively. This includes completed studies.
Attachment #8898348 - Attachment is obsolete: true
(Reporter)

Comment 34

3 months ago
Paolo, 

Can you please work with Marco to get the latest version of the addon uploaded to the consent page? I think the consent page shows 2.2.1 which is the old one. 

https://addons.mozilla.org/en-US/firefox/shield_study_8

Please confirm once the latest version is uploaded. Shield team can begin enrollment soonafter.

Thanks
Shilpi
Flags: needinfo?(paolo.mozmail)
(Assignee)

Comment 35

3 months ago
(In reply to shgupta@mozilla.com from comment #34)
> I think the consent page shows 2.2.1 which is the old one.

As I specified already in private e-mail and on IRC, we are blocked on the review from the addons.mozilla.org (AMO) reviewers. We would have been ready to start onboarding last week, as the add-on was ready and uploaded. I think you requested an expedited review and I also independently sent a private e-mail on August 18 to that effect to AMO administrators.

Philipp Kewisch replied with some questions on August 22 but he hasn't since marked the add-on as reviewed. Yesterday I uploaded the updated version 3.0.5 with the changes listed in comment 33. At the time of the upload, the previous version 3.0.4 wasn't reviewed.
Flags: needinfo?(paolo.mozmail) → needinfo?(shilpi)
(Reporter)

Comment 36

3 months ago
Thanks for the clarification Paolo! 
AMO has reviewed the addon and I can follow up with them to make sure it is marked for you to upload.

Can you point me where to look for add-on marked as reviewed? Is it just r+ on this bug or elsewhere?
Flags: needinfo?(shilpi)
Comment on attachment 8900379 [details]
Signed study add-on version 3.0.5b

Review outside of AMO is currently just a matter of sending email, I'm happy to set r+ if you prefer. If you would like us to review any version please do send us an email or request review.
Attachment #8900379 - Flags: review+
(Reporter)

Comment 38

3 months ago
Sounds great Philipp! Thanks for letting us know.

Paolo, can you please confirm once the latest addon is uploaded on the study consent page here?

https://addons.mozilla.org/en-US/firefox/shield_study_8

Thank you both!
Flags: needinfo?(paolo.mozmail)
(Assignee)

Comment 39

3 months ago
We actually need a review on the "addons.mozilla.org" (AMO) listing, hopefully we can get that soon.
Flags: needinfo?(paolo.mozmail)
(Assignee)

Comment 40

3 months ago
The add-on is now reviewed and we should be ready to start onboarding!
(Reporter)

Comment 41

2 months ago
Hi Paolo,

I am being told that we are now ready to launch the opt out shield study using the addon that will enable us to test the expanded set of branches using all prefs. I will be sending the intent to ship email and obtain relman approval for beta 56 shortly.

Please confirm the latest version of addon that is ready to be deployed for the opt out study. I think it is add-on version 3.0.4b, but want to be sure.

Secondly, will we need a round of QA and/or AMO review or are we good?

Thanks
Shilpi
Flags: needinfo?(paolo.mozmail)
(Reporter)

Comment 42

2 months ago
Hey Matt, Can you approve the opt out unified search shield study here?

Design Doc: https://docs.google.com/document/d/1AstlOsrwDXQFR2ruLEy4PpeqPiJA-36XtZUZyr3e2bc/edit#heading=h.93rt7oafqxmz
Flags: needinfo?(mgrimes)
(Assignee)

Comment 43

2 months ago
(In reply to shgupta@mozilla.com from comment #41)
> Please confirm the latest version of addon that is ready to be deployed for
> the opt out study. I think it is add-on version 3.0.4b, but want to be sure.

The important bit of information is that, per comment 33, the recipeId of the opt-out study should contain the string "urlbar" case-insensitively. We can then use this as the add-on installation URL:

https://addons.mozilla.org/downloads/latest/unified-urlbar-shield-study/addon-749812-latest.xpi

We should also make sure that the onboarding for the opt-in study starts now for Release 55 in addition to Beta 56, to prepare for the opt-in studies in Release 56.

> Secondly, will we need a round of QA and/or AMO review or are we good?

We don't need a new review because the add-on is the same, but we should verify that when the add-on is installed using the recipeId specified above then extended telemetry is not enabled.
Flags: needinfo?(paolo.mozmail)
Approved!
Flags: needinfo?(mgrimes)
As per Paolo's comment 31, here is the final list of branches. I added the pref's behind them. First pref (int) is the maxRichResults number. The second pref is: browser.urlbar.matchBuckets (str - this is the primary bucket to vary composition), and 3 branches have a third pref (str) which is the value of browser.urlbar.matchBucketsSearch

    minimal-top-history     7 / general:1,suggestion:3,general:2
    minimal-top-same        7 / general:1,suggestion:3,general:2 / suggestion:1,general:2,suggestion:3
    minimal-top-opposite    7 / suggestion:1,general:2,suggestion:3 / general:1,suggestion:3,general:2
    minimal-history         7 / general:3,suggestion:3
    minimal-search          7 / suggestion:3,general:3
    minimal-same            7 / general:3,suggestion:3 / suggestion:3,general:3
    minimal-mosthistory     7 / general:6
    minimal-mostsearch      7 / suggestion:6
    unified-top-history    10 / general:1,suggestion:3,general:5
    unified-history        10 / general:5,suggestion:4
    unified-search         10 / suggestion:4,general:5
    control                10 (search bar still present)
(Reporter)

Comment 46

2 months ago
Thanks Paolo! We are now preparing to launch opt-out study using the addon with 12 branches. Target launch date: Sep 7
I have sent the intent to ship email to relman. Once it is approved we can begin enrollment.

I have a few questions regarding readiness of addon:

1) Who needs to add the recipeID string you mentioned in the addon? Or is that already contained in the link below? 
https://addons.mozilla.org/downloads/latest/unified-urlbar-shield-study/addon-749812-latest.xpi

2) How can we verify that extended telemetry is not enabled upon install of this addon? 

3) Does the opt out addon contain the survey URL? (Old URL is fine too and will work) 


Re: opt-in study enrollment, What is the benefit of on-boarding opt-in users in release 55 since the prefs we are using are not available in 55. If there is a need we can do it. I want to validate the benefit.
Flags: needinfo?(paolo.mozmail)
(Assignee)

Comment 47

2 months ago
(In reply to shgupta@mozilla.com from comment #46)
> 1) Who needs to add the recipeID string you mentioned in the addon? Or is
> that already contained in the link below? 
> https://addons.mozilla.org/downloads/latest/unified-urlbar-shield-study/
> addon-749812-latest.xpi

No, I think a recipe is a remotely sent instruction to install an add-on from a given URL. So, the URL and the recipeId are parameters of the recipe. I imagine that the recipe is created by those responsible for the deployment of the experiment.

> 2) How can we verify that extended telemetry is not enabled upon install of
> this addon? 

In "about:telemetry", verify that the state is "Telemetry is enabled and extended telemetry is disabled".

> 3) Does the opt out addon contain the survey URL? (Old URL is fine too and
> will work) 

Yes, the use the same survey URL. The can be distinguished using the clientId.

> Re: opt-in study enrollment, What is the benefit of on-boarding opt-in users
> in release 55 since the prefs we are using are not available in 55. If there
> is a need we can do it. I want to validate the benefit.

When version 56 is released in less than 3 weeks, we will already have a cohort of Release users who chose to participate in the experiment, and they will get the new experience right away when the browser is updated, saving us quite some time. We can also choose to update the branches that are offered to this group of users at any time.
Flags: needinfo?(paolo.mozmail)
Adding 55 (even if they won't get the experience until 56) sounds like creep at this point. We'd need additional relman approvals for that. We will be able to scale up quickly in 56 Release even if we wait until launch.
(Reporter)

Comment 49

2 months ago
Hi Paolo,

Shield team has asked for the unsigned version of the addon to ship the opt-out study. They can't use the add-on on AMO as the installation URL, opt-out studies need to be hosted on the Shield service. Once you can provide the unsigned add-on Shield team will sign and upload it.

Thanks
Shilpi
Flags: needinfo?(paolo.mozmail)
(Reporter)

Comment 50

2 months ago
1) The recipeID field seems to be a numeric ID. Panos is hardcoding the urlbar string in it. 
2) Addon ID will need to be changed from the opt in addon ID so that can differentiate the survey answers coming in from optin vs opt out.

Once these changes are in and tested, we should be able to launch the study! Thanks all for the prompt work :)
Created attachment 8905617 [details]
Unsigned study v3.0.5 with recipeId check removed and addon ID updated

I removed the signature and the check for recipeId (turns out it is a number), so we always take the opt-out branch. I also updated the add-on ID (@unified-urlbar-shield-study-opt-out) to distinguish it from the opt-in study.
Attachment #8900379 - Attachment is obsolete: true
(Reporter)

Comment 52

2 months ago
Thank you Panos! Matt, Please confirm once we are live as pilot opt-out! :)
I replied on the email thread to give relman's OK to ship this in beta 56 until Sept. 26. Thanks!
Study is now live on Beta for all locales. We'll end the study on 9/26.
(Assignee)

Comment 55

2 months ago
Thanks!
Flags: needinfo?(paolo.mozmail)

Updated

2 months ago
Depends on: 1400335
Is it possible to disable the experiment when keyword.enabled is set to false ? Not having the search bar in this case is a bit sad :)
It is too late to make updates for this particular study, but we can take this into consideration for future ones.
(Reporter)

Comment 58

a month ago
We plan to re-ship the unified search shield study (v4) to Rel 56 to validate our findings from beta (v3) and resolving any differences.

The last study, Unified v3 shield study ran for 25% of users in beta and ended on 9/26. This new study, Unified v4 shield will run for about 2% of release users in 56 (all locales).

It is the same experiment that is testing for result composition of the awesome bar.

Please let Javaun or I know if you have any questions.
Sounds like this is ready to ship to 56 release. Please go ahead. 
What is the end date?
status-firefox56: --- → affected
tracking-firefox56: --- → +
(Assignee)

Comment 60

a month ago
Created attachment 8916720 [details]
Unsigned study add-on version 3.0.6

This version is updated so that the study runs for 30 days, and for 25% of the users the changes are applied after 7 days. There are no changes related to the method used to hide the search bar, as this was not deemed an important change to make in this version.
Attachment #8905617 - Attachment is obsolete: true
(Assignee)

Comment 61

a month ago
Created attachment 8916721 [details]
Unsigned study add-on version 3.0.6 with alternative ID

This is the same version but with the ID changed, which I understand is a requirement of the deployment method that will be used.
Created attachment 8916822 [details]
unified-urlbar-shield-study-opt-out-new-users-3.0.6-signed.xpi

I've signed the add-on from comment 61 and uploaded it to Shield under the name "Unified Search Bar Study 3.0.6 (New Users)". I've also attached the signed add-on to this bug.
Created attachment 8916823 [details]
unified-urlbar-shield-study-opt-out-3.0.6-signed.xpi

I've signed the add-on from comment 60 and uploaded it to Shield under the name "Unified Search Bar Study 3.0.6 (Old Users)". I've also attached the signed add-on to this bug.
Both of these have been pushed live on the 56 Release channel

Comment 65

a month ago
We're getting a number of frustrated posters on SuMo. They try to restore their search bar in Customize, but when they exit Customize, the search bar will not appear. Is this the intended design?

For example:

https://support.mozilla.org/questions/1179570
https://support.mozilla.org/questions/1179590
https://support.mozilla.org/questions/1179601
https://support.mozilla.org/questions/1179611
https://support.mozilla.org/questions/1179612
https://support.mozilla.org/questions/1179613
(Assignee)

Comment 66

a month ago
Yes, this design was originally introduced in the second version of the opt-in experiment to work around some uncertainty in the data that was collected by the first version. The uncertainty was related to the search bar being reported as still being present when in fact it was absent. In bug 1336982, this turned out to be purely a data issue that would not invalidate the results. However, when the deployment method was changed to opt-out and the experiment deployed to Beta, this design was retained.

As I noted in comment 60, at the time of deciding whether the original method should have been restored when deploying the experiment to Release, it wasn't deemed important enough to warrant development and testing time, with the associated delays these would have brought to the deployment and the possible uncertainty in the data collection.
Paolo, is there some workaround for these users reporting the issue in SUMO?  Prefs for them to set, or uninstalling the experiment? 

Will this be different in 57?
Flags: needinfo?(paolo.mozmail)
Is the idea here that, we aren't going to fix this for 56, but will put the option back in 57 for users to either keep or do away with the search bar ?
(Assignee)

Comment 69

29 days ago
(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #67)
> Paolo, is there some workaround for these users reporting the issue in SUMO?
> Prefs for them to set, or uninstalling the experiment? 

Yes, all the changes will be reverted automatically after uninstalling the experiment. This can be done either by uninstalling the add-on manually from the Extensions tab of the Add-ons Manager, or by disabling the "Allow Firefox to install and run studies" checkbox in the Privacy & Security tab of the Options window. The search bar cannot be restored using Customize mode, which would normally work when the experiment is not installed.

(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #68)
> Is the idea here that, we aren't going to fix this for 56, but will put the
> option back in 57 for users to either keep or do away with the search bar ?

The experiment only runs in Firefox 56 for a maximum of 30 days. We don't plan to run this experiment in Firefox 57.

In Firefox 57, the Search tab in the Options window allows to select whether to keep the search bar or not, and of course Customize mode will work as expected.
Flags: needinfo?(paolo.mozmail)
OK. Anything else to do here on this bug?
Flags: needinfo?(paolo.mozmail)
(Assignee)

Comment 71

22 days ago
As far as I know, right now there are no plans to run any other experiment based on this add-on.
Status: ASSIGNED → RESOLVED
Last Resolved: 22 days ago
Flags: needinfo?(paolo.mozmail)
Resolution: --- → FIXED
(Assignee)

Updated

22 days ago
status-firefox56: affected → ---
tracking-firefox56: + → ---
You need to log in before you can comment on or make changes to this bug.