Closed Bug 1387245 Opened 7 years ago Closed 7 years ago

[Shield] Unified Search Bar Study: Add-on version 3.0

Categories

(Shield :: Shield Study, defect)

Product:
Shield
Component:
Shield Study
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: shilpi, Assigned: Paolo)

References

Details

Attachments

(4 files, 5 obsolete files)

Unsigned study add-on version 3.0.6
24.80 KB, application/x-xpinstall
Details
Unsigned study add-on version 3.0.6 with alternative ID
24.81 KB, application/x-xpinstall
Details
unified-urlbar-shield-study-opt-out-new-users-3.0.6-signed.xpi
30.78 KB, application/x-xpinstall
Details
unified-urlbar-shield-study-opt-out-3.0.6-signed.xpi
30.76 KB, application/x-xpinstall
Details 
1. Testing result composition: We don't yet know the optimal result composition in a unified Awesome Bar. We're inclined to believe it's not what we have now (10 results, history first). We need ASAP to test that in Shield.  Our ENG plan all along was to land pref'able options prior to 57 and that we could use off-the-trains tech (i.e. Shield) to experiment with composition and even work right up to November to deploy the final best option via Shield, if that's needed.

Panos team landed new pref-able options in 56 to allow us to put search suggestions on top (right now they're only bottom) and also to allow us to vary the number of Awesome Bar results (such as 5 total results) where a user doesn't just see 5 history results, but sees at least a few search too. 

    
Branches for Testing Result Composition and Unified vs. 2 Bar preferences:

    Control: Normal Fx (two separate bars, as they exist now)
    Unified Bar with 10 results, history on top (currently default under unification)
    Unified Bar with 10 results, search on top
    Unified Bar with 5-6 results, history on top
    Unified Bar with 5-6 results, search on top
    Unified Bar with 10 results, history on top (currently default under unification) + top hit on top
    Unified Bar with 10 results, search on top + top hit on top
    Unified Bar with 5-6 results, history on top + top hit on top
    Unified Bar with 5-6 results, search on top + top hit on top


2. Testing to understand who gets unified vs. keeps search bar.
Current plan and next steps:

1. Experiment design document is WIP - Javaun ETA: Aug 7
2. Coding add on for 8+ branches with 3 prefs - Paolo ETA: Aug 9
3. Generate survey URL: Matt G
4. File shield + legal bug: Shilpi & Javaun
5. Write up survey questions and translate for top tier markets: Javaun
6. QA on Add on - Softvision/ PI (Shilpi to figure QA resource)
7. Intent to ship email - Shilpi
8. Launch shield in 56 beta: tentative ETA Aug 15

9. Collect and Analyse Data
10. Relaunch version 2 in 56 Release
Latest status:

* Add on development in progress. ETA end of week.  
* PI request for QA filed.
* Experiment design here: https://docs.google.com/document/d/1AstlOsrwDXQFR2ruLEy4PpeqPiJA-36XtZUZyr3e2bc/edit 
* Legal bug for the study approved
Matt, Can you please update on the status of opt out uplifting to beta? Is it still on track for this week? 
Secondly, can you send us a blank survey URL?

Thanks
Shilpi
Flags: needinfo?(mgrimes)
+ The Normandy engineering team for status on landing the action
Flags: needinfo?(mgrimes) → needinfo?(mcooper)
To have search suggestions always present at the top:

"browser.search.widget.inNavBar": false
"browser.urlbar.maxRichResults": 10
"browser.urlbar.matchBuckets": "suggestion:5,general:4"

To have history always present at the top and less results:

"browser.search.widget.inNavBar": false
"browser.urlbar.maxRichResults": 6
"browser.urlbar.matchBuckets": "general:3,suggestion:2"

To get a second top result that is the opposite type of the first:

"browser.search.widget.inNavBar": false
"browser.urlbar.maxRichResults": 10
"browser.urlbar.matchBuckets": "suggestion:1,general:4,suggestion:4"
"browser.urlbar.matchBucketsSearch": "general:1,suggestion:4,general:4"

This requires Firefox 56 and should work in Firefox 57 Nightly as well.
> Can you please update on the status of opt out uplifting to beta? Is it still on track for this week?

We have never planned to uplift to Beta this week. We are planning to land on Nightly and begin QA this week. We need to complete QA before we can uplift to Beta. Our current target is the end of August for having opt-out studies on Beta.
Flags: needinfo?(mcooper)
Matt and Mike, 

We were told that opt out was being uplifted to beta this week. From c6 above, looks like it wasn't planned.
There seems to be some disconnect on this. Can someone throw some light?

Thanks
Shilpi
Flags: needinfo?(mgrimes)
We have discussed the creation of a rolling study that uses the same deployment method and data collection infrastructure as the previous version from bug 1359118. This will be in parallel to the investigation of other study venues. I'm creating a new version of the study add-on with the following changes:

1. The add-on can be installed in Firefox 55 and Firefox 56. The study period will not start before the profile is updated to Firefox 56. For 25% of the participants there will still be a guaranteed delay of 15 days before the study starts, but will be absorbed if the experiment is installed in Firefox 55 first.

2. The study will now run for 60 days from the date of first installation. The study can also be deployed as an update to a previous version, in which case the date of first installation is the original one. Data from these updated profiles can be used or discarded by looking at the first-run revision that is already sent in the telemetry ping.

3. Since one-off search buttons and search suggestions are present by default in Firefox 56, they are not explicitly enabled, and there is no special code to test variations of the onboarding tip anymore.

4. The study branches now test the "result composition" by setting the new preferences available in Firefox 56. The code to restore the original preferences when the study has ended was updated to handle these special hidden preferences correctly.

5. All the existing diagnostic measurements are kept, with the addition of the result composition preferences.

6. It is now possible to test a new study branch by changing the "variation" preference in about:config and restarting.
Attached file Signed study add-on version 3.0.0b (obsolete) — 
We may still have to edit the study branches to reflect what we need more precisely, and we still have to update the survey URL, but we can start some preliminary testing on this version for the changes mentioned above.
TEST PLAN - INSTALLATION
------------------------

1. Create a new profile in Firefox 55, and start the browser with that profile.

2. For convenience, pin the tabs for "about:addons", "about:telemetry", and "about:config", unchecking the checkbox on the latter.

3. Download the add-on attached to this bug and install it, for example by dragging the XPI file to the "about:addons" tab.

4. Verify that the installed version in "about:addons" matches the latest attachment, at the time of this comment it is "3.0.0b".

5. Go to "about:config" and search for "study.shield". Keep this filter, because you'll have to look at these preferences often.

6. Verify that the preferences for "@unified-urlbar-shield-study" are present. The rest of the steps will refer to these preferences using only the last part of their name.

7. Restart the browser. The add-on should still be installed and the preferences present.
TEST PLAN - DATA COLLECTION
---------------------------

1. Go to "Options", "Advanced", "Data Choices". Verify that "Share additional data (i.e., Telemetry)" is already checked.

2. Go to "about:temeletry", select "Archived ping data", and verify that some of the entries in the "Ping" dropdown have the type "shield-study".

3. Select one of the "shield-study" pings, expand the "General Data" section below, and look for the "clientId" value at the end.

4. Paste the client ID in this bug, so we can use it to verify the pings have been received on the server side.
TEST PLAN - DAILY DATA COLLECTION
---------------------------------

1. Ensure you are on the same profile as all the other tests, so the client ID is the same.

2. Perform the following actions a variable number of times:

- Start a search by typing in the address bar and pressing ENTER.
- Navigate to a search engine by typing "yahoo.com" or "google.com" and pressing ENTER, then start a search from the search engine's home page instead of the Firefox interface.
- If the search bar is visible, start a search from it.

3. With the study installed and the browser open, move the computer clock forward by one day.

4. Send the computer to sleep and resume.

5. Go to "about:telemetry", refresh the page, and check that the most recent "shield-study" ping is from the new day.

6. Repeat step 2, but with a different number of actions of each type. Fore example, if you did two searches from the address bar, do only one search now.

7. Restart the browser, this time without moving the computer clock.

8. Repeat steps 3 to 6 two more times without restarting the browser in-between, then close the browser.

9. Now you can restore the computer clock.
TEST PLAN - ACTIVATION ON BROWSER UPDATE
----------------------------------------

1. In Firefox 55, ensure that:

- The "variation" preference does *not* have the value "control"
- The "phasedVariation" preference has the value "false"

If this is not the case, uninstall the add-on from "about:addons" and reinstall it until the values become the expected ones. Uninstalling the add-on displays a survey, but you can close the tab as you don't have to complete the survey.

2. Verify that the search bar is still visible in the toolbar.

3. Close Firefox 55, then start Firefox 56 with the same profile where the experiment was installed.

4. Verify that the search bar is not visible in the toolbar anymore.
TEST PLAN - END OF STUDY
------------------------

1. In Firefox 56, check that the preference "browser.urlbar.matchBuckets" exists and has been assigned a custom value.

2. Move the computer clock forward by 50 days and restart the browser.

3. Verify that the preference is still present and the search bar is not visible in the toolbar.

4. Move the computer clock forward by 20 more days, for a total of 70 days, and restart the browser.

5. The end of study survey should be displayed, and the URL should include "reason=end-of-study".

6. Check that the preference "browser.urlbar.matchBuckets" does not exist anymore.

7. Verify that the search bar has been restored in the toolbar.

8. In "about:telemetry", verify that the state is "Telemetry is enabled and extended telemetry is disabled".

9. Now you can restore the computer clock.
I've started to test here but I ran into some issues with Ubuntu (suspend mode) and Windows 10 (changing date), I'll have to try on Mac once we are back at the office. 
I've tested until step 3 from comment 12 where I got into problems, but so far everything looks good. Will be back Wednesday (since we have a holiday tomorrow and we will not be at the office) and conclude the testing.
Also here are the clientIds I ran into during testing:
"clientId": "bf7b1a4c-889e-4c56-b643-7cd71574683d"
"clientId": "1d14474a-e08d-4544-a2e2-54b98519e423"
"clientId": "1dc500da-9f14-4890-8080-ace8c5c6fc4a"
After the study add-on is installed in Firefox 56, the branches can be tested by changing this preference in about:config and restarting:

  extensions.@unified-urlbar-shield-study.shield.variation

In version 3.0.0b of the add-on, there are 10 branches defined, plus the control branch. I'm providing a summary of the current settings without describing the implementation details at this time, because we're still defining the exact behavior we want to test.

The branch names are:

  control                10 results: search bar still present
  unified-topfirst       10 results: 1 heuristic, 1 "top" opposite, 4 same, 4 opposite
  unified-searchfirst     9 results: 1 heuristic, 4 suggestions, 4 history
  unified-historyfirst    9 results: 1 heuristic, 4 history, 4 suggestions
  unified-mostsearch      9 results: 1 heuristic, 8 suggestions
  unified-mosthistory     9 results: 1 heuristic, 8 history
  minimal-topfirst        8 results: 1 heuristic, 1 "top" opposite, 3 same, 3 opposite
  minimal-searchfirst     7 results: 1 heuristic, 3 suggestions, 3 history
  minimal-historyfirst    7 results: 1 heuristic, 3 history, 3 suggestions
  minimal-mostsearch      7 results: 1 heuristic, 6 suggestions
  minimal-mosthistory     7 results: 1 heuristic, 6 history

The "mostsearch" and "mosthistory" are control branches that will likely not be implemented, but are interesting for getting comparative data. We may want to remove some of them from the mix but still keep others.
Since we can push updates to the add-on, we don't necessarily need the final survey URL to start onboarding new participants.

We can also adjust study branches, push in-flight surveys before the end of the study, or choose to terminate the experiment earlier or later based on arbitrary criteria.
Flags: needinfo?(mgrimes)
Link sent via email. Sorry for the delay.
Javaun, you mentioned you were working on the definition of the study branches. Any progress there, or any updates for the branches in comment 17?
Flags: needinfo?(jmoradi)
Blocks: 1390592
I finished testing version 3.0.0b of the add-on across platforms (Windows 10 64bit, macOS 10.12.6 and Ubuntu 16.04 64bit), and the only issue I found is that after the ending of the study the pref "browser.urlbar.matchBuckets" is still visible in about:config and has the same value as before (comment 14 step 6).
Attached file Signed study add-on version 3.0.2b (obsolete) — 
This fixes the bug from comment 21, loads on Firefox 56, and updates the branches.
Attachment #8896975 - Attachment is obsolete: true
Bogdan, I've also found a new bug related to direct installation instead of upgrade. Can you try the steps from comment 10 on a new profile directly in Firefox 56 and see if they work?

I'll post some new tests for the branches that are currently in the add-on.
Flags: needinfo?(bogdan.maris)
TEST PLAN - STUDY BRANCHES
--------------------------

1. Ensure that the installed version of the experiment is 3.0.2b and it is running on Firefox 56.

2. Ensure that the "phasedVariation" preference is set to "false", changing it if necessary.

3. Change the "variation" preference to one of the values listed below.
   IMPORTANT: The value should be typed in lowercase and without extra spaces before or after.

4. Restart the browser.

5. Verify that the preferences under "browser.urlbar." are either set to the expected values or absent, depending on the branch.


Variation                maxRichResults / matchBuckets / matchBucketsSearch

  control                10 (search bar still present)
  unified-topfirst       11 / suggestion:1,general:5,suggestion:4 / general:1,suggestion:4,general:5
  unified-historyfirst   10 / general:5,suggestion:4
  unified-searchfirst    10 / suggestion:4,general:5
  unified-samefirst      10 / general:5,suggestion:4 / suggestion:4,general:5
  unified-mosthistory    10 / general:9
  unified-mostsearch     10 / suggestion:9
  minimal-topfirst        8 / suggestion:1,general:3,suggestion:3 / general:1,suggestion:3,general:3
  minimal-historyfirst    7 / general:3,suggestion:3
  minimal-searchfirst     7 / suggestion:3,general:3
  minimal-samefirst       7 / general:3,suggestion:3 / suggestion:3,general:3
  minimal-mosthistory     7 / general:6
  minimal-mostsearch      7 / suggestion:6
I verified the steps from comment 24, I only tried 2 variations (unified-historyfirst and minimal-mosthistory) on Ubuntu (due to lack of time) and they worked as expected (meaning the browser.urlbar preferences had the expected values), I will try a few more tomorrow, across all platforms and also the request from comment 23. It's good to point out that I used the new version of the add-on 3.0.2b.
I added some (the total branches swelled to 13) and then I proposed cuts. I kept all the smaller result sets (8 or less) and cut the bigger ones. In addition to the dynamic "top opposite" you have, I created a "top same". We also have a dynamic smallset, where the first bucket of results is always the same as the autocomplete. 

Here's the branches broken down by prefs. I need to QA this. 
https://docs.google.com/document/d/1AstlOsrwDXQFR2ruLEy4PpeqPiJA-36XtZUZyr3e2bc/edit#heading=h.93rt7oafqxmz


================

  control                10 results: search bar still present



  minimal-historyfirst    7 results: 1 heuristic, 3 history, 3 suggestions
  minimal-searchfirst     7 results: 1 heuristic, 3 suggestions, 3 history

  fixed-tophit		   8 results: 1 heuristic, 1 history, 3 suggestions, 3 history


  dynamic-smallset        7 results: 1 heuristic: 3 same, 3 opposite

  dynamic-top-opposite    8 results: 1 heuristic, 1 "top" opposite, 3 same, 3 opposite
  dynamic-top-same        8 results: 1 heuristic, 1 "top" same, 3 opposite, 3 same



//------------------------------------------------

//Consider DROPPING...

  unified-historyfirst    9 results: 1 heuristic, 4 history, 4 suggestions
  unified-searchfirst     9 results: 1 heuristic, 4 suggestions, 4 history

  minimal-mostsearch      7 results: 1 heuristic, 6 suggestions
  minimal-mosthistory     7 results: 1 heuristic, 6 history

  dynamic-bigset         10 results: 1 heuristic: 4 same, 5 opposite

  fixed-tophit		  10 results: 1 heuristic, 1 history, 4 suggestions, 4 history
Flags: needinfo?(jmoradi)
(In reply to Bogdan Maris, QA [:bogdan_maris] from comment #25)
> I verified the steps from comment 24, I only tried 2 variations
> (unified-historyfirst and minimal-mosthistory) on Ubuntu (due to lack of
> time) and they worked as expected (meaning the browser.urlbar preferences
> had the expected values), I will try a few more tomorrow, across all
> platforms and also the request from comment 23. It's good to point out that
> I used the new version of the add-on 3.0.2b.

Just finished testing using add-on version 3.0.2b using steps from comment 24 and request from comment 23 across platforms. Everything worked as expected.

Here are the clientIds used for today testing:

macOS: "clientId": "65f4f536-3f97-7548-b417-43f7a5538841"
Windows 10: "clientId": "9f322061-7fe4-4456-9403-7dd1db33eec9"
Linuxx 64: "clientId": "334105a3-ef0c-44d0-bf4d-11e36cc86ec8"

I also verified across platforms using 3.0.2b that "browser.urlbar.matchBuckets" is successfully removed (along with the Search Shield Study add-on from about:addons / Extensions) from about:config once the study has ended (comment 14 step 6).
Flags: needinfo?(bogdan.maris)
Updating title per bug 1390592 comment 3.
Summary: [Shield] Opt-out Study: Unified Search 3 → [Shield] Unified Search Bar Study: Add-on version 3.0
This is a new full matrix that cross-references the branches from the opt-out pref-flipping study in bug 1390584 comment 12.

It is close to the branches listed in comment 26 with some small variations to match the pref-flipping study.


Variation                  maxRichResults / matchBuckets / matchBucketsSearch

=1  minimal-top-history     7 / general:1,suggestion:3,general:2
    minimal-top-same        7 / general:1,suggestion:3,general:2 / suggestion:1,general:2,suggestion:3
    minimal-top-opposite    7 / suggestion:1,general:2,suggestion:3 / general:1,suggestion:3,general:2
=2  minimal-history         7 / general:3,suggestion:3
=3  minimal-search          7 / suggestion:3,general:3
    minimal-same            7 / general:3,suggestion:3 / suggestion:3,general:3
~4  minimal-mosthistory     7 / general:6
    minimal-mostsearch      7 / suggestion:6
=5  unified-top-history    10 / general:1,suggestion:3,general:5
//  unified-top-same       10 / general:1,suggestion:3,general:5 / suggestion:1,general:5,suggestion:3
//  unified-top-opposite   10 / suggestion:1,general:5,suggestion:3 / general:1,suggestion:3,general:5
=8  unified-history        10 / general:5,suggestion:4
    unified-search         10 / suggestion:4,general:5
//  unified-same           10 / general:5,suggestion:4 / suggestion:4,general:5
//  unified-mosthistory    10 / general:9
//  unified-mostsearch     10 / suggestion:9
    control                10 (search bar still present)


Branches marked with "//" can be excluded. For the "mostsearch" and "mosthistory" branches I propose to keep the "minimal" variation of them, given the similarity of the "minimal-mosthistory" branch with what we're doing in the pref-flipping study.

Note that the "unified-history" branch is a control branch that should match what we've already tested in the previous study. It makes sense to me to keep "unified-search" as it is symmetric to both "unified-history" and "minimal-search".
Attached file Signed study add-on version 3.0.4b (obsolete) — 
Assignee: shilpi → paolo.mozmail

        Attachment #8897856 -
        Attachment is obsolete: true
Status: NEW → ASSIGNED

    
    

    
  
Version 3.0.4b of the add-on implements the branches from comment 29. To test them, you can install the add-on on Firefox 56 and set the variation preferences in about:config.

extensions.@unified-urlbar-shield-study.shield.phasedVariation
  Ensure this is set to false.

extensions.@unified-urlbar-shield-study.shield.variation
  Set this to the branch name, lowercase and without extra spaces before or after.
  Restart the browser for the changes to take effect.

The valid branch names are:

minimal-top-history
minimal-top-same
minimal-top-opposite
minimal-history
minimal-search
minimal-same
minimal-mosthistory
minimal-mostsearch
unified-top-history
unified-history
unified-search
control

    
    

    
  
On the data side, I ran a quick verification of the data coming back from the test client running 3.0.3, and everything looks as expected.

I created a table of the reported payload values at https://docs.google.com/spreadsheets/d/1C_b8LLF1GIqX_a3HnMNMa9zVfVBa_SAPtr49Nd-onZQ/edit#gid=759867014

    
    

    
  

      
      
      
      

        Attached file
          
        Signed study add-on version 3.0.5b (obsolete)
        — 
      

    


  
This version has the following changes:

1. For the duration of the study, a new entry is added to the "experiments" list in the Telemetry environment.

2. A new "installMode" field is set to "opt-in", "opt-out", or rarely "undetermined" based on how the add-on was installed.

3. Extended Telemetry is not enabled if the add-on is installed in opt-out mode.

Opt-out mode is assumed if the recipeId of any installed add-on-based study contains the string "urlbar" case-insensitively. This includes completed studies.

        Attachment #8898348 -
        Attachment is obsolete: true

    
    

    
  
Paolo, 

Can you please work with Marco to get the latest version of the addon uploaded to the consent page? I think the consent page shows 2.2.1 which is the old one. 

https://addons.mozilla.org/en-US/firefox/shield_study_8

Please confirm once the latest version is uploaded. Shield team can begin enrollment soonafter.

Thanks
Shilpi
Flags: needinfo?(paolo.mozmail)

    
    

    
  
(In reply to shgupta@mozilla.com from comment #34)
> I think the consent page shows 2.2.1 which is the old one.

As I specified already in private e-mail and on IRC, we are blocked on the review from the addons.mozilla.org (AMO) reviewers. We would have been ready to start onboarding last week, as the add-on was ready and uploaded. I think you requested an expedited review and I also independently sent a private e-mail on August 18 to that effect to AMO administrators.

Philipp Kewisch replied with some questions on August 22 but he hasn't since marked the add-on as reviewed. Yesterday I uploaded the updated version 3.0.5 with the changes listed in comment 33. At the time of the upload, the previous version 3.0.4 wasn't reviewed.
Flags: needinfo?(paolo.mozmail) → needinfo?(shilpi)

    
    

    
  
Thanks for the clarification Paolo! 
AMO has reviewed the addon and I can follow up with them to make sure it is marked for you to upload.

Can you point me where to look for add-on marked as reviewed? Is it just r+ on this bug or elsewhere?
Flags: needinfo?(shilpi)

    
    

    
  
Comment on attachment 8900379 [details]
Signed study add-on version 3.0.5b

Review outside of AMO is currently just a matter of sending email, I'm happy to set r+ if you prefer. If you would like us to review any version please do send us an email or request review.

        Attachment #8900379 -
        Flags: review+

    
    

    
  
Sounds great Philipp! Thanks for letting us know.

Paolo, can you please confirm once the latest addon is uploaded on the study consent page here?

https://addons.mozilla.org/en-US/firefox/shield_study_8

Thank you both!
Flags: needinfo?(paolo.mozmail)

    
    

    
  
We actually need a review on the "addons.mozilla.org" (AMO) listing, hopefully we can get that soon.
Flags: needinfo?(paolo.mozmail)

    
    

    
  
The add-on is now reviewed and we should be ready to start onboarding!

    
    

    
  
Hi Paolo,

I am being told that we are now ready to launch the opt out shield study using the addon that will enable us to test the expanded set of branches using all prefs. I will be sending the intent to ship email and obtain relman approval for beta 56 shortly.

Please confirm the latest version of addon that is ready to be deployed for the opt out study. I think it is add-on version 3.0.4b, but want to be sure.

Secondly, will we need a round of QA and/or AMO review or are we good?

Thanks
Shilpi
Flags: needinfo?(paolo.mozmail)

    
    

    
  
Hey Matt, Can you approve the opt out unified search shield study here?

Design Doc: https://docs.google.com/document/d/1AstlOsrwDXQFR2ruLEy4PpeqPiJA-36XtZUZyr3e2bc/edit#heading=h.93rt7oafqxmz
Flags: needinfo?(mgrimes)

    
    

    
  
(In reply to shgupta@mozilla.com from comment #41)
> Please confirm the latest version of addon that is ready to be deployed for
> the opt out study. I think it is add-on version 3.0.4b, but want to be sure.

The important bit of information is that, per comment 33, the recipeId of the opt-out study should contain the string "urlbar" case-insensitively. We can then use this as the add-on installation URL:

https://addons.mozilla.org/downloads/latest/unified-urlbar-shield-study/addon-749812-latest.xpi

We should also make sure that the onboarding for the opt-in study starts now for Release 55 in addition to Beta 56, to prepare for the opt-in studies in Release 56.

> Secondly, will we need a round of QA and/or AMO review or are we good?

We don't need a new review because the add-on is the same, but we should verify that when the add-on is installed using the recipeId specified above then extended telemetry is not enabled.
Flags: needinfo?(paolo.mozmail)

    
    

    
  
Approved!
Flags: needinfo?(mgrimes)

    
    

    
  
As per Paolo's comment 31, here is the final list of branches. I added the pref's behind them. First pref (int) is the maxRichResults number. The second pref is: browser.urlbar.matchBuckets (str - this is the primary bucket to vary composition), and 3 branches have a third pref (str) which is the value of browser.urlbar.matchBucketsSearch

    minimal-top-history     7 / general:1,suggestion:3,general:2
    minimal-top-same        7 / general:1,suggestion:3,general:2 / suggestion:1,general:2,suggestion:3
    minimal-top-opposite    7 / suggestion:1,general:2,suggestion:3 / general:1,suggestion:3,general:2
    minimal-history         7 / general:3,suggestion:3
    minimal-search          7 / suggestion:3,general:3
    minimal-same            7 / general:3,suggestion:3 / suggestion:3,general:3
    minimal-mosthistory     7 / general:6
    minimal-mostsearch      7 / suggestion:6
    unified-top-history    10 / general:1,suggestion:3,general:5
    unified-history        10 / general:5,suggestion:4
    unified-search         10 / suggestion:4,general:5
    control                10 (search bar still present)

    
    

    
  
Thanks Paolo! We are now preparing to launch opt-out study using the addon with 12 branches. Target launch date: Sep 7
I have sent the intent to ship email to relman. Once it is approved we can begin enrollment.

I have a few questions regarding readiness of addon:

1) Who needs to add the recipeID string you mentioned in the addon? Or is that already contained in the link below? 
https://addons.mozilla.org/downloads/latest/unified-urlbar-shield-study/addon-749812-latest.xpi

2) How can we verify that extended telemetry is not enabled upon install of this addon? 

3) Does the opt out addon contain the survey URL? (Old URL is fine too and will work) 


Re: opt-in study enrollment, What is the benefit of on-boarding opt-in users in release 55 since the prefs we are using are not available in 55. If there is a need we can do it. I want to validate the benefit.
Flags: needinfo?(paolo.mozmail)

    
    

    
  
(In reply to shgupta@mozilla.com from comment #46)
> 1) Who needs to add the recipeID string you mentioned in the addon? Or is
> that already contained in the link below? 
> https://addons.mozilla.org/downloads/latest/unified-urlbar-shield-study/
> addon-749812-latest.xpi

No, I think a recipe is a remotely sent instruction to install an add-on from a given URL. So, the URL and the recipeId are parameters of the recipe. I imagine that the recipe is created by those responsible for the deployment of the experiment.

> 2) How can we verify that extended telemetry is not enabled upon install of
> this addon? 

In "about:telemetry", verify that the state is "Telemetry is enabled and extended telemetry is disabled".

> 3) Does the opt out addon contain the survey URL? (Old URL is fine too and
> will work) 

Yes, the use the same survey URL. The can be distinguished using the clientId.

> Re: opt-in study enrollment, What is the benefit of on-boarding opt-in users
> in release 55 since the prefs we are using are not available in 55. If there
> is a need we can do it. I want to validate the benefit.

When version 56 is released in less than 3 weeks, we will already have a cohort of Release users who chose to participate in the experiment, and they will get the new experience right away when the browser is updated, saving us quite some time. We can also choose to update the branches that are offered to this group of users at any time.
Flags: needinfo?(paolo.mozmail)

    
    

    
  
Adding 55 (even if they won't get the experience until 56) sounds like creep at this point. We'd need additional relman approvals for that. We will be able to scale up quickly in 56 Release even if we wait until launch.

    
    

    
  
Hi Paolo,

Shield team has asked for the unsigned version of the addon to ship the opt-out study. They can't use the add-on on AMO as the installation URL, opt-out studies need to be hosted on the Shield service. Once you can provide the unsigned add-on Shield team will sign and upload it.

Thanks
Shilpi
Flags: needinfo?(paolo.mozmail)

    
    

    
  
1) The recipeID field seems to be a numeric ID. Panos is hardcoding the urlbar string in it. 
2) Addon ID will need to be changed from the opt in addon ID so that can differentiate the survey answers coming in from optin vs opt out.

Once these changes are in and tested, we should be able to launch the study! Thanks all for the prompt work :)

    
    

    
  


  
I removed the signature and the check for recipeId (turns out it is a number), so we always take the opt-out branch. I also updated the add-on ID (@unified-urlbar-shield-study-opt-out) to distinguish it from the opt-in study.

        Attachment #8900379 -
        Attachment is obsolete: true

    
    

    
  
Thank you Panos! Matt, Please confirm once we are live as pilot opt-out! :)

    
    

    
  
I replied on the email thread to give relman's OK to ship this in beta 56 until Sept. 26. Thanks!

    
    

    
  
Study is now live on Beta for all locales. We'll end the study on 9/26.

    
    

    
  
Thanks!
Flags: needinfo?(paolo.mozmail)
Depends on: 1400335

    
    

    
  
Is it possible to disable the experiment when keyword.enabled is set to false ? Not having the search bar in this case is a bit sad :)

    
    

    
  
It is too late to make updates for this particular study, but we can take this into consideration for future ones.

    
    

    
  
We plan to re-ship the unified search shield study (v4) to Rel 56 to validate our findings from beta (v3) and resolving any differences.

The last study, Unified v3 shield study ran for 25% of users in beta and ended on 9/26. This new study, Unified v4 shield will run for about 2% of release users in 56 (all locales).

It is the same experiment that is testing for result composition of the awesome bar.

Please let Javaun or I know if you have any questions.

    
    

    
  
Sounds like this is ready to ship to 56 release. Please go ahead. 
What is the end date?

          status-firefox56:
          --- → affected

          tracking-firefox56:
          --- → +

    
    

    
  


  
This version is updated so that the study runs for 30 days, and for 25% of the users the changes are applied after 7 days. There are no changes related to the method used to hide the search bar, as this was not deemed an important change to make in this version.

        Attachment #8905617 -
        Attachment is obsolete: true

    
    

    
  


  
This is the same version but with the ID changed, which I understand is a requirement of the deployment method that will be used.

    
    

    
  


  
I've signed the add-on from comment 61 and uploaded it to Shield under the name "Unified Search Bar Study 3.0.6 (New Users)". I've also attached the signed add-on to this bug.

    
    

    
  


  
I've signed the add-on from comment 60 and uploaded it to Shield under the name "Unified Search Bar Study 3.0.6 (Old Users)". I've also attached the signed add-on to this bug.

    
    

    
  
Both of these have been pushed live on the 56 Release channel

    
    

    
  
We're getting a number of frustrated posters on SuMo. They try to restore their search bar in Customize, but when they exit Customize, the search bar will not appear. Is this the intended design?

For example:

https://support.mozilla.org/questions/1179570
https://support.mozilla.org/questions/1179590
https://support.mozilla.org/questions/1179601
https://support.mozilla.org/questions/1179611
https://support.mozilla.org/questions/1179612
https://support.mozilla.org/questions/1179613

    
    

    
  
Yes, this design was originally introduced in the second version of the opt-in experiment to work around some uncertainty in the data that was collected by the first version. The uncertainty was related to the search bar being reported as still being present when in fact it was absent. In bug 1336982, this turned out to be purely a data issue that would not invalidate the results. However, when the deployment method was changed to opt-out and the experiment deployed to Beta, this design was retained.

As I noted in comment 60, at the time of deciding whether the original method should have been restored when deploying the experiment to Release, it wasn't deemed important enough to warrant development and testing time, with the associated delays these would have brought to the deployment and the possible uncertainty in the data collection.

    
    

    
  
Paolo, is there some workaround for these users reporting the issue in SUMO?  Prefs for them to set, or uninstalling the experiment? 

Will this be different in 57?
Flags: needinfo?(paolo.mozmail)

    
    

    
  
Is the idea here that, we aren't going to fix this for 56, but will put the option back in 57 for users to either keep or do away with the search bar ?

    
    

    
  
(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #67)
> Paolo, is there some workaround for these users reporting the issue in SUMO?
> Prefs for them to set, or uninstalling the experiment? 

Yes, all the changes will be reverted automatically after uninstalling the experiment. This can be done either by uninstalling the add-on manually from the Extensions tab of the Add-ons Manager, or by disabling the "Allow Firefox to install and run studies" checkbox in the Privacy & Security tab of the Options window. The search bar cannot be restored using Customize mode, which would normally work when the experiment is not installed.

(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #68)
> Is the idea here that, we aren't going to fix this for 56, but will put the
> option back in 57 for users to either keep or do away with the search bar ?

The experiment only runs in Firefox 56 for a maximum of 30 days. We don't plan to run this experiment in Firefox 57.

In Firefox 57, the Search tab in the Options window allows to select whether to keep the search bar or not, and of course Customize mode will work as expected.
Flags: needinfo?(paolo.mozmail)

    
    

    
  
OK. Anything else to do here on this bug?
Flags: needinfo?(paolo.mozmail)

    
    

    
  
As far as I know, right now there are no plans to run any other experiment based on this add-on.
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
Flags: needinfo?(paolo.mozmail)
Resolution: --- → FIXED

    
  

          status-firefox56:
          affected → ---

          tracking-firefox56:
          + → ---

          You need to log in
          before you can comment on or make changes to this bug.
        






  

    
  







  

    

      
Attachment

      

      
      
      
      
    

    

      

        

          

            
General

            

              Creator: 



            

            
Created: 

            
Updated: 

            
Size: