Closed
Bug 1387245
Opened 7 years ago
Closed 7 years ago
[Shield] Unified Search Bar Study: Add-on version 3.0
Categories
(Shield :: Shield Study, defect)
Shield
Shield Study
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: shilpi, Assigned: Paolo)
References
Details
Attachments
(4 files, 5 obsolete files)
1. Testing result composition: We don't yet know the optimal result composition in a unified Awesome Bar. We're inclined to believe it's not what we have now (10 results, history first). We need ASAP to test that in Shield. Our ENG plan all along was to land pref'able options prior to 57 and that we could use off-the-trains tech (i.e. Shield) to experiment with composition and even work right up to November to deploy the final best option via Shield, if that's needed. Panos team landed new pref-able options in 56 to allow us to put search suggestions on top (right now they're only bottom) and also to allow us to vary the number of Awesome Bar results (such as 5 total results) where a user doesn't just see 5 history results, but sees at least a few search too. Branches for Testing Result Composition and Unified vs. 2 Bar preferences: Control: Normal Fx (two separate bars, as they exist now) Unified Bar with 10 results, history on top (currently default under unification) Unified Bar with 10 results, search on top Unified Bar with 5-6 results, history on top Unified Bar with 5-6 results, search on top Unified Bar with 10 results, history on top (currently default under unification) + top hit on top Unified Bar with 10 results, search on top + top hit on top Unified Bar with 5-6 results, history on top + top hit on top Unified Bar with 5-6 results, search on top + top hit on top 2. Testing to understand who gets unified vs. keeps search bar.
Reporter | ||
Comment 1•7 years ago
|
||
Current plan and next steps: 1. Experiment design document is WIP - Javaun ETA: Aug 7 2. Coding add on for 8+ branches with 3 prefs - Paolo ETA: Aug 9 3. Generate survey URL: Matt G 4. File shield + legal bug: Shilpi & Javaun 5. Write up survey questions and translate for top tier markets: Javaun 6. QA on Add on - Softvision/ PI (Shilpi to figure QA resource) 7. Intent to ship email - Shilpi 8. Launch shield in 56 beta: tentative ETA Aug 15 9. Collect and Analyse Data 10. Relaunch version 2 in 56 Release
Reporter | ||
Comment 2•7 years ago
|
||
Latest status: * Add on development in progress. ETA end of week. * PI request for QA filed. * Experiment design here: https://docs.google.com/document/d/1AstlOsrwDXQFR2ruLEy4PpeqPiJA-36XtZUZyr3e2bc/edit * Legal bug for the study approved
Reporter | ||
Comment 3•7 years ago
|
||
Matt, Can you please update on the status of opt out uplifting to beta? Is it still on track for this week? Secondly, can you send us a blank survey URL? Thanks Shilpi
Flags: needinfo?(mgrimes)
Comment 4•7 years ago
|
||
+ The Normandy engineering team for status on landing the action
Flags: needinfo?(mgrimes) → needinfo?(mcooper)
Assignee | ||
Comment 5•7 years ago
|
||
To have search suggestions always present at the top: "browser.search.widget.inNavBar": false "browser.urlbar.maxRichResults": 10 "browser.urlbar.matchBuckets": "suggestion:5,general:4" To have history always present at the top and less results: "browser.search.widget.inNavBar": false "browser.urlbar.maxRichResults": 6 "browser.urlbar.matchBuckets": "general:3,suggestion:2" To get a second top result that is the opposite type of the first: "browser.search.widget.inNavBar": false "browser.urlbar.maxRichResults": 10 "browser.urlbar.matchBuckets": "suggestion:1,general:4,suggestion:4" "browser.urlbar.matchBucketsSearch": "general:1,suggestion:4,general:4" This requires Firefox 56 and should work in Firefox 57 Nightly as well.
Comment 6•7 years ago
|
||
> Can you please update on the status of opt out uplifting to beta? Is it still on track for this week?
We have never planned to uplift to Beta this week. We are planning to land on Nightly and begin QA this week. We need to complete QA before we can uplift to Beta. Our current target is the end of August for having opt-out studies on Beta.
Flags: needinfo?(mcooper)
Reporter | ||
Comment 7•7 years ago
|
||
Matt and Mike, We were told that opt out was being uplifted to beta this week. From c6 above, looks like it wasn't planned. There seems to be some disconnect on this. Can someone throw some light? Thanks Shilpi
Flags: needinfo?(mgrimes)
Assignee | ||
Comment 8•7 years ago
|
||
We have discussed the creation of a rolling study that uses the same deployment method and data collection infrastructure as the previous version from bug 1359118. This will be in parallel to the investigation of other study venues. I'm creating a new version of the study add-on with the following changes: 1. The add-on can be installed in Firefox 55 and Firefox 56. The study period will not start before the profile is updated to Firefox 56. For 25% of the participants there will still be a guaranteed delay of 15 days before the study starts, but will be absorbed if the experiment is installed in Firefox 55 first. 2. The study will now run for 60 days from the date of first installation. The study can also be deployed as an update to a previous version, in which case the date of first installation is the original one. Data from these updated profiles can be used or discarded by looking at the first-run revision that is already sent in the telemetry ping. 3. Since one-off search buttons and search suggestions are present by default in Firefox 56, they are not explicitly enabled, and there is no special code to test variations of the onboarding tip anymore. 4. The study branches now test the "result composition" by setting the new preferences available in Firefox 56. The code to restore the original preferences when the study has ended was updated to handle these special hidden preferences correctly. 5. All the existing diagnostic measurements are kept, with the addition of the result composition preferences. 6. It is now possible to test a new study branch by changing the "variation" preference in about:config and restarting.
Assignee | ||
Comment 9•7 years ago
|
||
We may still have to edit the study branches to reflect what we need more precisely, and we still have to update the survey URL, but we can start some preliminary testing on this version for the changes mentioned above.
Assignee | ||
Comment 10•7 years ago
|
||
TEST PLAN - INSTALLATION ------------------------ 1. Create a new profile in Firefox 55, and start the browser with that profile. 2. For convenience, pin the tabs for "about:addons", "about:telemetry", and "about:config", unchecking the checkbox on the latter. 3. Download the add-on attached to this bug and install it, for example by dragging the XPI file to the "about:addons" tab. 4. Verify that the installed version in "about:addons" matches the latest attachment, at the time of this comment it is "3.0.0b". 5. Go to "about:config" and search for "study.shield". Keep this filter, because you'll have to look at these preferences often. 6. Verify that the preferences for "@unified-urlbar-shield-study" are present. The rest of the steps will refer to these preferences using only the last part of their name. 7. Restart the browser. The add-on should still be installed and the preferences present.
Assignee | ||
Comment 11•7 years ago
|
||
TEST PLAN - DATA COLLECTION --------------------------- 1. Go to "Options", "Advanced", "Data Choices". Verify that "Share additional data (i.e., Telemetry)" is already checked. 2. Go to "about:temeletry", select "Archived ping data", and verify that some of the entries in the "Ping" dropdown have the type "shield-study". 3. Select one of the "shield-study" pings, expand the "General Data" section below, and look for the "clientId" value at the end. 4. Paste the client ID in this bug, so we can use it to verify the pings have been received on the server side.
Assignee | ||
Comment 12•7 years ago
|
||
TEST PLAN - DAILY DATA COLLECTION --------------------------------- 1. Ensure you are on the same profile as all the other tests, so the client ID is the same. 2. Perform the following actions a variable number of times: - Start a search by typing in the address bar and pressing ENTER. - Navigate to a search engine by typing "yahoo.com" or "google.com" and pressing ENTER, then start a search from the search engine's home page instead of the Firefox interface. - If the search bar is visible, start a search from it. 3. With the study installed and the browser open, move the computer clock forward by one day. 4. Send the computer to sleep and resume. 5. Go to "about:telemetry", refresh the page, and check that the most recent "shield-study" ping is from the new day. 6. Repeat step 2, but with a different number of actions of each type. Fore example, if you did two searches from the address bar, do only one search now. 7. Restart the browser, this time without moving the computer clock. 8. Repeat steps 3 to 6 two more times without restarting the browser in-between, then close the browser. 9. Now you can restore the computer clock.
Assignee | ||
Comment 13•7 years ago
|
||
TEST PLAN - ACTIVATION ON BROWSER UPDATE ---------------------------------------- 1. In Firefox 55, ensure that: - The "variation" preference does *not* have the value "control" - The "phasedVariation" preference has the value "false" If this is not the case, uninstall the add-on from "about:addons" and reinstall it until the values become the expected ones. Uninstalling the add-on displays a survey, but you can close the tab as you don't have to complete the survey. 2. Verify that the search bar is still visible in the toolbar. 3. Close Firefox 55, then start Firefox 56 with the same profile where the experiment was installed. 4. Verify that the search bar is not visible in the toolbar anymore.
Assignee | ||
Comment 14•7 years ago
|
||
TEST PLAN - END OF STUDY ------------------------ 1. In Firefox 56, check that the preference "browser.urlbar.matchBuckets" exists and has been assigned a custom value. 2. Move the computer clock forward by 50 days and restart the browser. 3. Verify that the preference is still present and the search bar is not visible in the toolbar. 4. Move the computer clock forward by 20 more days, for a total of 70 days, and restart the browser. 5. The end of study survey should be displayed, and the URL should include "reason=end-of-study". 6. Check that the preference "browser.urlbar.matchBuckets" does not exist anymore. 7. Verify that the search bar has been restored in the toolbar. 8. In "about:telemetry", verify that the state is "Telemetry is enabled and extended telemetry is disabled". 9. Now you can restore the computer clock.
Comment 15•7 years ago
|
||
I've started to test here but I ran into some issues with Ubuntu (suspend mode) and Windows 10 (changing date), I'll have to try on Mac once we are back at the office. I've tested until step 3 from comment 12 where I got into problems, but so far everything looks good. Will be back Wednesday (since we have a holiday tomorrow and we will not be at the office) and conclude the testing.
Comment 16•7 years ago
|
||
Also here are the clientIds I ran into during testing: "clientId": "bf7b1a4c-889e-4c56-b643-7cd71574683d" "clientId": "1d14474a-e08d-4544-a2e2-54b98519e423" "clientId": "1dc500da-9f14-4890-8080-ace8c5c6fc4a"
Assignee | ||
Comment 17•7 years ago
|
||
After the study add-on is installed in Firefox 56, the branches can be tested by changing this preference in about:config and restarting: extensions.@unified-urlbar-shield-study.shield.variation In version 3.0.0b of the add-on, there are 10 branches defined, plus the control branch. I'm providing a summary of the current settings without describing the implementation details at this time, because we're still defining the exact behavior we want to test. The branch names are: control 10 results: search bar still present unified-topfirst 10 results: 1 heuristic, 1 "top" opposite, 4 same, 4 opposite unified-searchfirst 9 results: 1 heuristic, 4 suggestions, 4 history unified-historyfirst 9 results: 1 heuristic, 4 history, 4 suggestions unified-mostsearch 9 results: 1 heuristic, 8 suggestions unified-mosthistory 9 results: 1 heuristic, 8 history minimal-topfirst 8 results: 1 heuristic, 1 "top" opposite, 3 same, 3 opposite minimal-searchfirst 7 results: 1 heuristic, 3 suggestions, 3 history minimal-historyfirst 7 results: 1 heuristic, 3 history, 3 suggestions minimal-mostsearch 7 results: 1 heuristic, 6 suggestions minimal-mosthistory 7 results: 1 heuristic, 6 history The "mostsearch" and "mosthistory" are control branches that will likely not be implemented, but are interesting for getting comparative data. We may want to remove some of them from the mix but still keep others.
Assignee | ||
Comment 18•7 years ago
|
||
Since we can push updates to the add-on, we don't necessarily need the final survey URL to start onboarding new participants. We can also adjust study branches, push in-flight surveys before the end of the study, or choose to terminate the experiment earlier or later based on arbitrary criteria.
Flags: needinfo?(mgrimes)
Comment 19•7 years ago
|
||
Link sent via email. Sorry for the delay.
Assignee | ||
Comment 20•7 years ago
|
||
Javaun, you mentioned you were working on the definition of the study branches. Any progress there, or any updates for the branches in comment 17?
Flags: needinfo?(jmoradi)
Comment 21•7 years ago
|
||
I finished testing version 3.0.0b of the add-on across platforms (Windows 10 64bit, macOS 10.12.6 and Ubuntu 16.04 64bit), and the only issue I found is that after the ending of the study the pref "browser.urlbar.matchBuckets" is still visible in about:config and has the same value as before (comment 14 step 6).
Assignee | ||
Comment 22•7 years ago
|
||
This fixes the bug from comment 21, loads on Firefox 56, and updates the branches.
Attachment #8896975 -
Attachment is obsolete: true
Assignee | ||
Comment 23•7 years ago
|
||
Bogdan, I've also found a new bug related to direct installation instead of upgrade. Can you try the steps from comment 10 on a new profile directly in Firefox 56 and see if they work? I'll post some new tests for the branches that are currently in the add-on.
Flags: needinfo?(bogdan.maris)
Assignee | ||
Comment 24•7 years ago
|
||
TEST PLAN - STUDY BRANCHES -------------------------- 1. Ensure that the installed version of the experiment is 3.0.2b and it is running on Firefox 56. 2. Ensure that the "phasedVariation" preference is set to "false", changing it if necessary. 3. Change the "variation" preference to one of the values listed below. IMPORTANT: The value should be typed in lowercase and without extra spaces before or after. 4. Restart the browser. 5. Verify that the preferences under "browser.urlbar." are either set to the expected values or absent, depending on the branch. Variation maxRichResults / matchBuckets / matchBucketsSearch control 10 (search bar still present) unified-topfirst 11 / suggestion:1,general:5,suggestion:4 / general:1,suggestion:4,general:5 unified-historyfirst 10 / general:5,suggestion:4 unified-searchfirst 10 / suggestion:4,general:5 unified-samefirst 10 / general:5,suggestion:4 / suggestion:4,general:5 unified-mosthistory 10 / general:9 unified-mostsearch 10 / suggestion:9 minimal-topfirst 8 / suggestion:1,general:3,suggestion:3 / general:1,suggestion:3,general:3 minimal-historyfirst 7 / general:3,suggestion:3 minimal-searchfirst 7 / suggestion:3,general:3 minimal-samefirst 7 / general:3,suggestion:3 / suggestion:3,general:3 minimal-mosthistory 7 / general:6 minimal-mostsearch 7 / suggestion:6
Comment 25•7 years ago
|
||
I verified the steps from comment 24, I only tried 2 variations (unified-historyfirst and minimal-mosthistory) on Ubuntu (due to lack of time) and they worked as expected (meaning the browser.urlbar preferences had the expected values), I will try a few more tomorrow, across all platforms and also the request from comment 23. It's good to point out that I used the new version of the add-on 3.0.2b.
Comment 26•7 years ago
|
||
I added some (the total branches swelled to 13) and then I proposed cuts. I kept all the smaller result sets (8 or less) and cut the bigger ones. In addition to the dynamic "top opposite" you have, I created a "top same". We also have a dynamic smallset, where the first bucket of results is always the same as the autocomplete. Here's the branches broken down by prefs. I need to QA this. https://docs.google.com/document/d/1AstlOsrwDXQFR2ruLEy4PpeqPiJA-36XtZUZyr3e2bc/edit#heading=h.93rt7oafqxmz ================ control 10 results: search bar still present minimal-historyfirst 7 results: 1 heuristic, 3 history, 3 suggestions minimal-searchfirst 7 results: 1 heuristic, 3 suggestions, 3 history fixed-tophit 8 results: 1 heuristic, 1 history, 3 suggestions, 3 history dynamic-smallset 7 results: 1 heuristic: 3 same, 3 opposite dynamic-top-opposite 8 results: 1 heuristic, 1 "top" opposite, 3 same, 3 opposite dynamic-top-same 8 results: 1 heuristic, 1 "top" same, 3 opposite, 3 same //------------------------------------------------ //Consider DROPPING... unified-historyfirst 9 results: 1 heuristic, 4 history, 4 suggestions unified-searchfirst 9 results: 1 heuristic, 4 suggestions, 4 history minimal-mostsearch 7 results: 1 heuristic, 6 suggestions minimal-mosthistory 7 results: 1 heuristic, 6 history dynamic-bigset 10 results: 1 heuristic: 4 same, 5 opposite fixed-tophit 10 results: 1 heuristic, 1 history, 4 suggestions, 4 history
Flags: needinfo?(jmoradi)
Comment 27•7 years ago
|
||
(In reply to Bogdan Maris, QA [:bogdan_maris] from comment #25) > I verified the steps from comment 24, I only tried 2 variations > (unified-historyfirst and minimal-mosthistory) on Ubuntu (due to lack of > time) and they worked as expected (meaning the browser.urlbar preferences > had the expected values), I will try a few more tomorrow, across all > platforms and also the request from comment 23. It's good to point out that > I used the new version of the add-on 3.0.2b. Just finished testing using add-on version 3.0.2b using steps from comment 24 and request from comment 23 across platforms. Everything worked as expected. Here are the clientIds used for today testing: macOS: "clientId": "65f4f536-3f97-7548-b417-43f7a5538841" Windows 10: "clientId": "9f322061-7fe4-4456-9403-7dd1db33eec9" Linuxx 64: "clientId": "334105a3-ef0c-44d0-bf4d-11e36cc86ec8" I also verified across platforms using 3.0.2b that "browser.urlbar.matchBuckets" is successfully removed (along with the Search Shield Study add-on from about:addons / Extensions) from about:config once the study has ended (comment 14 step 6).
Flags: needinfo?(bogdan.maris)
Assignee | ||
Comment 28•7 years ago
|
||
Updating title per bug 1390592 comment 3.
Summary: [Shield] Opt-out Study: Unified Search 3 → [Shield] Unified Search Bar Study: Add-on version 3.0
Assignee | ||
Comment 29•7 years ago
|
||
This is a new full matrix that cross-references the branches from the opt-out pref-flipping study in bug 1390584 comment 12. It is close to the branches listed in comment 26 with some small variations to match the pref-flipping study. Variation maxRichResults / matchBuckets / matchBucketsSearch =1 minimal-top-history 7 / general:1,suggestion:3,general:2 minimal-top-same 7 / general:1,suggestion:3,general:2 / suggestion:1,general:2,suggestion:3 minimal-top-opposite 7 / suggestion:1,general:2,suggestion:3 / general:1,suggestion:3,general:2 =2 minimal-history 7 / general:3,suggestion:3 =3 minimal-search 7 / suggestion:3,general:3 minimal-same 7 / general:3,suggestion:3 / suggestion:3,general:3 ~4 minimal-mosthistory 7 / general:6 minimal-mostsearch 7 / suggestion:6 =5 unified-top-history 10 / general:1,suggestion:3,general:5 // unified-top-same 10 / general:1,suggestion:3,general:5 / suggestion:1,general:5,suggestion:3 // unified-top-opposite 10 / suggestion:1,general:5,suggestion:3 / general:1,suggestion:3,general:5 =8 unified-history 10 / general:5,suggestion:4 unified-search 10 / suggestion:4,general:5 // unified-same 10 / general:5,suggestion:4 / suggestion:4,general:5 // unified-mosthistory 10 / general:9 // unified-mostsearch 10 / suggestion:9 control 10 (search bar still present) Branches marked with "//" can be excluded. For the "mostsearch" and "mosthistory" branches I propose to keep the "minimal" variation of them, given the similarity of the "minimal-mosthistory" branch with what we're doing in the pref-flipping study. Note that the "unified-history" branch is a control branch that should match what we've already tested in the previous study. It makes sense to me to keep "unified-search" as it is symmetric to both "unified-history" and "minimal-search".
Assignee | ||
Comment 30•7 years ago
|
||
Assignee: shilpi → paolo.mozmail
Attachment #8897856 -
Attachment is obsolete: true
Status: NEW → ASSIGNED
Assignee | ||
Comment 31•7 years ago
|
||
Version 3.0.4b of the add-on implements the branches from comment 29. To test them, you can install the add-on on Firefox 56 and set the variation preferences in about:config. extensions.@unified-urlbar-shield-study.shield.phasedVariation Ensure this is set to false. extensions.@unified-urlbar-shield-study.shield.variation Set this to the branch name, lowercase and without extra spaces before or after. Restart the browser for the changes to take effect. The valid branch names are: minimal-top-history minimal-top-same minimal-top-opposite minimal-history minimal-search minimal-same minimal-mosthistory minimal-mostsearch unified-top-history unified-history unified-search control
Comment 32•7 years ago
|
||
On the data side, I ran a quick verification of the data coming back from the test client running 3.0.3, and everything looks as expected. I created a table of the reported payload values at https://docs.google.com/spreadsheets/d/1C_b8LLF1GIqX_a3HnMNMa9zVfVBa_SAPtr49Nd-onZQ/edit#gid=759867014
Assignee | ||
Comment 33•7 years ago
|
||
This version has the following changes: 1. For the duration of the study, a new entry is added to the "experiments" list in the Telemetry environment. 2. A new "installMode" field is set to "opt-in", "opt-out", or rarely "undetermined" based on how the add-on was installed. 3. Extended Telemetry is not enabled if the add-on is installed in opt-out mode. Opt-out mode is assumed if the recipeId of any installed add-on-based study contains the string "urlbar" case-insensitively. This includes completed studies.
Attachment #8898348 -
Attachment is obsolete: true
Reporter | ||
Comment 34•7 years ago
|
||
Paolo, Can you please work with Marco to get the latest version of the addon uploaded to the consent page? I think the consent page shows 2.2.1 which is the old one. https://addons.mozilla.org/en-US/firefox/shield_study_8 Please confirm once the latest version is uploaded. Shield team can begin enrollment soonafter. Thanks Shilpi
Flags: needinfo?(paolo.mozmail)
Assignee | ||
Comment 35•7 years ago
|
||
(In reply to shgupta@mozilla.com from comment #34) > I think the consent page shows 2.2.1 which is the old one. As I specified already in private e-mail and on IRC, we are blocked on the review from the addons.mozilla.org (AMO) reviewers. We would have been ready to start onboarding last week, as the add-on was ready and uploaded. I think you requested an expedited review and I also independently sent a private e-mail on August 18 to that effect to AMO administrators. Philipp Kewisch replied with some questions on August 22 but he hasn't since marked the add-on as reviewed. Yesterday I uploaded the updated version 3.0.5 with the changes listed in comment 33. At the time of the upload, the previous version 3.0.4 wasn't reviewed.
Flags: needinfo?(paolo.mozmail) → needinfo?(shilpi)
Reporter | ||
Comment 36•7 years ago
|
||
Thanks for the clarification Paolo! AMO has reviewed the addon and I can follow up with them to make sure it is marked for you to upload. Can you point me where to look for add-on marked as reviewed? Is it just r+ on this bug or elsewhere?
Flags: needinfo?(shilpi)
Comment 37•7 years ago
|
||
Comment on attachment 8900379 [details]
Signed study add-on version 3.0.5b
Review outside of AMO is currently just a matter of sending email, I'm happy to set r+ if you prefer. If you would like us to review any version please do send us an email or request review.
Attachment #8900379 -
Flags: review+
Reporter | ||
Comment 38•7 years ago
|
||
Sounds great Philipp! Thanks for letting us know. Paolo, can you please confirm once the latest addon is uploaded on the study consent page here? https://addons.mozilla.org/en-US/firefox/shield_study_8 Thank you both!
Flags: needinfo?(paolo.mozmail)
Assignee | ||
Comment 39•7 years ago
|
||
We actually need a review on the "addons.mozilla.org" (AMO) listing, hopefully we can get that soon.
Flags: needinfo?(paolo.mozmail)
Assignee | ||
Comment 40•7 years ago
|
||
The add-on is now reviewed and we should be ready to start onboarding!
Reporter | ||
Comment 41•7 years ago
|
||
Hi Paolo, I am being told that we are now ready to launch the opt out shield study using the addon that will enable us to test the expanded set of branches using all prefs. I will be sending the intent to ship email and obtain relman approval for beta 56 shortly. Please confirm the latest version of addon that is ready to be deployed for the opt out study. I think it is add-on version 3.0.4b, but want to be sure. Secondly, will we need a round of QA and/or AMO review or are we good? Thanks Shilpi
Flags: needinfo?(paolo.mozmail)
Reporter | ||
Comment 42•7 years ago
|
||
Hey Matt, Can you approve the opt out unified search shield study here? Design Doc: https://docs.google.com/document/d/1AstlOsrwDXQFR2ruLEy4PpeqPiJA-36XtZUZyr3e2bc/edit#heading=h.93rt7oafqxmz
Flags: needinfo?(mgrimes)
Assignee | ||
Comment 43•7 years ago
|
||
(In reply to shgupta@mozilla.com from comment #41) > Please confirm the latest version of addon that is ready to be deployed for > the opt out study. I think it is add-on version 3.0.4b, but want to be sure. The important bit of information is that, per comment 33, the recipeId of the opt-out study should contain the string "urlbar" case-insensitively. We can then use this as the add-on installation URL: https://addons.mozilla.org/downloads/latest/unified-urlbar-shield-study/addon-749812-latest.xpi We should also make sure that the onboarding for the opt-in study starts now for Release 55 in addition to Beta 56, to prepare for the opt-in studies in Release 56. > Secondly, will we need a round of QA and/or AMO review or are we good? We don't need a new review because the add-on is the same, but we should verify that when the add-on is installed using the recipeId specified above then extended telemetry is not enabled.
Flags: needinfo?(paolo.mozmail)
Comment 45•7 years ago
|
||
As per Paolo's comment 31, here is the final list of branches. I added the pref's behind them. First pref (int) is the maxRichResults number. The second pref is: browser.urlbar.matchBuckets (str - this is the primary bucket to vary composition), and 3 branches have a third pref (str) which is the value of browser.urlbar.matchBucketsSearch minimal-top-history 7 / general:1,suggestion:3,general:2 minimal-top-same 7 / general:1,suggestion:3,general:2 / suggestion:1,general:2,suggestion:3 minimal-top-opposite 7 / suggestion:1,general:2,suggestion:3 / general:1,suggestion:3,general:2 minimal-history 7 / general:3,suggestion:3 minimal-search 7 / suggestion:3,general:3 minimal-same 7 / general:3,suggestion:3 / suggestion:3,general:3 minimal-mosthistory 7 / general:6 minimal-mostsearch 7 / suggestion:6 unified-top-history 10 / general:1,suggestion:3,general:5 unified-history 10 / general:5,suggestion:4 unified-search 10 / suggestion:4,general:5 control 10 (search bar still present)
Reporter | ||
Comment 46•7 years ago
|
||
Thanks Paolo! We are now preparing to launch opt-out study using the addon with 12 branches. Target launch date: Sep 7 I have sent the intent to ship email to relman. Once it is approved we can begin enrollment. I have a few questions regarding readiness of addon: 1) Who needs to add the recipeID string you mentioned in the addon? Or is that already contained in the link below? https://addons.mozilla.org/downloads/latest/unified-urlbar-shield-study/addon-749812-latest.xpi 2) How can we verify that extended telemetry is not enabled upon install of this addon? 3) Does the opt out addon contain the survey URL? (Old URL is fine too and will work) Re: opt-in study enrollment, What is the benefit of on-boarding opt-in users in release 55 since the prefs we are using are not available in 55. If there is a need we can do it. I want to validate the benefit.
Flags: needinfo?(paolo.mozmail)
Assignee | ||
Comment 47•7 years ago
|
||
(In reply to shgupta@mozilla.com from comment #46) > 1) Who needs to add the recipeID string you mentioned in the addon? Or is > that already contained in the link below? > https://addons.mozilla.org/downloads/latest/unified-urlbar-shield-study/ > addon-749812-latest.xpi No, I think a recipe is a remotely sent instruction to install an add-on from a given URL. So, the URL and the recipeId are parameters of the recipe. I imagine that the recipe is created by those responsible for the deployment of the experiment. > 2) How can we verify that extended telemetry is not enabled upon install of > this addon? In "about:telemetry", verify that the state is "Telemetry is enabled and extended telemetry is disabled". > 3) Does the opt out addon contain the survey URL? (Old URL is fine too and > will work) Yes, the use the same survey URL. The can be distinguished using the clientId. > Re: opt-in study enrollment, What is the benefit of on-boarding opt-in users > in release 55 since the prefs we are using are not available in 55. If there > is a need we can do it. I want to validate the benefit. When version 56 is released in less than 3 weeks, we will already have a cohort of Release users who chose to participate in the experiment, and they will get the new experience right away when the browser is updated, saving us quite some time. We can also choose to update the branches that are offered to this group of users at any time.
Flags: needinfo?(paolo.mozmail)
Comment 48•7 years ago
|
||
Adding 55 (even if they won't get the experience until 56) sounds like creep at this point. We'd need additional relman approvals for that. We will be able to scale up quickly in 56 Release even if we wait until launch.
Reporter | ||
Comment 49•7 years ago
|
||
Hi Paolo, Shield team has asked for the unsigned version of the addon to ship the opt-out study. They can't use the add-on on AMO as the installation URL, opt-out studies need to be hosted on the Shield service. Once you can provide the unsigned add-on Shield team will sign and upload it. Thanks Shilpi
Flags: needinfo?(paolo.mozmail)
Reporter | ||
Comment 50•7 years ago
|
||
1) The recipeID field seems to be a numeric ID. Panos is hardcoding the urlbar string in it. 2) Addon ID will need to be changed from the opt in addon ID so that can differentiate the survey answers coming in from optin vs opt out. Once these changes are in and tested, we should be able to launch the study! Thanks all for the prompt work :)
Comment 51•7 years ago
|
||
I removed the signature and the check for recipeId (turns out it is a number), so we always take the opt-out branch. I also updated the add-on ID (@unified-urlbar-shield-study-opt-out) to distinguish it from the opt-in study.
Attachment #8900379 -
Attachment is obsolete: true
Reporter | ||
Comment 52•7 years ago
|
||
Thank you Panos! Matt, Please confirm once we are live as pilot opt-out! :)
Comment 53•7 years ago
|
||
I replied on the email thread to give relman's OK to ship this in beta 56 until Sept. 26. Thanks!
Comment 54•7 years ago
|
||
Study is now live on Beta for all locales. We'll end the study on 9/26.
Comment 56•7 years ago
|
||
Is it possible to disable the experiment when keyword.enabled is set to false ? Not having the search bar in this case is a bit sad :)
Comment 57•7 years ago
|
||
It is too late to make updates for this particular study, but we can take this into consideration for future ones.
Reporter | ||
Comment 58•7 years ago
|
||
We plan to re-ship the unified search shield study (v4) to Rel 56 to validate our findings from beta (v3) and resolving any differences. The last study, Unified v3 shield study ran for 25% of users in beta and ended on 9/26. This new study, Unified v4 shield will run for about 2% of release users in 56 (all locales). It is the same experiment that is testing for result composition of the awesome bar. Please let Javaun or I know if you have any questions.
Comment 59•7 years ago
|
||
Sounds like this is ready to ship to 56 release. Please go ahead. What is the end date?
status-firefox56:
--- → affected
tracking-firefox56:
--- → +
Assignee | ||
Comment 60•7 years ago
|
||
This version is updated so that the study runs for 30 days, and for 25% of the users the changes are applied after 7 days. There are no changes related to the method used to hide the search bar, as this was not deemed an important change to make in this version.
Attachment #8905617 -
Attachment is obsolete: true
Assignee | ||
Comment 61•7 years ago
|
||
This is the same version but with the ID changed, which I understand is a requirement of the deployment method that will be used.
Comment 62•7 years ago
|
||
I've signed the add-on from comment 61 and uploaded it to Shield under the name "Unified Search Bar Study 3.0.6 (New Users)". I've also attached the signed add-on to this bug.
Comment 63•7 years ago
|
||
I've signed the add-on from comment 60 and uploaded it to Shield under the name "Unified Search Bar Study 3.0.6 (Old Users)". I've also attached the signed add-on to this bug.
Comment 64•7 years ago
|
||
Both of these have been pushed live on the 56 Release channel
Comment 65•7 years ago
|
||
We're getting a number of frustrated posters on SuMo. They try to restore their search bar in Customize, but when they exit Customize, the search bar will not appear. Is this the intended design? For example: https://support.mozilla.org/questions/1179570 https://support.mozilla.org/questions/1179590 https://support.mozilla.org/questions/1179601 https://support.mozilla.org/questions/1179611 https://support.mozilla.org/questions/1179612 https://support.mozilla.org/questions/1179613
Assignee | ||
Comment 66•7 years ago
|
||
Yes, this design was originally introduced in the second version of the opt-in experiment to work around some uncertainty in the data that was collected by the first version. The uncertainty was related to the search bar being reported as still being present when in fact it was absent. In bug 1336982, this turned out to be purely a data issue that would not invalidate the results. However, when the deployment method was changed to opt-out and the experiment deployed to Beta, this design was retained. As I noted in comment 60, at the time of deciding whether the original method should have been restored when deploying the experiment to Release, it wasn't deemed important enough to warrant development and testing time, with the associated delays these would have brought to the deployment and the possible uncertainty in the data collection.
Comment 67•7 years ago
|
||
Paolo, is there some workaround for these users reporting the issue in SUMO? Prefs for them to set, or uninstalling the experiment? Will this be different in 57?
Flags: needinfo?(paolo.mozmail)
Comment 68•7 years ago
|
||
Is the idea here that, we aren't going to fix this for 56, but will put the option back in 57 for users to either keep or do away with the search bar ?
Assignee | ||
Comment 69•7 years ago
|
||
(In reply to Liz Henry (:lizzard) (needinfo? me) from comment #67) > Paolo, is there some workaround for these users reporting the issue in SUMO? > Prefs for them to set, or uninstalling the experiment? Yes, all the changes will be reverted automatically after uninstalling the experiment. This can be done either by uninstalling the add-on manually from the Extensions tab of the Add-ons Manager, or by disabling the "Allow Firefox to install and run studies" checkbox in the Privacy & Security tab of the Options window. The search bar cannot be restored using Customize mode, which would normally work when the experiment is not installed. (In reply to Liz Henry (:lizzard) (needinfo? me) from comment #68) > Is the idea here that, we aren't going to fix this for 56, but will put the > option back in 57 for users to either keep or do away with the search bar ? The experiment only runs in Firefox 56 for a maximum of 30 days. We don't plan to run this experiment in Firefox 57. In Firefox 57, the Search tab in the Options window allows to select whether to keep the search bar or not, and of course Customize mode will work as expected.
Flags: needinfo?(paolo.mozmail)
Assignee | ||
Comment 71•7 years ago
|
||
As far as I know, right now there are no plans to run any other experiment based on this add-on.
Status: ASSIGNED → RESOLVED
Closed: 7 years ago
Flags: needinfo?(paolo.mozmail)
Resolution: --- → FIXED
Assignee | ||
Updated•7 years ago
|
status-firefox56:
affected → ---
tracking-firefox56:
+ → ---
You need to log in
before you can comment on or make changes to this bug.
Description
•