1359118 - [Shield] Unified URL Bar v 2

Status: RESOLVED FIXED

(Shield :: Shield Study, enhancement, P1)

Description

[:Paolo Amadini]

This bug tracks the implementation of Round 2 of the "Unified Search Bar" study, also known as "Unified URL Bar".

Comment 1

[:Paolo Amadini]

The new version of the study add-on in the works has the following improvements:

1. The code is updated for compatibility with Firefox 53. The one-off search buttons are now enabled by flipping a preference, and the reimplementation of the feature in the add-on code has been removed.

2. We now hide the search bar using CSS. This means that its presence cannot be restored by the user without disabling the experiment.

3. There is a new diagnostic measurement to verify that the search bar is not visible. This reads the state from the most recent browser window at the time of the Shield telemetry ping.

4. There is only a 50% chance that a user will see the yellow tip about search suggestions when enrolling in the experiment.

5. There is a 25% chance that the experiment will do nothing for the first week, allowing a two-phase study. This is a trade-off between getting statistically significant data earlier and allowing a two-phase study to occur.

6. The previous version didn't properly restore the extended telemetry setting, and this is now fixed.

Comment 2

[:Paolo Amadini]

7. The experiment is designed to run for the rest of the Firefox 53 release, which will be about a month or so from the time of deployment, so the installation time limit is updated to 45 days.

Comment 3

[:Paolo Amadini]

5. Amended: the two-phase study now runs without changes for two weeks.

8. We now mirror relevant telemetry objects from both classic telemetry pings and subsession pings.

9. The previous version didn't send the study name in the ping on recent browser versions, and this is now fixed.

Comment 4

[:Paolo Amadini]

Attachment: Unsigned study add-on version 2.2.0

Comment 5

[:Paolo Amadini]

TEST PLAN - INSTALLATION
------------------------

1. Download and install the unbranded Release build of Firefox 53.0. For example, this is the link of the Windows build:

http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-release-win32-add-on-devel/1492596980/firefox-53.0.en-US.win32-add-on-devel.installer.exe

2. Follow the usual steps to create a new profile, and start the browser with that profile.

3. For convenience, pin the tabs for "about:addons", "about:telemetry", and "about:config", unchecking the checkbox on the latter.

4. Flip the "xpinstall.signatures.required" preference.

5. Download the unsigned add-on attached to this bug and install it, for example by dragging the XPI file to the "about:addons" tab.

6. Verify that the installed version in "about:addons" matches the latest attachment, at the time of this comment it is "2.2.0".

7. Go to "about:config" and search for "study.shield". Keep this filter, because you'll have to look at these preferences often.

8. Verify that the preferences for "@unified-urlbar-shield-study" are present. The rest of the steps will refer to these preferences using only the last part of their name.

9. Restart the browser. The add-on should still be installed and the preferences present.



TEST PLAN - DATA COLLECTION
---------------------------

1. Go to "Options", "Advanced", "Data Choices". Verify that "Share additional data (i.e., Telemetry)" is already checked. If it is not already checked, it's a bug.

2. Go to "about:temeletry", select "Archived ping data", and verify that some of the entries in the "Ping" dropdown have the type "shield-study".

3. Select one of the "shield-study" pings, expand the "General Data" section below, and look for the "clientId" value at the end.

4. Paste the client ID in this bug, so David can use it to verify the pings have been received on the server side.



TEST PLAN - USER UNINSTALLATION
-------------------------------

1. Disable the experiment from "about:addons". This should display a survey, but you can close the tab as you don't have to complete the survey.

2. Verify that disabling the experiment has also uninstalled the add-on completely.

3. Reinstall the add-on. Repeat steps 1 to 3 until both of the following preferences have these values:

variation: unified
phasedVariation: false

4. Verify that the search bar has been removed from the toolbar. 

5. Uninstall the add-on, and verify that the search bar has been restored.

6. Go to "Options", "Advanced", "Data Choices". Verify that "Share additional data (i.e., Telemetry)" is not checked. If it is still checked after the uninstallation, it's a bug.

Comment 6

[:Paolo Amadini]

Hi Brindusa, can you or someone from QA run the steps in comment 5, and others I'll add below, as soon as reasonably possible?

We'd like to get the study out there as soon as possible, and QA is a major part of the remaining steps.

There will be a new version to test later this week, but it will be much faster to re-test on the same profile because we won't need to repeat all of the steps again, we can test only the few things that changed.

Comment 7

[:Paolo Amadini]

TEST PLAN - DAILY DATA COLLECTION
---------------------------------

1. Ensure you are on the same profile as all the other tests, so the client ID is the same.

2. Perform the following actions a variable number of times:

- Start a search by typing in the address bar and pressing ENTER.
- Navigate to a search engine by typing "yahoo.com" or "google.com" and pressing ENTER, then start a search from the search engine's home page instead of the Firefox interface.
- If the search bar is visible, start a search from it.

3. With the study installed and the browser open, move the computer clock forward by one day.

4. Send the computer to sleep and resume.

5. Go to "about:telemetry", refresh the page, and check that the most recent "shield-study" ping is from the new day.

6. Repeat step 2, but with a different number of actions of each type. Fore example, if you did two searches from the address bar, do only one search now.

7. Restart the browser, this time without moving the computer clock.

8. Repeat steps 3 to 6 two more times without restarting the browser in-between, then close the browser.

9. Now you can restore the computer clock.



TEST PLAN - END OF STUDY AND TWO PHASE STUDY
--------------------------------------------

1. Uninstall and reinstall the study until both of the following preferences have these values:

variation: unified
phasedVariation: true

This happens for 1 out of 12 installations, so you may have to repeat a few times, but for proper validation these preferences should never be switched manually.

2. Verify that the search bar is still present in the toolbar. 

3. Move the computer clock forward by 15 days, that is two weeks and one day, and restart the browser.

4. Verify that the search bar is not in the toolbar anymore.

5. Move the computer clock forward by another month, that is more than 45 days from the original, and restart the browser.

6. The end of study survey should be displayed, and the URL should include "reason=end-of-study".

7. Verify that the search bar has been restored in the toolbar.

8. Now you can restore the computer clock.

Comment 8

[:Paolo Amadini]

TEST PLAN - STUDY BRANCHES
--------------------------

1. To test study branches, you have to uninstall and reinstall the study multiple times, but never change the preferences manually.

2. When "phasedVariation" is "true" or "variation" is "control":

- None of the changes below should have taken effect. For all the other tests, ensure that "phasedVariation" is "false" and "variation" is not "control".
- Start typing in the address bar. You should not see any search suggestions or "Search for <...> with:" buttons.
- The search bar should be present in the toolbar.

3. When "variation" is "oneoff":

- Start typing in the address bar. You should see search suggestions that update while you type, and also "Search for <...> with:" buttons at the bottom.
- The search bar should be present in the toolbar.

4. When "variation" is "control":

- Start typing in the address bar. You should see search suggestions that update while you type, and also "Search for <...> with:" buttons at the bottom.
- The search bar should have been removed from the toolbar.

5. When "onboardingVariation" is "true":

- Start typing in the address bar. You should see a yellow animated "Tip" message for the first few times.
- When the "tipShownCount" preference reaches 0, the tip is not displayed anymore.

6. When "onboardingVariation" is "false":

- Verify that the "Tip" message is not displayed the first time you type in the address bar after the study is installed.

Comment 9

[:Paolo Amadini]

TEST PLAN - SEARCH BAR REMOVAL
------------------------------

1. Again, ensure you are on the same profile as all the other tests, so the client ID is the same.

2. Uninstall and reinstall the study until both of the following preferences have these values:

variation: unified
phasedVariation: false

3. Verify that the search bar is not visible in the toolbar.

4. Enter "Customize" mode. The search bar should now be visible again.

5. Drag and drop the search bar to the customization palette, effectively removing it from the toolbar.

6. Exit customize mode and restart the browser.

7. Verify that the experiment is still installed, then uninstall it.

8. Verify that the search bar is still not visible in the toolbar.

9. Install the experiment again, and verify that it will immediately uninstall itself.

10. Enter "Customize" mode and restore the search bar in the toolbar.

11. Install the experiment again, and verify that this time it will stay installed.

Comment 10

[:Paolo Amadini]

TEST PLAN - VERSION INELIGIBILITY
---------------------------------

1. Try to install the add-on on any version other than Firefox 53, ensuring the add-on signing requirement is still disabled. This should either be blocked because of add-on incompatibility, or the experiment should immediately uninstall itself.

2. Execute any other exploratory testing to try and break the experiment logic. Remember to never change any of the "about:config" preferences of the study manually. This is important so that proper data collection can be verified on the server side.

Comment 11

[:Paolo Amadini]

David, I also ran all the tests above, you should see the Shield pings from clientId 72e8a007-0657-4755-9bfc-5d4186f026cf. All the data from that ID should be self-consistent and conform to the final schema. I had a connectivity issue, but I think all the archived pings are now sent.

Brindusa, for reference writing and executing the tests took about 3 hours, so someone from QA will probably be able to run them in the same time or even less.

Comment 12

[:Paolo Amadini]

A note about add-on updates. In very rare cases, we may find that the old experiment is still installed in some profiles, and is automatically updated to the new version without a browser restart. This creates an issue because we don't unload the JSM modules, and things may break until the next restart, since we're mixing code from two different versions of the add-on.

This is properly detected by the new add-on, because in these cases the "firstrunRevision" preference will have the value "1" instead of "2". We can exclude those profiles because this value is part of the diagnostic data we send in the ping.

Comment 13

[Brindusa Tot, Desktop QA]

Ovidiu, please verify this as Paolo mentioned in Comment 6.

Paolo, is it enough if we verify this only on Windows? As I see you gave us the unbranded Release build for Windows(in comment 5).

Comment 14

[:Paolo Amadini]

(In reply to Brindusa Tot[:brindusat] from comment #13)
> Paolo, is it enough if we verify this only on Windows? As I see you gave us
> the unbranded Release build for Windows(in comment 5).

Yes, definitely, the add-on and the modified code is the same on all platforms, so testing on Windows is sufficient.

Comment 15

[ovidiu boca[:Ovidiu]]

Attachment: Comment 8 Scenario 4.png

Hi Paolo,
I verified the mentioned scenarios under Windows 7 x32.
Below are the results of testing [Shield] Unified URL Bar v2 for each scenario:  

 From Comment 5 
 
 Test Plan - Installation

Everything worked as expected. 

Test Plan - Data Collection 

1. Ok
2. Ok
3. OK
4. Here is the result: clientId  1aab3034-abb6-4470-af72-07e67f2c9774

Test Plan - User Uninstallation

1. Ok
2. OK
3. OK
4.  Ok   The search bar is removed from the toolbar
5.  Ok    Yes, the search bar has been restored 
6.  Ok    The Share additional data (i.e., Telemetry)

From Comment 7

Test Plan - Daily Data Collection

1. Ok
2. Ok
3.Ok
4.Ok
5. The ping is from the new day (2017/04/28)
6. Ok
7. Ok
8. Ok
9. Ok

Test Plan - End of Study and two-phase study

1. After long struggles I did it :)
2. Ok
3. I move it to 12 May
4. Ok (The search bar is not in the toolbar)
5. Ok
6. Ok https://qsurvey.mozilla.com/s3/search-study-1?variation=unified&xname=%40unified-urlbar-shield-study&who=1aab3034-abb6-4470-af72-07e67f2c9774&updateChannel=default&fxVersion=53.0&reason=end-of-study
7. Ok, the search bar is restored 
8. Ok


 From  Comment 8
 
Test Plan - Study Branches 

  3. OK
  4.The search bar from the toolbar is not removed, please see the attached print screen.  
I also did a new profile for this, I followed all the steps from comment 5 INSTRUCTIONS and I have the same result, here is the client ID for the second profile: clientId  47cbd26a-fde7-47d1-b162-62e6de2445b1    .   
       
  5. Ok
  6. The "Tip" is not displayed the first time you type in the address bar after the study is installed, this is good, this is the expected results. I also observed that the "Tip" is not displayed second or third time. Is this the intended behavior? 

 From  Comment 9
 
Test Plan - Search Bar Removal 

1. Ok
2 .Ok
3. Ok
4. Ok
5. Ok
6. Ok
7. Ok
8. Ok
9. Ok
10. Ok
11. Ok

 From  Comment 10
 
 Test Plan - Version Ineligibility 

1. Ok
2. Ok   - Here I think it will be a good idea to verify this add-on with other OSes - please tell me your opinion about this and if the answer is yes please provide the links for the testing builds.

Comment 16

[:Paolo Amadini]

(In reply to ovidiu boca[:Ovidiu] from comment #15)
>   4.The search bar from the toolbar is not removed, please see the attached
> print screen.

This was a typo in my comment, sorry! This should read:

4. When "variation" is "unified"

>   6. The "Tip" is not displayed the first time you type in the address bar
> after the study is installed, this is good, this is the expected results. I
> also observed that the "Tip" is not displayed second or third time. Is this
> the intended behavior? 

Yes, thanks for checking!

> 2. Ok   - Here I think it will be a good idea to verify this add-on with
> other OSes - please tell me your opinion about this and if the answer is yes
> please provide the links for the testing builds.

Sounds good, you can do some basic testing on Mac and Linux using these builds:

http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-release-macosx64-add-on-devel/1492596980/firefox-53.0.en-US.mac-add-on-devel.dmg
http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-release-linux64-add-on-devel/1492596980/firefox-53.0.en-US.linux-x86_64-add-on-devel.tar.bz2

Comment 17

[:Paolo Amadini]

Attachment: Unsigned study add-on version 2.2.1

Comment 18

[:Paolo Amadini]

(In reply to :Paolo Amadini from comment #16)
> 4. When "variation" is "unified"

Actually, I'll post an updated test plan for the study branches of version 2.2.1 tomorrow, so you don't need to repeat this test on version 2.2.0.

Comment 19

[:Paolo Amadini]

TEST PLAN - UPGRADE AND RESTORE
-------------------------------

1. Install the old add-on version 2.2.0, until "phasedVariation" is "false" and the branch is not "control". Don't change the study preferences manually.

2. Install the old add-on version 2.2.0 again upon itself. You should see this error in the Browser Console:

"UnifiedUrlbar.init() was invoked multiple times?"

3. Restart the browser

4. Install the new add-on version 2.2.1 over the already installed version 2.2.0, and check that the previous error message does not appear in the Browser Console.

5. Uninstall the add-on, then check the value of these preferences:

browser.urlbar.oneOffSearches
browser.urlbar.suggest.searches

They should both be "true", but this is because of a bug in previous versions.

6. Reset the two preferences above and restart the browser.

7. Install the new add-on version 2.2.1, until "phasedVariation" is "false" and the branch is not "control".

8. Check that the two preferences above are set to "true".

9. Restart the browser, then uninstall the add-on.

10. Check that the two preferences above are set to "false".



TEST PLAN - STUDY BRANCHES
--------------------------

1. To test the number of lines shown when typing in the address bar, you have to visit more than 10 pages whose address or title match the search terms you will type during the verification process. With the study uninstalled, visit at least 10 pages that will match your search, and ensure you see all of them when you start typing in the address bar again.

2. To test study branches, you have to uninstall and reinstall the study multiple times, but never change the preferences manually.

3. When "phasedVariation" is "true" or "variation" is "control":

- None of the changes below should have taken effect. For all the other tests, ensure that "phasedVariation" is "false" and "variation" is not "control".
- Start typing in the address bar. You should see at most 10 lines, and you should not see any search suggestions or "Search for <...> with:" buttons.
- The search bar should be present in the toolbar.

4. When "variation" is "oneoff":

- Start typing in the address bar. You should see search suggestions that update while you type, and also "Search for <...> with:" buttons at the bottom.
- You should see at most 10 lines. It's not important whether they are search suggestions or sites from local history.
- The search bar should be present in the toolbar.

5. When "variation" is "unified":

- Start typing in the address bar. You should see search suggestions that update while you type, and also "Search for <...> with:" buttons at the bottom.
- You should see at most 10 lines. It's not important whether they are search suggestions or sites from local history.
- The search bar should have been removed from the toolbar.

6. When "variation" is "minimal":

- Start typing in the address bar. You should see search suggestions that update while you type, and also "Search for <...> with:" buttons at the bottom.
- You should see at most 6 lines. It's not important whether they are search suggestions or sites from local history.
- The search bar should have been removed from the toolbar.

7. When "onboardingVariation" is "true":

- Start typing in the address bar. You should see a yellow animated "Tip" message for the first few times.
- When the "tipShownCount" preference reaches 0, the tip is not displayed anymore.

8. When "onboardingVariation" is "false":

- Verify that the "Tip" message is not displayed the first time you type in the address bar after the study is installed.

Comment 20

[:Paolo Amadini]

Ovidiu, I've added the latest tests for you in comment 19, and if you want you can run the basic tests on Mac and Linux.

Comment 21

[Dave Zeber [:dzeber]]

(In reply to :Paolo Amadini from comment #11)
> David, I also ran all the tests above, you should see the Shield pings from
> clientId 72e8a007-0657-4755-9bfc-5d4186f026cf. All the data from that ID
> should be self-consistent and conform to the final schema. I had a
> connectivity issue, but I think all the archived pings are now sent.

Yes, I've checked the server-side data received from both this profile and the previous test profile (980cecf9-92b7-43be-bc46-97d312a323ec), and everything looks good.

One thing I noticed is that `changesApplied` appears to always be true on the "control" branch. Not an issue, it's just not what I would have expected. I'm assuming it gets set to true regardless of branch when the study actually starts to take effect.

For reference, [1] is the notebook I used to gather the pings, and [2] is a spreadsheet containing flattened versions of the ping payloads for both test profiles. Each profile has two sheets, one listing the contents ("Ping summary") and one listing whether or not each expected field was present in the payload ("Payload fields").

[1] https://gist.github.com/dzeber/90db5f9730ba5ce13e51e334c02726f9
[2] https://docs.google.com/a/mozilla.com/spreadsheets/d/1y3FSkNV7bjYzr8rVQooC2wKgCQrg1nXOloYYS6Optko/edit?usp=sharing

Comment 22

[:Paolo Amadini]

(In reply to Dave Zeber [:dzeber] from comment #21)
> One thing I noticed is that `changesApplied` appears to always be true on
> the "control" branch. Not an issue, it's just not what I would have
> expected. I'm assuming it gets set to true regardless of branch when the
> study actually starts to take effect.

Yes, it can be either true or false on the "control" branch, like all other branches.

Maybe you never observed "false" because of the small volume of the data. You can get more data from Ovidiu's clientId 47cbd26a-fde7-47d1-b162-62e6de2445b1.

Comment 23

[Dave Zeber [:dzeber]]

(In reply to :Paolo Amadini from comment #22)
> (In reply to Dave Zeber [:dzeber] from comment #21)
> > One thing I noticed is that `changesApplied` appears to always be true on
> > the "control" branch. Not an issue, it's just not what I would have
> > expected. I'm assuming it gets set to true regardless of branch when the
> > study actually starts to take effect.
> 
> Yes, it can be either true or false on the "control" branch, like all other
> branches.

Ah I think I misunderstood this measurement.

If I now understand correctly, it's "true" when the study period has begun, which is from installation in the 1-phase case, or at the beginning of the second phase in the 2-phase case, regardless of branch.

Comment 24

[Shilpi Gupta [:shilpi]]

Ovidiu, Brindusa - Can you please confirm if we have QA sign off on the add on?

Thanks
Shilpi

Comment 25

[ovidiu boca[:Ovidiu]]

The tests were executed on Mac OS X 10.10 and Windows 7 x32.


(In reply to :Paolo Amadini from comment #19)
> TEST PLAN - UPGRADE AND RESTORE
> -------------------------------
> 
> 1. Install the old add-on version 2.2.0, until "phasedVariation" is "false"
> and the branch is not "control". Don't change the study preferences manually.
> 
OK

> 2. Install the old add-on version 2.2.0 again upon itself. You should see
> this error in the Browser Console:
> 
> "UnifiedUrlbar.init() was invoked multiple times?"
>
OK

> 3. Restart the browser
> 
OK

> 4. Install the new add-on version 2.2.1 over the already installed version
> 2.2.0, and check that the previous error message does not appear in the
> Browser Console.
> 
OK
> 5. Uninstall the add-on, then check the value of these preferences:
> 
> browser.urlbar.oneOffSearches
> browser.urlbar.suggest.searches
> 
> They should both be "true", but this is because of a bug in previous
> versions.

The both are on true:  browser.urlbar.oneOffSearches - user set on true
                       browser.urlbar.suggest.searches - default on true


> 
> 6. Reset the two preferences above and restart the browser.
> 
Based on the fact that both preferences were already on true, I didn't do any changes.

> 7. Install the new add-on version 2.2.1, until "phasedVariation" is "false"
> and the branch is not "control".
> 
OK

> 8. Check that the two preferences above are set to "true".
> 
OK

> 9. Restart the browser, then uninstall the add-on.
> 
OK

> 10. Check that the two preferences above are set to "false".
> 
They are on “true”. The same result with Windows and Mac

> 
> 
> TEST PLAN - STUDY BRANCHES
> --------------------------
> 
> 1. To test the number of lines shown when typing in the address bar, you
> have to visit more than 10 pages whose address or title match the search
> terms you will type during the verification process. With the study
> uninstalled, visit at least 10 pages that will match your search, and ensure
> you see all of them when you start typing in the address bar again.

At this step I did a search after “bet” and I opened more than 10 pages that matched my search, I open another tab and did a new search for the word “bet” but I don’t see all of the pages is search suggestion. Please see the attached print screen and tell me if that is the expected behavior. 



> 
> 2. To test study branches, you have to uninstall and reinstall the study
> multiple times, but never change the preferences manually.
> 
 OK

> 3. When "phasedVariation" is "true" or "variation" is "control":
> 
> - None of the changes below should have taken effect. For all the other
> tests, ensure that "phasedVariation" is "false" and "variation" is not
> "control".
> - Start typing in the address bar. You should see at most 10 lines, and you
> should not see any search suggestions or "Search for <...> with:" buttons.
> - The search bar should be present in the toolbar.
> 
 I can see search suggestions and buttons. 


> 4. When "variation" is "oneoff":
> 
> - Start typing in the address bar. You should see search suggestions that
> update while you type, and also "Search for <...> with:" buttons at the
> bottom.
> - You should see at most 10 lines. It's not important whether they are
> search suggestions or sites from local history.
> - The search bar should be present in the toolbar.
> 
OK

> 5. When "variation" is "unified":
> 
> - Start typing in the address bar. You should see search suggestions that
> update while you type, and also "Search for <...> with:" buttons at the
> bottom.
> - You should see at most 10 lines. It's not important whether they are
> search suggestions or sites from local history.
> - The search bar should have been removed from the toolbar.
> 
OK

> 6. When "variation" is "minimal":
> 
> - Start typing in the address bar. You should see search suggestions that
> update while you type, and also "Search for <...> with:" buttons at the
> bottom.
> - You should see at most 6 lines. It's not important whether they are search
> suggestions or sites from local history.
> - The search bar should have been removed from the toolbar.
> 
OK

> 7. When "onboardingVariation" is "true":
> 
> - Start typing in the address bar. You should see a yellow animated "Tip"
> message for the first few times.
> - When the "tipShownCount" preference reaches 0, the tip is not displayed
> anymore.
> 
OK 

> 8. When "onboardingVariation" is "false":
> 
> - Verify that the "Tip" message is not displayed the first time you type in
> the address bar after the study is installed.

OK

Comment 26

[ovidiu boca[:Ovidiu]]

Attachment: Search after "GET" .png

Comment 27

[ovidiu boca[:Ovidiu]]

Attachment: Recording #30.mp4

I tried to test the scenario from comment 9 on Mac OS X 10.10 with the Shield version 2.2.1 and I can't install the add-on, please see the attached video. Before I started to install the add-on the computer was in sleep mode.

Comment 28

[:Paolo Amadini]

(In reply to ovidiu boca[:Ovidiu] from comment #27)
> I tried to test the scenario from comment 9 on Mac OS X 10.10 with the
> Shield version 2.2.1 and I can't install the add-on, please see the attached
> video. Before I started to install the add-on the computer was in sleep mode.

In the video, it looks like the search bar was already removed manually before you tried to install the add-on. Can you try the steps on a new profile?

Comment 29

[:Paolo Amadini]

(In reply to ovidiu boca[:Ovidiu] from comment #25)
> The both are on true:  browser.urlbar.oneOffSearches - user set on true
>                        browser.urlbar.suggest.searches - default on true

Hm, "browser.urlbar.suggest.searches" should default to "false", so it should be user set on "true" here. Do you confirm "false" is the default value you see on a new profile with the unbranded test build?

> > 6. Reset the two preferences above and restart the browser.
> > 
> Based on the fact that both preferences were already on true, I didn't do
> any changes.

At this point, these should both be "false" for the next steps to be relevant.

> At this step I did a search after “bet” and I opened more than 10 pages that
> matched my search, I open another tab and did a new search for the word
> “bet” but I don’t see all of the pages is search suggestion. Please see the
> attached print screen and tell me if that is the expected behavior. 

This is correct, no search suggestions should appear by default until you install the experiment, just the results from the local history.

> > 3. When "phasedVariation" is "true" or "variation" is "control":
> > 
>  I can see search suggestions and buttons. 

This might be an artifact of the preferences being set to "true".

Comment 30

[ovidiu boca[:Ovidiu]]

(In reply to :Paolo Amadini from comment #28)
> (In reply to ovidiu boca[:Ovidiu] from comment #27)
> > I tried to test the scenario from comment 9 on Mac OS X 10.10 with the
> > Shield version 2.2.1 and I can't install the add-on, please see the attached
> > video. Before I started to install the add-on the computer was in sleep mode.
> 
> In the video, it looks like the search bar was already removed manually
> before you tried to install the add-on. Can you try the steps on a new
> profile?

You are right, the search bar was removed before I tried to install the add-on. So the expected result is to not be able to install the add-on if the search bar is removed manually, right? 

I will let the need info for comment 29.

Comment 31

[:Paolo Amadini]

(In reply to ovidiu boca[:Ovidiu] from comment #30)
> You are right, the search bar was removed before I tried to install the
> add-on. So the expected result is to not be able to install the add-on if
> the search bar is removed manually, right? 

Correct.

Comment 32

[ovidiu boca[:Ovidiu]]

(In reply to :Paolo Amadini from comment #29)
                       
> 
> Hm, "browser.urlbar.suggest.searches" should default to "false", so it
> should be user set on "true" here. Do you confirm "false" is the default
> value you see on a new profile with the unbranded test build?
> 


I just opened a new profile of the unbranded test build and in about:config the browser.urlbar.suggest.searches pref is default false

Comment 33

[:Paolo Amadini]

(In reply to ovidiu boca[:Ovidiu] from comment #32)
> I just opened a new profile of the unbranded test build and in about:config
> the browser.urlbar.suggest.searches pref is default false

Thanks for checking. If you try the steps again on this new profile, I believe you should get the expected results.

Comment 34

[ovidiu boca[:Ovidiu]]

Thanks for suggesting that, you were right, I managed to get the expected results. 

1. I need to do some more test about:

> > 3. When "phasedVariation" is "true" or "variation" is "control":
> > 
>  I can see search suggestions and buttons. 
>This might be an artifact of the preferences being set to "true".


2. Do you think we should run the full tests on Ubuntu?

Thanks

Comment 35

[:Paolo Amadini]

(In reply to ovidiu boca[:Ovidiu] from comment #34)
> 2. Do you think we should run the full tests on Ubuntu?

No, I don't think it's necessary.

Comment 36

[:Paolo Amadini]

(In reply to ovidiu boca[:Ovidiu] from comment #34)
> 1. I need to do some more test about:

It will definitely be good to repeat this test with the right starting conditions. I believe there's a high chance it will pass, but it's good to be sure.

Given the above and the need to launch soon, I'd be fine with starting user enrollment on this version. The live update tests already passed, this means we can always push an update later if we find out that there is a problem.

Comment 37

[Shilpi Gupta [:shilpi]]

Ovidiu, Please provide an update here once you can sign off on the add on. 
We are targeting to launch in shield tomorrow early morning PT.

Comment 38

[ovidiu boca[:Ovidiu]]

From the QA perspective, this add-on is ready to be launch, during the tests we didn't find any issue that should block this add-on. We performed all the tests mentioned in the previous comments, on Windows 7 x32 and Mac OS X 10.10.

Comment 39

[Shilpi Gupta [:shilpi]]

Thanks much!

Comment 40

[Panos Astithas (he/him) [:past] (please ni?)]

The study shipped a couple of weeks ago.