1518393 - [Shield] Pref-Flip Study: Search Volume in Private Browsing Observational Study, v64, release

Status: RESOLVED FIXED

(Shield :: Shield Study, enhancement)

Description

[Experimenter Bot]

Search Volume in Private Browsing Observational Study

Search volume is currently recorded in telemetry in all modes (regular and private browsing), but because it is aggregated together we have no sense of how much users search within private browsing windows making it harder to discern whether or not we should invest in specific search functionality in private browsing mode.
If, for example, a significant percentage of overall searching happens from private browsing, it may make sense to explore adding search access points to about:privatebrowsing to improve the user experience

More information: https://experimenter.services.mozilla.com/experiments/search-volume-in-private-browsing-observational-study/

Comment 1

[Julien Cristau [:jcristau] (back April 22)]

QA signed off in bug 1504370 comment 33. I'm assuming this will also target 65, so updating tracking flags. According to experimenter this is set to run from Jan 22 to Feb 19, for 0.3% of the release channel. Signing off for relman on that basis.

Comment 2

[Experimenter Bot]

    Experiment Type: Pref Flip Study

    What is the preference we will be changing

browser.engagement.search_counts.pbm

    What are the branches of the study and what values should
    each branch be set to?

- Treatment Control 50%:

Value: false

Vanilla Firefox, no changes
        
- Treatment Treatment 50%:

Value: true

Sets the pref true which will trigger search counts to NOT be collected for this branch.
        

    What version and channel do you intend to ship to?

0.3% of Release Firefox 64.0

    Are there specific criteria for participants?

Locales: All

Geographic regions: All

Prefs: None

Studies: None

Any additional filters: We'd like to target Firefox Release 64 *and* Release 65 to account for the merge day of 1/29.

    What is your intended go live date and how long will the study run?

Jan 22, 2019 - Feb 19, 2019 (28 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

What is the main effect you are looking for and what data will
you use to make these decisions? What metrics are you using to measure success?

We are looking to understand the percentage of search volume (measured via search counts) coming from pbm and anticipate that if at least 20% of searches occur from pbm, that information will be used to inform investment decisions related to new search features within pbm. We will use the already-existing search telemetry to measure searches from pbm

Do you plan on surveying users at the end of the study? No

    Who is the owner of the data analysis for this study?

Ben Miroglio

    Will this experiment require uplift?

False

    QA Status of your code:

QA Status of your code: - (bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1504370)

    Link to more information about this study:

https://experimenter.services.mozilla.com/experiments/search-volume-in-private-browsing-observational-study/

Comment 3

[Experimenter Bot]

    Experiment Type: Pref Flip Study

    What is the preference we will be changing

browser.engagement.search_counts.pbm

    What are the branches of the study and what values should
    each branch be set to?

- Treatment Control 50%:

Value: false

Vanilla Firefox, no changes
        
- Treatment Treatment 50%:

Value: true

Sets the pref true which will trigger search counts to NOT be collected for this branch.
        

    What version and channel do you intend to ship to?

0.3% of Release Firefox 64.0

    Are there specific criteria for participants?

Locales: All

Geographic regions: All

Prefs: None

Studies: None

Any additional filters: We'd like to target Firefox Release 64 *and* Release 65 to account for the merge day of 1/29.

    What is your intended go live date and how long will the study run?

Jan 22, 2019 - Feb 19, 2019 (28 days)

    What is the main effect you are looking for and what data will you use to
    make these decisions?

What is the main effect you are looking for and what data will
you use to make these decisions? What metrics are you using to measure success?

We are looking to understand the percentage of search volume (measured via search counts) coming from pbm and anticipate that if at least 20% of searches occur from pbm, that information will be used to inform investment decisions related to new search features within pbm. We will use the already-existing search telemetry to measure searches from pbm

Do you plan on surveying users at the end of the study? No

    Who is the owner of the data analysis for this study?

Ben Miroglio

    Will this experiment require uplift?

False

    QA Status of your code:

QA Status of your code: - (bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1504370)

    Link to more information about this study:

https://experimenter.services.mozilla.com/experiments/search-volume-in-private-browsing-observational-study/

Comment 4

[Ryan VanderMeulen [:RyanVM]]

Study has been marked as complete in Experimenter and disabled. I think we're done with this bug.