1587874 - Add private browsing default search engine to search_* datasets

Status: RESOLVED FIXED

(Data Platform and Tools :: General, task)

Description

[Ben Miroglio [:bmiroglio]]

In bug 1411340 we added an option to Firefox so that users can specify a different search engine when in private browsing mode. I'd like to add this to search_clients_daily and search_aggregates to support future analysis.

Comment 1

[William Lachance (:wlach)]

Ben, might it make sense to do this before backfilling bigquery-etl?

Comment 2

[Ben Miroglio [:bmiroglio]]

Definitely!

Getting Bug 1576712 landed would be ideal pre-backfill as well.

Comment 3

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/bigquery-etl/pull/456

Comment 4

[Ben Wu [:benwu]]

This is now in the latest search dataset in the fields default_private_search_engine, default_private_search_engine_data_load_path, and default_private_search_engine_data_submission_url. The fields only started to get populated 2019-11-19 because the field needed to be added to main summary after the main_summary->GCP migration and main summary was not backfilled and won't be.

bmiroglio: Can you take a look to see if the fields are fine and this can be resolved? search.search_clients_daily and search.search_aggregates

Comment 5

[Ben Miroglio [:bmiroglio]]

The default_private_search_engine field is mostly null after 11-19-2019. Is this only populated when is it explicitly set by the user?

Comment 6

[Ben Wu [:benwu]]

I believe it is: I just checked my own telemetry and the field only appeared after I unchecked the "Use this search engine in Private Windows" box and selected a different one than my default. You may want to double check with whoever implemented this though.

I'm also not sure which versions this setting is in. In my beta installation the setting was there for 71.0b4 but disappeared when I updated to 71.0b12 which is interesting. It's in nightly though and if you filter by nightly only the percentage is higher but still <1%.

Comment 7

[Ben Miroglio [:bmiroglio]]

From a discussion with :standard8 on Slack:

It only gets populated when the ui is enabled (feature pref), and the user has deselected “Use this search engine in Private Windows” in the preferences UI

So many nulls are to be expected.

With that, things LGTM!