1482158 - Expand organic search telemetry to include major engines

Status: RESOLVED FIXED

(Data Platform and Tools :: General, enhancement, P1)

Description

[Ryan Harter [:harter]]

In Bug 1475571 we implemented organic search telemetry for Google. We are interested in measuring partner market share. To do so, we need to expand this telemetry to include major search engines.

Comment 1

[Ryan Harter [:harter]]

I have an initial engine list below. We may need to add engines to this list once we hear back from our partners.

* Yahoo!
* Bing
* DDG

Comment 2

[Mike Kaply [:mkaply]]

Attachment: First pass at Bing/DuckDuckGo

Here's Bing and DuckDuckGo

For Bing, we can't distinguish between organic followon and sap followon once there has been an sap followon (it's cookie based).

For DuckDuckGo, we can't distinguish sap followon from sap.

Comment 3

[Florian Quèze [:florian]]

Comment on attachment 8999285 [details] [diff] [review]
First pass at Bing/DuckDuckGo

Review of attachment 8999285 [details] [diff] [review]:
-----------------------------------------------------------------

::: browser/modules/BrowserUsageTelemetry.jsm
@@ +301,4 @@
>  
> +          // This code is used for all Bing follow-on searches
> +          if (secondaryCode == "QBRE") {
> +            let enumerator = Services.cookies.getCookiesFromHost("www.bing.com");

Are we sure the cookies are retrieved from memory and this call won't cause main thread I/O?

If we don't provide the second parameter (origin attribute), isn't this going to return the wrong results for pages loaded in container tabs?

Comment 4

[Mike Kaply [:mkaply]]

> Are we sure the cookies are retrieved from memory and this call won't cause main thread I/O?

I don't know the answer to that. What's the best way to find out?

> If we don't provide the second parameter (origin attribute), isn't this going to return the wrong results for pages loaded in container tabs?

I'll fix.

Comment 5

[Florian Quèze [:florian]]

(In reply to Mike Kaply [:mkaply] from comment #4)
> > Are we sure the cookies are retrieved from memory and this call won't cause main thread I/O?
> 
> I don't know the answer to that. What's the best way to find out?

From reading a bit of the code in nsCookieService.cpp, it seems this will block only if accessed during early startup. If you call this after an HTTP request has already been performed, then it should be fine.

Comment 6

[Mike Kaply [:mkaply]]

Attachment: Bug 1482158 - Add telemetry for other major engines.

Comment 7

[Mike Kaply [:mkaply]]

Mike De Boer:

This new patch incorporates a regex for additional engines and for handling cases where we get a URL before the search service is intialized (which allows us to eliminate some code).

I do not handle Yandex because they load their URLs 3 or 4 times so I can't track them properly unless we add some code to eliminate duplicate URLS.

Comment 8

[Mike de Boer [:mikedeboer]]

Comment on attachment 9002500 [details]
Bug 1482158 - Add telemetry for other major engines.

Mike de Boer [:mikedeboer] has approved the revision.

Comment 9

[Tiberius Oros[:tiberius_oros]]

Backed out changeset b38e10e9dcf8 (bug 1482158) for eslint failure at builds/worker/checkouts/gecko/browser/modules/BrowserUsageTelemetry.jsm

Backout: https://hg.mozilla.org/integration/mozilla-inbound/rev/216533f4ee8e07bf38d2da097c4afb507be1b65d

Push with the failure: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=b38e10e9dcf8008b7f33cc5cef8e695d5a0bf98f

Failure log file: https://treeherder.mozilla.org/logviewer.html#?job_id=198494501&repo=mozilla-inbound&lineNumber=257

Comment 10

[Mike Kaply [:mkaply]]

Argh. Repushed.

Comment 11

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/b2e1ae4f75548489e995029090cedaad5b3f317d
https://hg.mozilla.org/mozilla-central/rev/b2e1ae4f7554

Comment 12

[Ryan Harter [:harter]]

Woot! Mike, can you confirm this includes Yahoo?

Comment 13

[Mike Kaply [:mkaply]]

Yep. Yahoo, google, duckduckgo, bing, baidu. Yandex wasn't possible using this mechanism.

Comment 14

[Tim Smith (inactive) 👨‍🔬 [:tdsmith]]

Can we deprotect this bug since this code has shipped? I'd just like third parties to be able to follow the bug_numbers in Histograms.json.

Comment 15

[Ryan Harter [:harter]]

SGTM. :mconnor to confirm we're OK making this bug public.

Comment 16

[Mike Connor [:mconnor]]

Yes, we can unhide this bug.