Ubuntu and Chinese searches with default engine reported as "other"

RESOLVED WORKSFORME

Status

Firefox Health Report
Client: Desktop
P3
normal
RESOLVED WORKSFORME
5 years ago
3 years ago

People

(Reporter: kmag, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [measurement:client])

(Reporter)

Description

5 years ago
I'm trying to do some analysis to correlate add-ons with the use of non-default search engines. My numbers are thrown off by the fact that Ubuntu's builtin search engines are all reported as "other". I haven't been able to test a Chinese localization, but searches from Chinese users seem to be almost exclusively via "other" engines as well.
Is that true for Firefox 27 and higher?

Prior to Bug 925521 we only report a small set of partner search engines.
(Reporter)

Comment 2

5 years ago
In Aurora 27 on Ubuntu, search with the stock engine shows up as other-Google, which is better but still not ideal. I'm still not sure what's going on with Chinese distributions, so I can't check there yet.
That implies that those search engines don't have identifiers, which implies that they're not the usual XML file in searchplugins. That might be a packaging bug. How does Ubuntu specify its search engines?
(Reporter)

Comment 4

5 years ago
It looks like they are XML files. The default search engine is:

/usr/lib/firefox/distribution/searchplugins/locale/en-US/google.xml:
<SearchPlugin xmlns="http://www.mozilla.org/2006/browser/search/">
<ShortName>Google</ShortName>
<Description>Google Search</Description>
<InputEncoding>UTF-8</InputEncoding>
<Image width="16" height="16">data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAABUUlEQVR42pWTzUsCYRCH9y9zu3SooCCkjhIRRLeIykXokiWCJ7PvDpZRlz6si1lIQZ3SQxQdOhREpgSm0JeQvfu0+i6I7LKLh4F5h5nnnRl+o6jTdHn8omAYbVqhXqvYFXcEBKFDwcoZZB8B4LkEB9cwGGmFKHb01A1EU9JXzfdvDYZi1lwLwBcVAIwsNWPesIwls7gDtB2Z7N9ujVe+IX2LO2AgItB1OL9vJqsmILDrOoK02IkBAdYy4FsQJC5h+VQCHQDWTqYSgo8fuHuRxS4Ae3stQ7UGE5ttAHqCUgfxC7m4ryrowOyeO6CxqHwZxtYFqtYc5+kNan/gDTsAeueEIRj7n/rmRQMwueUAGF0VAAT3rQBTC0Y3DoDOGbm00icML4oWHYSTgo0MFqjlmPpDgqMcFCuQf4erBzjOwXjcriu9qHg0uutO2+es6fl67T9ptebvFRjBVgAAAABJRU5ErkJggg==</Image>
<Url type="application/x-suggestions+json" method="GET" template="https://www.google.com/complete/search?client=firefox&amp;q={searchTerms}"/>
<Url type="text/html" method="GET" template="https://www.google.com/search">
  <Param name="client" value="ubuntu"/>
  <Param name="channel" value="fs"/>
  <Param name="q" value="{searchTerms}"/>
  <Param name="ie" value="utf-8"/>
  <Param name="oe" value="utf-8"/>
</Url>
<SearchForm>https://www.google.com/</SearchForm>
</SearchPlugin>
Yeah, that's packaged into a distribution, not in the browser itself. Compare to the official tarball:

./browser/searchplugins/google.xml

So the questions are:

* Why does a distribution search engine not have an identifier?
* Should it?
* Why is Ubuntu packaging Firefox with its own client value and not all of our attribution tags?

Official: 

<Url type="application/x-suggestions+json" method="GET" template="https://www.google.com/complete/search?client=firefox&amp;q={searchTerms}"/>
<Url type="text/html" method="GET" template="https://www.google.com/search">
  <Param name="q" value="{searchTerms}"/>
  <Param name="ie" value="utf-8"/>
  <Param name="oe" value="utf-8"/>
  <Param name="aq" value="t"/>
  <Param name="rls" value="{moz:distributionID}:{moz:locale}:{moz:official}"/>
(Reporter)

Comment 6

5 years ago
Gerv, do you know who's responsible with our licensing relationship with Ubuntu, or who can answer the questions in comment 5?
Flags: needinfo?(gerv)
I do not :-| Jishnu - who manages the Canonical relationship?

Gerv
Flags: needinfo?(gerv)

Comment 8

5 years ago
I'll talk to Chris at Ubuntu in the am, probably quicker/easier.

Comment 9

5 years ago
(and remember that Ubuntu localization is a wee bit different than how we do it. that's tripped us up with other things, and I suspect it's either the l10n framework or the ubufox extension getting in the way.)
I don't think this is a new thing, we've had an agreement with Canonical for quite some time and they've been using their codes.  Using /distribution seems like the obvious way for them to include their chosen search engines with their codes.

I suspect the same thing applies for builds distributed by Mozilla Online, if that's what you mean by the Chinese localization.  Note that there are vanilla builds we produce as well as the "China Edition" they distribute (along with a set of add-ons).

Kev: happy to have you touch base and verify things, I don't have a current relationship with the Ubuntu guys.
(Reporter)

Comment 11

5 years ago
(In reply to Mike Connor [:mconnor] from comment #10)
> I don't think this is a new thing, we've had an agreement with Canonical for
> quite some time and they've been using their codes.  Using /distribution
> seems like the obvious way for them to include their chosen search engines
> with their codes.

Either way, we should find some way to report these sensibly in FHR.

> I suspect the same thing applies for builds distributed by Mozilla Online,
> if that's what you mean by the Chinese localization.  Note that there are
> vanilla builds we produce as well as the "China Edition" they distribute
> (along with a set of add-ons).

The only detail I have is that users with regional add-ons for China, most obviously, but not exclusively, add-ons with @mozillaonline.com IDs, tend to exclusively use an "other" engine. I haven't figured out what that means yet.

Comment 12

5 years ago
Interesting. I'll have a look at both in the am. Thanks Kris & Mike.
Back to this particular bug:

Distribution search engines _intentionally_ don't include their identifier in the engine object. That decision probably didn't anticipate the use of distributions for "official" builds of Firefox that aren't actually official builds -- that is, builds where we do want search engine identifiers, and where Mozilla or a Mozilla partner included the search engine, but it's not actually in the app dir.

So, we could do one of these things:

* Continue on the current path, which will probably lead to Kris parsing "other-Google" as equivalent to "google".
* Provide identifiers for distribution engines that might collide (either on the same machine, or a collision in that it might be a different engine definition with the same name) with built-ins.
* Provide a different namespace for identifiers for these engines: "d:google", say.

I favor the third option. mconnor, have an opinion?
Flags: needinfo?(mconnor)
(Reporter)

Comment 14

5 years ago
I'd also prefer the third option.
I think the third option makes the most sense as well.  I'm not sure whether it is useful to lump all distribution-shipped variants into a single namespace, vs. using the value of distribution.id to prepend (i.e. canonical-Google).  I'd probably argue that, given that a user could conceivably switch distributions, recording the specific distribution makes the most sense, rather than correlating all distribution searches with the current value of distribution.id.

As we already disclose distribution.id in FHR, I do not think this requires significant discussion on privacy grounds.
Flags: needinfo?(mconnor)
OS: Linux → All
Hector, can you maybe take a look if that is still an issue now?
How do searches from the China builds show up in Telemetry?
Flags: needinfo?(bzhao)
Priority: -- → P4
Whiteboard: [measurement:client]
Duplicate of this bug: 1006132
Priority: P4 → P3
(In reply to Georg Fritzsche [:gfritzsche] from comment #16)
> Hector, can you maybe take a look if that is still an issue now?
> How do searches from the China builds show up in Telemetry?

In Fx 43 release:
defaultSearchEngine 	other-百度
defaultSearchEngineData.name 	百度
defaultSearchEngineData.loadPath 	[distribution]/searchplugins/locale/zh-CN/baidu.xml
defaultSearchEngineData.submissionURL 	http://www.baidu.com/baidu?wd=&tn=monline_4_dg

In Fx 46 nightly:
defaultSearchEngine	baidu
defaultSearchEngineData.name	百度
defaultSearchEngineData.loadPath	[distribution]/searchplugins/locale/zh-CN/baidu.xml
defaultSearchEngineData.submissionURL	http://www.baidu.com/baidu?wd=&tn=monline_4_dg

Seems to be fixed by bug 1203167 in Fx 45
Flags: needinfo?(bzhao)
In Fx 43.0.4 (Ubuntu, Release):

defaultSearchEngine 	other-Google
defaultSearchEngineData.name 	Google
defaultSearchEngineData.loadPath 	[distribution]/searchplugins/locale/en-US/google.xml
defaultSearchEngineData.submissionURL 	https://www.google.com/search?client=ubuntu&channel=fs&q=&ie=utf-8&oe=utf-8 

In Fx 46 nightly (Ubuntu, buildid 20160109070010):

defaultSearchEngine	google
defaultSearchEngineData.name	Google
defaultSearchEngineData.loadPath	[distribution]/searchplugins/locale/en-US/google.xml
defaultSearchEngineData.submissionURL	https://www.google.com/search?client=ubuntu&channel=fs&q=&ie=utf-8&oe=utf-8
Thanks for the details here.

I sent a heads-up to fhr-dev about the changes from bug 1203167:
https://mail.mozilla.org/pipermail/fhr-dev/2016-January/000760.html
Any follow-ups to that can happen in new bugs.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.