1427133 - Clean-up search engines in TB: Add Duck Duck Go and Google to the default list of search engines, remove disfunctional AOL search

Status: RESOLVED FIXED

(Thunderbird :: Search, enhancement)

Comment 1

[Matt]

Google was removed from the list of opensearch providers some years ago due to a contractual problem between Mozilla and Google.  I do not think we do our users any service by simply not having Google in the available search list.

With the rise of folk being concerned about tracking we should also include Duck Duck Go as one of the more popular non tracking search engines. This particular provider is included in Firefox,  but Not Thunderbird.

Comment 2

[Jorg K (CEST = GMT+2)]

Hmm, how would we do this? Google seems to be included in some locales, see:
https://dxr.mozilla.org/comm-central/source/mail/locales/search/list.json
So is it just a matter of adding google to "default" or "en-US"?

There are also four mentions of "duckduckgo" in that file, but don't we need an XML here?
https://dxr.mozilla.org/comm-central/source/mail/locales/en-US/searchplugins

I have zero idea, so please help!

Comment 3

[Mike Kaply [:mkaply]]

I'm honestly confused by what happened here because none of the needed engines are present in the build. I think it must be picking them up elsewhere? I'll try to take a look.

Comment 4

[Mike Kaply [:mkaply]]

Locale specific thunderbird engines are still stored in the locale directories:

https://dxr.mozilla.org/l10n-central/search?q=file%3A*.xml&redirect=false

So the files are referenced in list.json, but they are packaged via the locale files.

So if you wanted to add google and ddg for English, you should take the files from browser/locales/searchplugins, remove the codes and place them in

comm-central/mail/locales/en-US/searchplugins

Then update list.json (google is actually already there, so you just need to add it to list.json)

IF you want to put them in any locales other than English, they would need to go into the specific locale.

We should probably spin off a larger bug to unify the locales as we have for Firefox. It's made life a lot easier.

Comment 8

[Wayne Mery (:wsmwk)]

Magnus is the only currently active developer for https://addons.thunderbird.net/en-US/thunderbird/addon/google-search-for-thunderbi/

Comment 9

[Michael Loney]

Thus more reason to add them. Cannot rely on an incompatible add-on to get decent search in Thunderbird. At least please add DDG in a following release.

Comment 10

[Jorg K (CEST = GMT+2)]

The fix doesn't look too hard according to comment #4.

Comment 11

[Jorg K (CEST = GMT+2)]

Note: We're working on bug 1427317 that will allow to add search engines manually. Of course we should still clean-up the search engine area.

Comment 13

[Jorg K (CEST = GMT+2)]

Attachment: 1427133-search.patch

After a while we've come back to this. What I've done here is this:
Copied browser/components/search/searchplugins to mail/components/search/searchplugins absolutely 1:1.

From mail/components/search/searchplugins I removed the defunct AOL search and also Yahoo which appears to have fallen out of favour. I don't really want to spend the time to fiddle Yahoo back into list.json.

Mike, does this sound right to you?

In a local build it all looks good: There's Google, DDG, etc.

We're a bit under pressure to fix search engines in TB, so I hope this can be backported to TB 60.

Comment 14

[Jorg K (CEST = GMT+2)]

Attachment: 1427133-search-esr60.patch

OK, the mechanism in ESR 60 is different. At least we can add Google and remove the defunct AOL engine. That's better already.

Comment 15

[Jorg K (CEST = GMT+2)]

Attachment: 1427133-search-esr60.patch (v1b)

Added Google to en-US as well.

Comment 16

[Matt]

(In reply to Jorg K (GMT+1) (urgent reviews and bustage fix only, Dec 22nd to Jan 1st) from comment #15)
> Created attachment 9033890 [details] [diff] [review]
> 1427133-search-esr60.patch (v1b)
> 
> Added Google to en-US as well.

Thanks Jorg,  could you include duck duck go as well the two things that originally started this bug.

Comment 17

[Jorg K (CEST = GMT+2)]

Attachment: 1427133-search-esr60.patch (v2)

Here also removing aol-web-search.xml. Is that a good idea or can locales still refer to it. Also adding ddg.xml from trunk and adding it to defaults and en-US/en-GB. Mike will tell us whether it's correct like this.

Comment 18

[Florian Quèze [:florian]]

(In reply to Jorg K (GMT+1) from comment #13)
> What I've done here is this:
> Copied browser/components/search/searchplugins to
> mail/components/search/searchplugins absolutely 1:1.

This means including in Thunderbird all the partner codes that Firefox uses. I don't think this will be OK, but someone who knows how the Firefox search deals work will have the final word on this. Hopefully mkaply can comment on this, or point you to someone who can.

Comment 19

[Jorg K (CEST = GMT+2)]

Well, Mike said in comment #4: "remove the codes". I didn't understand what he meant. Looking at a particular file:

+++ b/mail/components/search/searchplugins/amazon-en-GB.xml
@@ -0,0 +1,18 @@
+<!-- This Source Code Form is subject to the terms of the Mozilla Public
+   - License, v. 2.0. If a copy of the MPL was not distributed with this
+   - file, You can obtain one at http://mozilla.org/MPL/2.0/. -->
+
+<SearchPlugin xmlns="http://www.mozilla.org/2006/browser/search/">
+<ShortName>Amazon.co.uk</ShortName>
+<Description>Amazon.co.uk Search</Description>
+<InputEncoding>UTF-8</InputEncoding>
+<Image width="16" height="16">resource://search-plugins/images/amazon.ico</Image>
+<Url type="text/html" method="GET" template="https://www.amazon.co.uk/exec/obidos/external-search/" resultdomain="amazon.co.uk">
+  <Param name="field-keywords" value="{searchTerms}"/>
+  <Param name="ie" value="{inputEncoding}"/>
+  <Param name="mode" value="blended"/>
+  <Param name="tag" value="firefox-uk-21"/>
+  <Param name="sourceid" value="Mozilla-search"/>
+</Url>
+<SearchForm>https://www.amazon.co.uk/</SearchForm>
+</SearchPlugin>

it's hard to tell where the "code" is. Maybe tag and sourceid. I can of course remove those whole-sale. The somewhat more pressing issue is the ESR60 patch which should be straight forward.

Comment 20

[Florian Quèze [:florian]]

Comment on attachment 9033886 [details] [diff] [review]
1427133-search.patch

Review of attachment 9033886 [details] [diff] [review]:
-----------------------------------------------------------------

::: mail/components/search/searchplugins/amazondotcom.xml
@@ +14,2 @@
>    <Param name="mode" value="blended"/>
> +  <Param name="tag" value="mozilla-20"/>

<Param name="tag" in amazon plugins is definitely a partner code.

@@ +14,3 @@
>    <Param name="mode" value="blended"/>
> +  <Param name="tag" value="mozilla-20"/>
> +  <Param name="sourceid" value="Mozilla-search"/>

I'm not sure about the sourceid param. Looks like you didn't have it before in Thunderbird.

::: mail/components/search/searchplugins/baidu.xml
@@ +14,5 @@
> +  <Param name="action" value="opensearch"/>
> +</Url>
> +<Url type="text/html" method="GET" template="https://www.baidu.com/baidu" resultdomain="baidu.com">
> +  <Param name="wd" value="{searchTerms}"/>
> +  <Param name="tn" value="monline_dg"/>

The 'tn' parameter on baidu is a referral tag.

::: mail/components/search/searchplugins/bing.xml
@@ +16,2 @@
>          <Param name="q" value="{searchTerms}"/>
> +        <Param name="pc" value="MOZI"/>

This looks like a partner code.

You are removing <Param name="form" value="MOZESB"/> which was definitely a partner code. IIRC Thunderbird had a search deal with Bing 8 years ago. Does this deal still exist?

::: mail/components/search/searchplugins/ddg.xml
@@ +13,5 @@
> +    <MozParam name="t" condition="purpose" purpose="contextmenu" value="ffcm"/>
> +    <MozParam name="t" condition="purpose" purpose="keyword"     value="ffab"/>
> +    <MozParam name="t" condition="purpose" purpose="searchbar"   value="ffsb"/>
> +    <MozParam name="t" condition="purpose" purpose="homepage"    value="ffhp"/>
> +    <MozParam name="t" condition="purpose" purpose="newtab"      value="ffnt"/>

This whole block is partner codes.

::: mail/components/search/searchplugins/ebay.xml
@@ +8,5 @@
> +<Image width="16" height="16">resource://search-plugins/images/ebay.ico</Image>
> +<Url type="text/html" method="GET" template="https://rover.ebay.com/rover/1/711-53200-19255-0/1" resultdomain="ebay.com">
> +  <Param name="ff3" value="4"/>
> +  <Param name="toolid" value="20004"/>
> +  <Param name="campid" value="5338192028"/>

campid contain a compaign id used for affiliation tracking.

::: mail/components/search/searchplugins/ecosia.xml
@@ +11,5 @@
> +  <Param name="type" value="list"/>
> +  <Param name="q" value="{searchTerms}"/>
> +</Url>
> +<Url type="text/html" method="GET" template="https://www.ecosia.org/search">
> +  <Param name="tt" value="mzl"/>

I think this is a partner code.

::: mail/components/search/searchplugins/google-2018.xml
@@ +1,1 @@
> +<!-- This Source Code Form is subject to the terms of the Mozilla Public

I don't think you need more than one google search plugin in Thunderbird, and you definitely don't need the google-2018 one which is specifically used by Firefox only in the US.

::: mail/components/search/searchplugins/google.xml
@@ +8,5 @@
>  <InputEncoding>UTF-8</InputEncoding>
> +<Image width="16" height="16">resource://search-plugins/images/google.ico</Image>
> +<Url type="application/x-suggestions+json" method="GET" template="https://www.google.com/complete/search?client=firefox&amp;q={searchTerms}"/>
> +<Url type="text/html" method="GET" template="https://www.google.com/search" rel="searchform">
> +  <MozParam name="client" condition="purpose" purpose="keyword" value="firefox-b-ab"/>

The client parameter is the partner code on the Google plugins.

::: mail/components/search/searchplugins/twitter.xml
@@ +11,2 @@
>    <Param name="q" value="{searchTerms}"/>
> +  <Param name="partner" value="Firefox"/>

This looks a lot like a partner code ;).

::: mail/components/search/searchplugins/yahoo-jp-auctions.xml
@@ +9,5 @@
> +<Image width="16" height="16">data:image/x-icon;base64,AAABAAIAEBAAAAEAIAAEAwAAJgAAACAgAAABACAARgcAACoDAACJUE5HDQoaCgAAAA1JSERSAAAAEAAAABAIBgAAAB/z/2EAAALLSURBVDiNfZNbSFRRFIb/dc4Zz2gz3iK1oFToJSPCpECoqBAiCPPFy0PFoBUlEWRmiFDjg4VmkYHdCDQIIiswM30IL5g6mQ1qeQuzcVQ0YWq00XScc87qQSfHAVsv+2Hv/1v/umxgjWBmWj4N9vdvzSNNr079u6uqEtfSrQD6qgKWAVGtN3Pmuouz2d5ef2YlwX8gbDYLAAgA2ooyTC9SjTMvD4EtRWd5tK0+28elIPnbbm4uFOmgWQXAlpJMk8s2UGHQheF3WLB7vOWBrDGXjzTXIOZA8j0i0iRfMREBgAIAlpJM06xtoMLj+AJN0HuCZKM8p210OxsfyiERm8pdo/29xi1xLZKPmL2wjzdMpmnbQIXH8RmaYPCwuE7Hs985InKXrO1Iap+ZcVTOOZ2DAEDMLBCRxsyBYx0Nub9GhvZMvLufRK5veg+FKhACJF4YcRui9snGuMTa3eeL04ho3ptYgtUqAtAAbPg11HvN3VImTk/YPOtCYzUBisTqgiouQp4XAhGTePTLslhcdq0ISEhQABARjUbGJ2Zp29NVvQLdgkdVFbcDsjQvRqRcH3YFx2rdzx7l27ss1wCoRKQ0NTVJq5YGAJy2AVNn+VWu2gt+c3InfyjLa2Tm4LE+6/nagtNcffkEj/Z0FKyauxkQ6uruygB0APBzeDDTUnqJe4rPsWPQetz7bry/+2J1fha/zjvJtm7LlaUmAkQAwy+mej9lT9u/xgfNTV3YnJYz7wvpfFJ2O8RoQPTh1IsEAE+PhAe71EW9qs6q4bHJkv1xzWI+4PRbMpGIVAAY6mjOsXW23orYui2XAKB0f+AdEoQMMCYAZT2k8K5LDZMpwNLHobQ01R/CzNEAxqgmPSpbJJTrRUDRAFEAmBl/1IBbx56P5QKAb5nevfE6k0DkAqhyEfjBIB0zKxojHOyZ9O8LluatMTPBapUKa2vVv/Q3b+fAt5TJAAAAAElFTkSuQmCCiVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAHDUlEQVRYha2Xa3AT1xXH/2f3rixZfie2eQ9JKK+EJinm0RRcPCkm00cSIKb1wDTNtIlJO3TSyfRLv7BpM518aHAmHejkVTJtEicycXAInYSXFGqoS+WkmJcfWEBpjI0B+SFZ2t27e/pBspGNBAZ6ZjQare4957f/87i7wG0aM6vJb0/0Qtci49LZeSn/0Y32K7cZnIjIZmZPtK97+3DHF0fO/6vppNHfvRYAAoEX1Nvxf0MLBoMaALA5+BPuaOY6wPy0uoBDn9WxTEL4/bqYiBK3ZKzrCgAwc9bh2uc/2LO+hBsfR/xvVUhC9K4FAPb51EwQ4nYAAggoAJyT23/9QP/5Y0v7oyay3bNcVuy0bHujWgB1O2R/9xNUMOVDXwLCISL+vwD4dV1U6Lpsr/vt8kunOw5khc8LZrYHYwNqXvYsYcVOy7Y3qwXw/g451LtW5JY2pIO4pdwEX3tGK6t53Tr5rl5+JdS5L3LKrxnGgCRRICKmhHRs5GXng2JdUgiIeU+/jxkPVawVuaUNzD6VaJ094uumuyA1eDjUuS96yq8ZRkSSVigAB15NgVBUDA4Pgt33CCkh2974Ef5z2P8hD/WuIVpn+3y+0e64KQVY1wXpukwG3xtpC7ji8SFJWp4A21fXMRC1HEjHRr63EBzrtFRAm1dTh5nLVm0QOUXvMrNCRM6EFfCPDb4v0uZ3xYxBS3HlC7Acs5YI8GqAJrIRDnfaInu2JnsB5UIXzMjAWgBAfT0B1ynCZNuoaGmhlpYWlNXUWFeDB7S4MWSpWoHGjnWNkMwOVOGFOxaSntIyIYru6HffeRc7xZO/8JR6fwwAqKpycM3Oq8EVInJSr3X6fv/ti+0n9kRO+V1xIyIVLV8wXxsc7ICEFzISsnKL5mqllU9HJs2dW1H8je8FmU+4iO41kxOU0wIwszoyXu34wAbV7RnuOXJAnN617dX4uZN5w8MXJWkjso/fziDFDTtyVuYW3ydKVz4VmfbgkoqCucuCmW5uTApSZrs72tf9tnPu+LpI3EJf51Fc+nIXlJwpTiK4nV48ErDNbsdb8oAoXv7EhekLFj6WP3dZMPjaM9onF6bYmzdv5vHKjvEy0qPG5XPzw63NJwKrf2grd4PjOSD1julKjiaJ0icNIAHVVYT4f1vljNW/Efeu3/RxdtHkx6oA1Ze4MyfdzjFdUF+f+HYVzYhw4bTI1J8+rGpTwYUz71eJoxQxHTjMYz2wA1LccKzLTuxMqxUWUA821qL90OeP2kb0d/WATUQOM6ftuHQ1oCQ3lPcc++fejjefc11ua7a0kvnaUKQXDAGvi6AQgdmBIryQQyHpLV4gJn2rGoHG7Rg+0Sm9ORCrtuzEPeWrdFXzvJDqO6MCAEBEjq7rChEdnLRgycrZP3vFLJqzVLMunrRyvSUgOIiaDIcdqCIbMhKS+SVfF6XLq7rnrHnqrys2/YELp0JYWbB2b3wcZ5r26rYR2zzie7wSGSdhMBjUysrKLGYu72lt3tfx1q+0K+3NUtw5RwxFrwCqG27rvJ1XNFudVFkTmb5w0cP5c8qPsG1uCv0j8Oqnz1fCJFi4BO37rzfi7uWV11UirfHIAwdzec/RZjOwaQl/tArWrg3z+J0fwN6x/j4++peXeqNdTWUA4NOrXMn1G0OH9/LWxeDapTBr54A79u9k24jpqam+IUByoRiBuNDabAQ2LeGPvgur4VHIv29+kge7zxwfUZOZFT2ZVmbeGDq0h7cuBW9ZDPOV+eDO/Y23COH3j0L0HG02P3+unN8Dhltf/iWHu47tGV2XgCAkU8vMz3Y17eFtD4FfXpiA6Ni/k20rURM3ZakQ3f8+FOve8Ufu3P3n/XzG705eH60nZqYUJZ7tOpSA2LI4kY7OA41sGddCkA4oDFC6j67rSkpNPGKGe9/hvr7cTFJeA9H0GW/9JnjLIhhb7gJ3BT5mM/zVgzevRPI9IOV3xjyOhwgllaj92lWA1LOA9Bs8oOgAJ88KBYGAghUr7Ou1ExExMwNEChH9iZnxnZd2b4O0MGPJSl3zeL4k/wqIigDk1sqcn8OhFwFcZEAQgZlHgWwCF4Bw8Bd7I1WceObgTIHTKDHa9xwfmGWYpurOK24HAFERgA0Ag8O2Z3KOWthvOIXKuBPHYSDHRbgSd2YCAAHsq4K6rh42JmDJCUhExOTOP50KJhpWTy3yZDvlPQPWIsu2keWCyeNGNAGOTXAVZJP3k+rCJ01Htqz5oPe4rkPRddx4omE0HYT6egVVAFDlEJEj1CyekaepDa5CBYZkBuDK5ISBedNylbfPDiovAjg+uRsqMDGAEQhgrGoCJO0BQ4DBYEb64z5pkoEBw4HiYBgAZndMvA4ymcjLyu4YNOT9ACBsi6XI/CLJNnHYIGFbzlcAsCIwsRq4nv0Phi4X5d/okEsAAAAASUVORK5CYII=</Image>
> +<Url type="text/html" method="GET" template="https://auctions.yahoo.co.jp/search/search" resultdomain="yahoo.co.jp">
> +  <Param name="p" value="{searchTerms}"/>
> +  <Param name="ei" value="EUC-JP"/>
> +  <Param name="fr" value="mozff" />

This looks like a partner code.

::: mail/components/search/searchplugins/yahoo-jp.xml
@@ +9,5 @@
> +<Image width="16" height="16">data:image/x-icon;base64,AAABAAIAEBAAAAEAIABoBAAAJgAAACAgAAABACAAqBAAAI4EAAAoAAAAEAAAACAAAAABACAAAAAAAAAEAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADMA/24zAP//MwD//zMA//8zAP//MwD//wAAAAAAAAAAMwD/7TMA/+MAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADkI//8zAP//MwD/bwAAAAAAAAAAAAAAADEA/zQyAP8pNwD/FwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA5CP//MwD//zIA/3AAAAAAAAAAAAAAAAAAAAAAMgD/wDIA/6cAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA0AP8sMwD/9zMA//8zAP//NwD/FwAAAAAAAAAAAAAAADQA/3EzAP//MQD/PgAAAAAAAAAAAAAAAAAAAAA3AP8OMwD/2DMA//80AP9PMwD/WjMA//8xAP8qAAAAAAAAAAAyAP8kMwD//zMA/9EAAAAIAAAAAAAAAAAAAAADMwD/tDMA//8yAP+JAAAAAAAAAAAxAP85MwD//zQA/1gAAAAAAAAAADMA/9UzAP//MwD/qQAAAAAAAAAAMgD/kzMA//8zAP/EAAAABAAAAAAAAAAAMwD//zMA//8zAP//MwD//zMA/30xAP8+MwD/+TEA/z4yAP+sMwD//zMA//8zAP//MwD//zMA/5oAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP//AAD//wAA//8AAP//AADgZwAA+OMAAPjzAADwcQAA4DAAAMMYAADDAAAAA/8AAP//AAD//wAA//8AAP//AAAoAAAAIAAAAEAAAAABACAAAAAAAAAQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMwD//zMA//8zAP//MwD//zMA//8zAP//MwD//zMA//8zAP//MwD//zMA//8zAP//AAAAAAAAAAAAAAAAMQD/HzQA/2MyAP+iAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAyAP+dMwD/9jMA//8zAP//MwD//zMA//8zAP//MwD//zMA//8zAP//MwD//zMA/3gAAAAAAAAAAAAAAAAzAP+/MwD//zMA//80AP9UAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAANAD/JzQA/4ozAP//MwD//zMA//8zAP/nMwD/NzQA/ycxAP8VAAAABAAAAAAAAAAAAAAAADMA/1AzAP//MwD/zjMA/18AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADMA//8zAP//MwD//zMA/98AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAjAA/yUAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMwD//zMA//8zAP//MwD/3wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADUA/yIzAP9zMwD/UAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAzAP//MwD//zMA//8zAP/fAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAANAD/njMA//8zAP/pMwD/DwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAANAD/TzMA//8zAP//MwD//zMA/+AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAzAP9QMwD//zMA//80AP+PAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAg0AP+UMwD//zMA//8zAP//MwD//jIA/1IAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAC4A/wszAP/2MwD//zMA//sxAP8qAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMwD/lzMA//8zAP//MwD//zMA/84zAP/5MwD/+TIA/1EAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADMA/7MzAP//MwD//zQA/7wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADIA/1szAP//MwD//zMA//8zAP/fOQD/EjMA/0EzAP/tMwD//DQA/2wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMgD/ZTMA//8zAP//MwD//zIA/1IAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAxAP8vMwD/9TMA//8zAP//MwD/+DEA/zQAAAAAAAAAADUA/yIzAP/UMwD//zMA/5YAAAAGAAAAAAAAAAAAAAAAAAAAAAAAAAAzAP8ZMwD//TMA//8zAP//MwD/3wAAAAkAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAALgD/FjMA/98zAP//MwD//zMA//8yAP9mAAAAAAAAAAAAAAAAAAAAADMA/w8zAP+4MwD//zMA/8MzAP8ZAAAAAAAAAAAAAAAAAAAAAAAAAAAzAP/IMwD//zMA//8zAP//MgD/fwAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAYzAP/DMwD//zMA//8zAP//MgD/ogAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAYzAP+zMwD//zMA/+k0AP9JAAAAAAAAAAAAAAAAAAAAADMA/3kzAP//MwD//zMA//8zAP/2MAD/IAAAAAAAAAAAAAAAAAAAAAAAAAABMwD/pDMA//8zAP//MwD//zMA/9YuAP8LAAAAAAAAAAAAAAAAAAAAAAAAAAA1AP8/NAD/YjMA/5ozAP//MwD//zMA//8zAP+/MwD/gzIA/00wAP8QNQD/KzMA//8zAP//MwD//zMA//8zAP+zAAAAAAAAAAAAAAAAAAAAADMA/40zAP//MwD//zMA//8zAP/3MwD/LQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAADMA/98zAP/fMwD/3zMA/98zAP/fMwD/3zMA/98zAP/fMwD/3zMA/58AAAAAMwD/3TMA//8zAP/fMwD/kTEA/0MzAP88MgD/YDMA/4IzAP+6MwD//zMA//8zAP//MwD//zMA/7oyAP9HNAD/NjAA/yUAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA0AP9FMgD/LgAAAAAAAAAAAAAAADMA//YzAP//MwD//zMA//8zAP//MwD//zMA//8zAP//MwD//zMA//8zAP//MwD//wAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAP/////////////////////////////////////+ABw//gAcP/+AHD//4fz//+H/H//h/w//wf8P/4D/B/+Af4f/AD+D/gYPgfwPB8H4H4PA8B8AAPA/ACAAD//nAA//////////////////////////////////////////////</Image>
> +<Url type="text/html" method="GET" template="https://search.yahoo.co.jp/search" resultdomain="yahoo.co.jp">
> +  <Param name="p" value="{searchTerms}"/>
> +  <Param name="ei" value="UTF-8"/>
> +  <Param name="fr" value="mozff" />

Same here.

::: mail/components/search/searchplugins/yandex-az.xml
@@ +10,5 @@
> +<Url type="application/x-suggestions+json" method="GET" template="https://yandex.az/suggest/suggest-ff.cgi">
> +  <Param name="part" value="{searchTerms}"/>
> +</Url>
> +<Url type="text/html" method="GET" template="https://yandex.az/search" resultdomain="yandex.az">
> +  <MozParam name="clid" condition="purpose" purpose="searchbar"   value="2186618"/>

The clid parameter looks like partner codes.

Comment 21

[Jorg K (CEST = GMT+2)]

(In reply to Florian Quèze [:florian] (PTO until Jan 2) from comment #20)
> IIRC Thunderbird had a search deal with Bing 8 years ago. Does
> this deal still exist?
Good question. Same goes for whether there's a deal with Box.com (FileLink provider).

Comment 22

[Mike Kaply [:mkaply]]

For Thunderbird, you should use just a generic google.xml with no codes at all (not our renames of google-b-d, etc.)

And to Florian's point, yes those are all partner codes that should be removed unless Thunderbird has separate agreements in place (I'll confirm this).

Comment 23

[Mike Kaply [:mkaply]]

Comment on attachment 9033912 [details] [diff] [review]
1427133-search-esr60.patch (v2)

r+ with this section removed:

+    <MozParam name="t" condition="purpose" purpose="contextmenu" value="ffcm"/>
+    <MozParam name="t" condition="purpose" purpose="keyword"     value="ffab"/>
+    <MozParam name="t" condition="purpose" purpose="searchbar"   value="ffsb"/>
+    <MozParam name="t" condition="purpose" purpose="homepage"    value="ffhp"/>
+    <MozParam name="t" condition="purpose" purpose="newtab"      value="ffnt"/>

These are the Firefox specific codes.

Comment 24

[Jorg K (CEST = GMT+2)]

(In reply to Mike Kaply [:mkaply] from comment #22)
> For Thunderbird, you should use just a generic google.xml with no codes at
> all (not our renames of google-b-d, etc.)
> 
> And to Florian's point, yes those are all partner codes that should be
> removed unless Thunderbird has separate agreements in place (I'll confirm
> this).
OK, I''l revise that. I thought, we're still called "Mozilla Thunderbird", so if TB creates some search traffic, no one should be upset ;-)

Comment 25

[Jorg K (CEST = GMT+2)]

Attachment: 1427133-search-esr60.patch (v2b)

Great, we've got TB 60 covered :-) - I've removed the partner codes and some trailing spaces.

Comment 26

[Jorg K (CEST = GMT+2)]

Notes for TB 60 ESR: Try build here:
https://treeherder.mozilla.org/#/jobs?repo=try-comm-central&revision=c692499bf776276db5ac7f1743e7856f02b9aa5b
Working nicely, we'll follow-up with getting the icons right in bug 1512067 (uplift required) and bug 1517352.

Comment 27

[Jorg K (CEST = GMT+2)]

TB 60.5 ESR:
https://hg.mozilla.org/releases/comm-esr60/rev/ebeea939a6e421d36d047a222b9d968cdd885bac

Mike, I've done the unorthodox thing to land the ESR 60 patch first. I hope we can make progress on the trunk patch soon. Well, I guess you want me to remove all the partner codes from the patch first, right? I'll get to this later today.

Comment 28

[Jorg K (CEST = GMT+2)]

Attachment: 1427133-search.patch (v2)

As per Florian's comments from comment #20:

- Only using one "google" now, removed MozParam name="client".
- removed partner codes as well as I could:
  + sed -i '/<Param name="tag"/d' *.xml - Amazon
  + sed -i '/<Param name="tn"/d' baidu.xml
  + sed -i '/<Param name="pc"/d' bing.xml
  + sed -i '/<Param name="campid"/d' ebay*.xml
  + sed -i '/<Param name="partner"/d' twitter*.xml
  + sed -i '/<Param name="fr"/d' yahoo*.xml
  + sed -i '/<MozParam name="clid"/d' yandex*.xml
  + sed -i '/<Param name="client"/d' marktplaats*.xml
  + sed -i '/<Param name="client"/d' qwant.xml, or should <Param name="client" value="opensearch"/> have stayed?

Left these:
<Param name="sourceid" value="Mozilla-search"/> which is in Amazon, Wikipedia, etc.
bing.xml:        <MozParam name="form" condition="purpose" purpose="contextmenu" value="MOZCON"/>
bing.xml:        <MozParam name="form" condition="purpose" purpose="searchbar" value="MOZSBR"/>
bing.xml:        <MozParam name="form" condition="purpose" purpose="homepage" value="MOZSPG"/>
bing.xml:        <MozParam name="form" condition="purpose" purpose="keyword" value="MOZLBR"/>
bing.xml:        <MozParam name="form" condition="purpose" purpose="newtab" value="MOZTSB"/>

Please give further instructions.

Comment 29

[GavinBrelstaff]

Are these patches going to land in TB any time soon?

Comment 30

[Jorg K (CEST = GMT+2)]

Yes, already landed on the TB 60 ESR branch and will be released in TB 60.5 at the end of the month. Do you want an unofficial TB 60.4.1? For which platform?

Comment 31

[GavinBrelstaff]

I am ready to wait - but Great news Jorg. Thanks!

Comment 32

[echosa]

I just downloaded Thunderbird to see how it's doing these days after reading that there have been new devs hired. I want to experience it now, so I can watch it improve. :-) That said, I was also trying to get DDG search working. I eventually got to this ticket, which is great to see this is being worked on.

However, I'm curious: is it even necessary to specify a search engine in Thunderbird? It would be awesome if there were an option to just "Use my browser's default search engine", and then I only have to set the search engine in one place (Firefox, in my case). Perhaps there are technical constraints that don't make this feasible, though.

Thanks for all the great, hard work!

Comment 33

[Mike Kaply [:mkaply]]

Comment on attachment 9034615 [details] [diff] [review]
1427133-search.patch (v2)

Review of attachment 9034615 [details] [diff] [review]:
-----------------------------------------------------------------

Remove this section from Bing:

        <MozParam name="form" condition="purpose" purpose="contextmenu" value="MOZCON"/>
        <MozParam name="form" condition="purpose" purpose="searchbar" value="MOZSBR"/>
        <MozParam name="form" condition="purpose" purpose="homepage" value="MOZSPG"/>
        <MozParam name="form" condition="purpose" purpose="keyword" value="MOZLBR"/>
        <MozParam name="form" condition="purpose" purpose="newtab" value="MOZTSB"/>

You should remove the eBay engines completely. Those URLs only work as affiliate URLs, so they won't work at all with any changes.

Otherwise, this looks good

Comment 34

[Jorg K (CEST = GMT+2)]

Attachment: 1427133-search.patch (v3)

Thanks, Mike. I've removed the said lines from Bing and I removed all eBay engines.

Comment 35

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/1337b98ae249
Fix search engines by copying browser/components/search/searchplugins and removing partner codes. r=mkaply

Comment 36

[Jorg K (CEST = GMT+2)]

TB 65 beta 3:
https://hg.mozilla.org/releases/comm-beta/rev/0fdc3179087f0d2982335da5667547c000488bf1

Comment 37

[Daniel Davis]

And confirmed live in 60.5.0 - thank you very much for this!