1477059 - Built-in topdomains.txt contains shady terraclicks.com domain

Status: VERIFIED FIXED

(Firefox for Android Graveyard :: General, defect)

Description

[Villa]

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:61.0) Gecko/20100101 Firefox/61.0
Build ID: 20180704192850

Steps to reproduce:

It is about Firefox Focus Mobile Browser - Android
there is no Bugreport section for that product or i didnt find it

the Autocomplete function of the URL advertises me 
by tping terra - terraclicks.com
that website is opening 403 - Forbidden

by searching for it the first results displaying Spyware


Actual results:

Autocomplete terraclicks.com
why is this website autocompleting? who is selecting this or checking it?


Expected results:

dont autocomplete rubbish URL

Comment 1

[Darkspirit]

Hi, thanks for the report!
I can also reproduce with Firefox 61 on Android and Firefox Nightly 63 on Android. (Tested with: Extras > Guest Session)

https://searchfox.org/mozilla-central/rev/51268dcbdff0f6f4a5cff7986df0f616efc5bcfd/mobile/android/app/src/main/res/raw/topdomains.txt#389
It was added by bug 858829.

Btw, uBlock Origin blocks http://terraclicks.com.
Filter: Adguard Mobile Filters. https://github.com/AdguardTeam/AdguardFilters/blob/fbf2b6012101b44f356dc5bfd3d8148ad1d6beb8/MobileFilter/sections/adservers.txt#L843

Comment 2

[Darkspirit]

[Tracking Requested - why for this release]:
Firefox 62 ships next month. It would be good to remove such a shady domain from topdomains.txt and rather add it to a blocklist. https://www.google.com/search?q=terraclicks.com&ie=utf-8&oe=utf-8&client=firefox-b

Comment 3

[Liz Henry (:lizzard) (relman/hg->git project)]

This is showing as 403 Forbidden for me and the domain also shows on lists of malware/virus related activity. Seems reasonable to remove it from topdomains to me. Michael what do you think?

Comment 4

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

Seems reasonable. I wonder if we should regenerate this autocomplete list – maybe we can borrow it from Focus Android?

NI Susheel to decide ^ and get someone to fix this.

Comment 5

[Kevin Brosnan [Ex-Mozilla]]

I tried to argue for a shorter list when this was first implemented 1k was too ambitious.

Comment 6

[Villa]

nobody gone through the URI list? i think this is something a human brain is helpful for decision if a webpage deserves to be in autocomplete of mozilla product. BUT there can probably be a phyton script thats comparing search-results uf the URI's with a dictionary of words we dont want to have in the first 5 searchresults: e.g. Spware Advertising Virus Malware [...] result of the script could be checked again by human eyes. plus: websites are known for changing their content - therefore such checks could take plave every now and than

Comment 7

[Kevin Brosnan [Ex-Mozilla]]

Yes humans went through the domain list bug 858829. Humans are failable with a list this long.

Comment 8

[:sdaswani]

Punting the decision to work on this to Product.

Comment 9

[Villa]

just noticed this has been reported already: 
https://bugzilla.mozilla.org/show_bug.cgi?id=1372968

Comment 10

[Ryan VanderMeulen [:RyanVM]]

We're building RCs for Fennec 62 next week. Any news on this?

Comment 11

[Barbara Bermes [:barbara] - NI please!]

Let's update all products using the top-domain list to accommodate for this change.

Based on our sanitizing methodology, it's ok to take out that website: https://firefox-source-docs.mozilla.org/mobile/android/fennec/defaultdomains.html

I'll create an issue for Focus iOS and Focus Android, and I'll let the team here in the bug go ahead with Fennec.

Comment 12

[Ryan VanderMeulen [:RyanVM]]

Susheel, is there enough time to regenerate this list by Wednesday?

Comment 13

[Villa]

please consider also to remove:

- Remove any site that routinely publish fake or misleading content.
regnok.com

- Remove any site that degrades [...] geographic location
pch.com

- Remove any sites that fail to load in mobile browsers. 
ijreview.com
buzzlie.com

- Remove pure search engines. [...]
myway.com
wow.com

Comment 14

[:sdaswani]

Vlad, are you aware of how to regenerate this list by tomorrow?

Comment 15

[Petru-Mugurel Lingurar [:petru]]

Attachment: Bug 1477059 - Remove terraclicks.com from topdomains; r=sdaswani

Comment 16

[Petru-Mugurel Lingurar [:petru]]

I've removed terraclicks.com from topdomains.

(In reply to Villa from comment #13)
> please consider also to remove:
> 
> - Remove any site that routinely publish fake or misleading content.
> regnok.com
> 
> - Remove any site that degrades [...] geographic location
> pch.com
> 
> - Remove any sites that fail to load in mobile browsers. 
> ijreview.com
> buzzlie.com
> 
> - Remove pure search engines. [...]
> myway.com
> wow.com

A more thorough sanitization is to be decided by Product (with Legal), a ticket for this being already filed - bug 1372968.

Comment 17

[Ryan VanderMeulen [:RyanVM]]

Hi Susheel, same as bug 1486200 :)

Comment 18

[:sdaswani]

Comment on attachment 9004810 [details]
Bug 1477059 - Remove terraclicks.com from topdomains; r=sdaswani

:sdaswani only needinfo has approved the revision.

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/acbc1d6c38c5
Remove terraclicks.com from topdomains; r=sdaswani

Comment 20

[:sdaswani]

Ryan, I'm sorry - I don't understand the NI for me here?

Comment 21

[Ryan VanderMeulen [:RyanVM]]

Nothing at this point, just wanted to make sure the review request was on your radar (which it obvious was!) :)

Comment 22

[Kevin Brosnan [Ex-Mozilla]]

(In reply to Petru-Mugurel Lingurar[:petru] from comment #16)
> I've removed terraclicks.com from topdomains.
> 
> (In reply to Villa from comment #13)
> > please consider also to remove:
> 
> A more thorough sanitization is to be decided by Product (with Legal), a
> ticket for this being already filed - bug 1372968.

That issue is for Focus iOS. Each program that has this list has a separate copy and they are not kept in sync.

Comment 23

[Arthur Iakab [arthur_iakab]]

https://hg.mozilla.org/mozilla-central/rev/acbc1d6c38c5

Comment 24

[Oana Horvath [:ohorvath]]

Verified as fixed on Nightly 63 (2018-08-30)
Devices:
Google Pixel (Android 9)
Sony Xperia Z5 Premium (Android 6.0.1)

Comment 25

[Petru-Mugurel Lingurar [:petru]]

Comment on attachment 9004810 [details]
Bug 1477059 - Remove terraclicks.com from topdomains; r=sdaswani

Approval Request Comment
[Feature/Bug causing the regression]: Bug 858829
[User impact if declined]: Site suggested is not available anymore. Error 403
[Is this code covered by automated tests?]: No
[Has the fix been verified in Nightly?]: Yes
[Needs manual test from QE? If yes, steps to reproduce]: --
[List of other uplifts needed for the feature/fix]: --
[Is the change risky?]: No
[Why is the change risky/not risky?]: Small change, QA tested
[String changes made/needed]: --

Comment 26

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9004810 [details]
Bug 1477059 - Remove terraclicks.com from topdomains; r=sdaswani

Approved for Fx62 RC2.

Comment 27

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-release/rev/ca9bd27bb10b

Comment 28

[Oana Horvath [:ohorvath]]

Verified as fixed on RC 62.0 build 2.
Device: Sony Xperia Z5 Premium (Android 6.0.1)