662046 - Remove firefox/its-a-trap.html and firefox/its-an-attack.html from UrlClassifier DB

Status: RESOLVED DUPLICATE of bug 971240

(Toolkit :: Safe Browsing, defect)

Description

[Mehdi Mulani [:mmm] (I don't check this)]

The pages http://mozilla.com/firefox/its-a-trap.html and http://mozilla.com/firefox/its-an-attack.html are both found in the UrlClassifier database but pose the compounding problem that:
 - any lookup to a mozilla.com page can not short-circuit and return clean, instead the lookup must hit our full store to check if the URL is bad.
 - if the full store is removed and instead a lookup goes straight to the Google SB servers, its-{a-trap,an-attack}.html will fail at the last step but cause mozilla.com pages to take much longer to load along with hitting Google's servers for a lot of false positives.

This isn't a problem yet because all lookups are done to the "full store", the SQLite database. However, if we move to a smaller data structure for prefix hashes and remove the full store (as we intend to on mobile) then we either suffer a penalty on .mozilla.com pages or we move off its-{a-trap,an-attack}.html.

If these pages are still thought to be useful, we could use a different subdomain (provided that little else is hosted there) and specifically handle that case.

Comment 1

[Mehdi Mulani [:mmm] (I don't check this)]

Didn't explicitly state this but we should notify the webdev people that own this code. Doug, could you ping the correct people?

This will require some code change in UrlClassifierDB, we need to push an update that deletes the corresponding entries. I think this can be done somewhat cleanly but need to look into it.

Comment 2

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

We use these in tests don't we?

Comment 3

[Mehdi Mulani [:mmm] (I don't check this)]

(In reply to comment #2)
> We use these in tests don't we?

Yup. If we switch to different URLs then hopefully they can be easily fixed.
I'm getting the feeling that I could push a fake update for tests but not in the release build, which could get around that problem. (and therefore, allow us to keep the tests pointing to its-{a-trap,an-attack}.html) Not completely sure on that one yet though.

Comment 4

[Gian-Carlo Pascutto [:gcp]]

> - if the full store is removed and instead a lookup goes straight to the Google 
>SB servers, its-{a-trap,an-attack}.html will fail at the last step but cause 
>mozilla.com pages to take much longer to load along with hitting Google's 
>servers for a lot of false positives.

Probing Google's servers for all URI is prohibitive, I don't see that happening. Even if we use a compressed store, we can still store the URL fragment prefixes instead of the domain prefixes, avoiding this issue.

The potential effect of this is mostly that we cannot use the fast path in the *current* url-classifier code.

Comment 5

[Gian-Carlo Pascutto [:gcp]]

There are some more issues related to how these links are currently handled:

- They require us to push a database update while the browser is starting up. There is a performance penalty for doing so. Bug 669410 will make it worse (complete prefixset rebuild), whereas bug 673470 will improve it again (because there are separate DB's per list).

- I believe there may be potential race conditions between pushing this update and pages being loaded. At the very least, I'm not entirely sure anything prevents the browser from loading an URL until the database update adding these entries is processed.

- They exploit an ambiguous part of the specification to work; specifically adding and deleting the same chunk in a single update. Bug 673470 has some code clutter specifically to make this work.

All 3 of these could be fixed by bug 662213 + asking Google to permanently add the new links in their database.
However, that will still break QA. We can't require them to sit around and wait until the entire database gets downloaded (takes a day).

We can't preload the database during the installation, because the exact format of the stores is machine-dependent (endianness etc).

I have no solution here - just wanted to document these issues.

Comment 6

[Anthony Ricaud (:rik)]

Those pages now live at http://www.itisatrap.org/ and http://www.itisatrap.org/its-an-attack.html.

If you want a redirection from the mozilla.org URLs to this domain, please open a new bug.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Not sure if we should really redirect requests, at least not as long as the new domain is not listed in the UrlClassifier database. Also not sure what the story is for Aurora, Beta, Release, and ESR10. Those would also need the patch for the db to continue to work with the page.

Comment 8

[Gian-Carlo Pascutto [:gcp]]

Redirecting would be fine, as IIRC the browser will show the warning page immediately and simply never load the redirect. 

The problem is if we update the docs/webpages to point to the new domain and that one isn't in the databases of the old browsers yet (that was the pain during the .com->.org move).

So I think redirecting first, pushing the DB update and letting it ride, and then updating the docs is proper.

I'll file followups.

Comment 9

[Gian-Carlo Pascutto [:gcp]]

For what it's worth, this bug is still blocked because the layout on itsatrap.org is wrong:
https://bugzilla.mozilla.org/show_bug.cgi?id=686731#c14

Given that today suddenly bug 906304 popped up, I'll file a separate one for that issue so hopefully it gets fixed someday.

Comment 10

[Chris More [:cmore]]

(In reply to Gian-Carlo Pascutto (:gcp) from comment #9)
> For what it's worth, this bug is still blocked because the layout on
> itsatrap.org is wrong:
> https://bugzilla.mozilla.org/show_bug.cgi?id=686731#c14
> 
> Given that today suddenly bug 906304 popped up, I'll file a separate one for
> that issue so hopefully it gets fixed someday.

All of these bugs and their relationships feel like circular logic. I keep reading over the bugs, what is blocking each other in the comments and I keep getting back to this one.

:gcp: Did you see bug 876581 where we want to move all of this content to SUMO? I really just want them off of mozilla.org and whatever it takes to get the itsatrap.org, its-an-attack.html, and the phishing-protection pages moved to somewhere other than mozilla.org. SUMO seems like the best solution.

Comment 11

[Ben (:bensternthal)]

Resolving as dupe to clean up.

The work for this is stated in 971240