633644 - nsUrlClassifierDBServiceWorker::GetLookupFragments returns duplicate fragments in some cases

Status: RESOLVED DUPLICATE of bug 665930

(Toolkit :: Safe Browsing, defect)

Description

[Mehdi Mulani [:mmm] (I don't check this)]

It seems that GetLookupFragments will create duplicate fragments when the path ends in a "/". The following messages show up with debug output turned on:

GetLookupFragments: called with spec mozilla.com/en-US/firefox/
669188096[124f96790]: Chking mozilla.com/en-US/firefox/
669188096[124f96790]: Chking mozilla.com/
669188096[124f96790]: Chking mozilla.com/en-US/
669188096[124f96790]: Chking mozilla.com/en-US/firefox/

Similarly, when called on a spec like "mozilla.com/", two identical fragments are returned.

Comment 1

[Mehdi Mulani [:mmm] (I don't check this)]

CC'ing Dave Camp as it looks like he has worked with this.

Comment 2

[Mehdi Mulani [:mmm] (I don't check this)]

Part of the problem seems to be here:
http://hg.mozilla.org/mozilla-central/annotate/1ed3464aaa92/toolkit/components/url-classifier/src/nsUrlClassifierDBService.cpp#l1498
along with line 1509.

Comment 3

[Mehdi Mulani [:mmm] (I don't check this)]

It looks like this is causing a problem with our HashCompleter requests.

Based on calls to the HashCompleter and requests seen with the HttpFox extension it looks like the following happens when you visit a "bad" site which ends in a slash. I'll use malware.testing.google.test/testing/malware/ as an example.

1. Type malware.testing.google.test/testing/malware/ into url bar.
2. testing.google.test is determined to be a bad domain but the complete hash of the bad URL is needed
3. SHA-256 hash is taken of "malware.testing.google.test/testing/malware/", the hex prefix ("51864045")
4. A few random entries from urlclassifier3.sqlite are picked as noise.
5. nsUrlClassifierHashCompleter::Complete is called for "51864045" twice and once for all the noise entries.
6. A request is made to Google's SB servers and in the request header we can see the "51864045" entry twice.

As I understand it, we add noise to obscure which site the user is actually making a request for. If so, this case renders the noise requirement useless.