1284178 - Implement HashStore for v4

Status: RESOLVED WONTFIX

(Toolkit :: Safe Browsing, defect, P2)

Description

[Dimi Lee [:dimi]]

V4 is not chunk based data anymore, we will need to implement v4 format

Comment 1

[Dimi Lee [:dimi]]

Partial update response may contain removal "index" for prefixes. Prefixes should be sorted by their hash values in binary string format in order to find correct index.

So the plan here is to store all the prefixes in database with sorted order.

Comment 2

[Dimi Lee [:dimi]]

Another approach is do not store .sbstore for v4 since we will have prefixes in .cache(completions from update), .pset(4-bytes prefix) and .vlpset(variable-length prefix set, bug 1283007).

But in this way we would need to merge all the prefixes from these files and then sort again, also .pset files store prefix and delta. So construct prefixes string from .pset requires extra effort.

Comment 3

[Dimi Lee [:dimi]]

Hi gcp,
Have one question regarding to v2 implementation. In v2 we will store addPrefixes in both .pset and .sbstore, why do we do that? can we just store them in .pset ?

Since v4 is not chunk based anymore, there is only prefixes and completions, so I am thinking keep 4-bytes prefixes in .pset like what we are doing right now and put others(server state, variable-length prefixes, completions) in .sbstore. how do you think ?

Comment 4

[Gian-Carlo Pascutto [:gcp]]

>Have one question regarding to v2 implementation. In v2 we will store addPrefixes in 
>both .pset and .sbstore, why do we do that? can we just store them in .pset ?

First of all, the data isn't duplicated: the actual prefixes are (only) in .pset, and the chunk data from those add prefixes is (only) in .sbtore. The chunk data is only needed for updates, not for lookups. So we don't have to open and read HashStore unless we update. That said, the real reason is that PrefixSet existed and was used before HashStore was written, so HashStore just carries the missing information that PrefixSet didn't carry yet (chunks). That avoided an incompatible change in the .pset format.

For v4 this all no longer applies and you can put all prefix data in .pset.

Comment 5

[Dimi Lee [:dimi]]

(In reply to Gian-Carlo Pascutto [:gcp] from comment #4)
> so HashStore
> just carries the missing information that PrefixSet didn't carry yet
> (chunks). That avoided an incompatible change in the .pset format.
> 
> For v4 this all no longer applies and you can put all prefix data in .pset.

Thank you for the clarification!

Comment 6

[Dimi Lee [:dimi]]

According to Comment 4, we will store all prefix data in .pset(Implement in Bug 1283007)
So we won't need HashStore for v4.

Set this bug as won't fix.