1367276 - Only load Safebrowsing db files when both metadata file and prefix set file exist

Status: RESOLVED WORKSFORME

(Toolkit :: Safe Browsing, enhancement, P3)

Description

[Dimi Lee [:dimi]]

I am not sure what would happen if there is only metadata file or prefix set file for a table.

We need to figure out the current behavior and maybe not load data if not both metadata and prefix set file exist.

Note.
Metadata file are files with .sbstore extension(V2) or files with .metadata extension(V4).
Prefix set file are files with .pset for both V2 and V4.

Comment 1

[Dimi Lee [:dimi]]

Some note.

1.[V4]Load PrefixSet (.pset) will also load Metadata file (.metadata) to verify checksum. Reset when checksum mismatch.
2.[V4]Load Metadata (.metadata) will also load .pset because of code flow.

3.[V2]Load PrefixSet (.pset) will not load HashStore (.sbstore)
  - .pset is faster to load because it's simpler data structure, we don't have to load .sbstore when
    we don't need to update
4.[V2]Load HashStore (.sbstore) will not load .pset. But we will verify .pset and .sbstore before applying update.

Comment 2

[François Marier [:francois]]

(In reply to Dimi Lee[:dimi][:dlee] from comment #1)
> 1.[V4]Load PrefixSet (.pset) will also load Metadata file (.metadata) to
> verify checksum. Reset when checksum mismatch.

If .metadata is missing then the checksum won't match. Nothing to fix.

> 2.[V4]Load Metadata (.metadata) will also load .pset because of code flow.

We will never attempt to load .metadata first, so this case cannot cause any issues.

Comment 3

[François Marier [:francois]]

(In reply to Dimi Lee[:dimi][:dlee] from comment #1)
> 3.[V2]Load PrefixSet (.pset) will not load HashStore (.sbstore)
>   - .pset is faster to load because it's simpler data structure, we don't
> have to load .sbstore when
>     we don't need to update

If .pset is on disk but .sbstore is missing, we will still be able to load the prefix set and do lookups but when it comes time to do an update, we will end up doing an update as if we had no initial data at all. When we get the update data, it will fail to apply. The failure to update will cause a full reset.

> 4.[V2]Load HashStore (.sbstore) will not load .pset. But we will verify
> .pset and .sbstore before applying update.

Again, the update will fail to apply and that will cause a full reset. The update after that will download a fresh DB and everything will be fine.

Comment 4

[François Marier [:francois]]

These edge cases will cause update failures, but the DB will be reset and so eventually users will get a working DB.

This is working as expected.