1147462 - Clone safebrowsing list into second list to test client-server multiple-list synchronization

Status: RESOLVED FIXED

(Cloud Services :: Server: Shavar, defect)

Description

[Toby Elliott [:telliott]]

We need to test that the safebrowsing client and server are correctly synchronizing two unrelated lists. Please set up a second copy of the safebrowsing list at the current endpoint with a different name.

Comment 1

[[:mmc] Monica Chew (no longer reading bugmail)]

Steps to end-to-end test:

0) Create a new profile
1) Point Firefox pref browser.trackingprotection.updateURL to the endpoint, 
2) Point browser.trackingprotection.gethashURL to the endpoint
3) Set urlclassifier.trackingTable to "mozpub-track-digest256,<new_table_name>"
4) Restart Firefox
5) Enable privacy.trackingprotection.enabled
6) Check for the presence of {mozpub-track-digest,new_table_name}.{cache,pset,sbstore} in the local profile.  There should also be the standard Google tables goog-{phish,malware}-shavar.*. On Linux the profile dir is ~/.cache/mozilla/firefox/<profile>/safebrowsing. On Mac that's somewhere in ~/Library/.

Comment 2

[Chris Kolosiwsky [:ckolos] (ckolos has left the building)]

This can be a stage environment deployment, correct?

Comment 3

[[:mmc] Monica Chew (no longer reading bugmail)]

(In reply to Chris Kolosiwsky [:ckolos] from comment #2)
> This can be a stage environment deployment, correct?

Yes.

Comment 4

[Ryan Tilder [:rtilder]]

For testing, we don't actually need a second data source.  We can simply reuse the data for the existing list.  You should be able to simply create another section in the INI file for the service and point that  at the same data file used for the existing list being served.  Something along the lines of:

[new_table_name]
type = digest256
source = <path from the mozpub-track-digest256 section, i.e. s3file://s3_bucket_name/data_file_key_name>

Comment 5

[Chris Kolosiwsky [:ckolos] (ckolos has left the building)]

I'll ping :rpapa and see if I can make the required server config changes on the .6 single instance he has for testing purposes

Comment 6

[Richard Pappalardo [:rpapa][:rpappalardo]]

Yea, let's do it.  thanks, Chris

Comment 7

[Richard Pappalardo [:rpapa][:rpappalardo]]

(In reply to Ryan Tilder [:rtilder] from comment #4)
> For testing, we don't actually need a second data source.  We can simply
> reuse the data for the existing list.  You should be able to simply create
> another section in the INI file for the service and point that  at the same
> data file used for the existing list being served.  Something along the
> lines of:
> 
> [new_table_name]
> type = digest256
> source = <path from the mozpub-track-digest256 section, i.e.
> s3file://s3_bucket_name/data_file_key_name>

Ryan, I see that Monica provided the steps for e2e verification, but for server load-testing do we need any additional tests (other than the one you provided before)?

Comment 8

[Ryan Tilder [:rtilder]]

(In reply to Richard Pappalardo [:rpapa][:rpappalardo] from comment #7)
> 
> Ryan, I see that Monica provided the steps for e2e verification, but for
> server load-testing do we need any additional tests (other than the one you
> provided before)?
>

In the long run?  Yes.  But for now the hacky curl command line I provided should be sufficient.

Comment 9

[Chris Kolosiwsky [:ckolos] (ckolos has left the building)]

I've made the following changes to the shavar.ini and restarted the server 

--- shavar.ini	2015-04-03 15:46:33.967002664 +0000
+++ shavar.new	2015-04-03 15:36:03.542523098 +0000
@@ -47,6 +47,7 @@
 [shavar]
 default_proto_ver = 2.0
 lists_served = mozpub-track-digest256
+               foo-track-digest256
                moz-abp-shavar
 lists_root = tests
 refresh_check_interval = 3600
@@ -57,6 +58,11 @@
 not_publishing_deltas = True
 source = s3+file://net-mozaws-stage-shavar/list//mozpub-track-digest256

+[foo-track-digest256]
+type = digest256
+not_publishing_deltas = True
+source = s3+file://net-mozaws-stage-shavar/list//foo-track-digest256
+
 [moz-abp-shavar]
 type = shavar
 source = s3+file://net-mozaws-stage-shavar/list//mozpub-track-digest256