1314094 - Add Shavar whitelist and blacklist for Flash Blocking

Status: VERIFIED FIXED

(Cloud Services :: Server: Shavar, defect)

Description

[Robin Steuber (she/her) [:bytesized]]

We would like to use Shavar to update the whitelist and blacklist for Flash Blocking.

Comment 1

[Robin Steuber (she/her) [:bytesized]]

ckolos - I have been told that you are the person who will probably make the server side changes. What do I need to do to get these lists updated via Shavar?

Comment 2

[Chris Kolosiwsky [:ckolos] (ckolos has left the building)]

:bytesized - Is this a new list or is it the same list as  https://github.com/mozilla-services/shavar-plugin-blocklist ?

Comment 3

[David Durst [:ddurst]]

It looks like our lists would supersede (or at least in part overlap) this -- I wasn't aware that so much of this was Flash-related. CC Chris Peterson to make the call on whether the Flash data in Shavar should be moved to our list.

Comment 4

[Chris Peterson [:cpeterson]]

I think the new lists would supersede or build on the current blocklist [1], which is currently only used for blocking Flash pixels (at the HTTP request level). We can rename the existing list and its scope to cover some of the new use cases. It might be convenient to manage these new lists in the existing shavar-plugin-blocklist GitHub repo because the Shavar service is already configured to watch that repo for new commits to the pixel blocklist.

Benjamin's descriptions of the proposed lists and new use cases:

- specific SWF URLs to never load (already covered by the existing blocklist [1])
- specific domains and *.domain wildcards where we should not ever load Flash (pretend Flash is not present)
- specific domains and *.domain wildcards where we should not load Flash in a 3rd-party context (pretend Flash is not present)
- specific domains where we should activate Flash by default (e.g. whitelist Facebook games)


[1] https://github.com/mozilla-services/shavar-plugin-blocklist

Comment 5

[Luke Crouch [:groovecoder]]

Adding :francois for more engineering input ...

Renaming/re-scoping existing lists can be trouble-some because we have to support older Firefox versions that are still out there using the older lists.

Since shavar simply serves lists, it sounds like we're talking about 4 separate lists served by shavar? I.e.,

* mozplugin-block-digest256 (urls)
* mozplugin-block-domains-digest256
* mozplugin-block-3rdparty-domains-digest256
* mozplugin-allow-domains-digest256

:francois - do I have this more-or-less right?

Comment 6

[François Marier [:francois]]

(In reply to Luke Crouch [:groovecoder] from comment #5)
> Since shavar simply serves lists, it sounds like we're talking about 4
> separate lists served by shavar? I.e.,
> 
> * mozplugin-block-digest256 (urls)
> * mozplugin-block-domains-digest256
> * mozplugin-block-3rdparty-domains-digest256
> * mozplugin-allow-domains-digest256

List names have a specific format:

  arbitraryname-listtype-listencoding

It sounds like we're talking about lists that cover only flash resources, so we'd need to define a new list type. Maybe "flash"?

(In reply to Chris Peterson [:cpeterson] from comment #4)
> - specific SWF URLs to never load (already covered by the existing blocklist
> [1])

I assume there's nothing to do here since that list is already shipping.

> - specific domains and *.domain wildcards where we should not ever load
> Flash (pretend Flash is not present)
> - specific domains and *.domain wildcards where we should not load Flash in
> a 3rd-party context (pretend Flash is not present)

These could be:

  firstpartyblock-flash-digest256
  thirdpartyblock-flash-digest256

> - specific domains where we should activate Flash by default (e.g. whitelist
> Facebook games)

I assume this is for both first-party and third-party contexts? So perhaps:

  allow-flash-digest256

What the expected order of magnitude (in # of urls) for these lists?

Comment 7

[Chris Peterson [:cpeterson]]

(In reply to François Marier [:francois] from comment #6)
> List names have a specific format:
> 
>   arbitraryname-listtype-listencoding
> 
> It sounds like we're talking about lists that cover only flash resources, so
> we'd need to define a new list type. Maybe "flash"?

Perhaps, but these lists have different semantics: blocking (in different contexts) or allowing. Maybe I don't understand the difference between arbitraryname and listtype. I think encoding the lists in the arbitraryname prefix makes the most sense.

Also, I think we can merge the current mozplugin-block-digest256 list into the "block in third-party context" list because the tracking pixels are only used in third-party contexts.

So perhaps we just need three lists like:

flashfirstparty-block-digest256
flashthirdparty-block-digest256 (includes the old pixel blocklist)
flash-allow-digest256


> What the expected order of magnitude (in # of urls) for these lists?

I don't expect any of these lists to have more than hundreds URLs.

Comment 8

[Chris Peterson [:cpeterson]]

I take back what I said about merging the pixel blocklist into the proposed third-party blocklist. The difference between those lists is that the new first-party and third-party "block" lists will actually be used to prevent Flash plugin detection, not just block SWF HTTP requests.

> - specific domains and *.domain wildcards where we should not ever load
> Flash (pretend Flash is not present)
> - specific domains and *.domain wildcards where we should not load Flash in
> a 3rd-party context (pretend Flash is not present)

Also, I'm not sure why we need separate first- and third-party lists. Are the lists blocking mechanisms different? Or do we expect there to be entries on one list that might be acceptable in the other context?

Comment 9

[Luke Crouch [:groovecoder]]

The Safe Browsing/Shavar list format is just a list of hashed urls - there's nothing in the list contents to distinguish which ones are for first- and third-party context urls.

So, AIUI, the only way for code to know which urls to block in first- vs. third-party context is for the urls to be split into separate lists for each purpose.

Comment 10

[Chris Kolosiwsky [:ckolos] (ckolos has left the building)]

:bytesized - what's the timetable for adding these lists to shavar?

Comment 11

[Robin Steuber (she/her) [:bytesized]]

:ckolos - I'm not really sure. :ddurst, can you answer this?

:cpeterson, :groovecoder - When discussing this, we decided that it makes sense to just have one whitelist and one blacklist. They will be used for first- and third-party lists. As :cpeterson touched on, we do not expect there to be entries that are acceptable in only one context. To allow a bit more flexibility, we want the lists to allow domains or full URLs. This will allow a bit more fine-grained control over what exactly is being allowed and blocked if we need it.

Comment 12

[David Durst [:ddurst]]

ckolos: unclear. We want to run a SHIELD experiment (which requires this and other functionality), with an eye towards being able to pref on in 52/53 timeframe, but we aren't yet scheduled. [Read: we are looking to schedule this as soon as we can, but we probably have weeks of coding before we would need production lists.]

Comment 13

[Benjamin Smedberg]

To be clear about timing: we want to land this preffed off into nightly ASAP (maybe even uplift to 51?). So we'd want to have some testing.

Initially the production lists will be empty. The goal of the experiment, once the code is on release or maybe beta, is to iterate weekly on the lists and start adding things to them on a weekly cadence until we've hit the right tradeoffs of frequency and annoyingness.

Comment 14

[François Marier [:francois]]

Currently, the "-block-" list type (used for the pixel blocking that Chris is referring to) is not flash-specific or even plugin-specific. It blocks all network requests on that list.

Since you want different semantics for this new flash blocking (pretending that Flash isn't installed), I think it makes sense to create a new list type just for that.

As for whether to use a single list type or two, the plugin code will be the only user of these lists so the list type could be the same. The plugin code would simply try to match a URL against the whitelist first, than then against the blacklist if the first match failed.

So we could have:

  mozplugin-block-digest256 (no changes to this one)
  disable-flash-digest256 (new blacklist)
  allow-flash-digest256 (new whitelist)

In both cases, the list entries can consist of domains, hostnames or full URLs.

Comment 15

[Chris Peterson [:cpeterson]]

(In reply to Kirk Steuber [:bytesized] from comment #11)
> :cpeterson, :groovecoder - When discussing this, we decided that it makes
> sense to just have one whitelist and one blacklist. They will be used for
> first- and third-party lists. As :cpeterson touched on, we do not expect
> there to be entries that are acceptable in only one context. To allow a bit
> more flexibility, we want the lists to allow domains or full URLs. This will
> allow a bit more fine-grained control over what exactly is being allowed and
> blocked if we need it.

No problem. The Shavar is very flexible and can match domain names (full or domain suffixes), with or without URL paths (full or directory prefixes):

https://developers.google.com/safe-browsing/v4/urls-hashing#canonicalization

Comment 16

[Robin Steuber (she/her) [:bytesized]]

Update: We now seem to have gone up from 2 lists to 6.

I believe that we want a blacklist, a whitelist and a third-party blacklist. Then, for each of those lists, we want a list of exceptions. For context, we are classifying plugins into three groups: Allowed (on whitelist), Denied (on blacklist), Click to Activate (not on either list).

Comment 17

[Chris Kolosiwsky [:ckolos] (ckolos has left the building)]

What's the status on this bug? How close are the lists to being ready for testing?

Comment 18

[Robin Steuber (she/her) [:bytesized]]

I am not sure who is in charge of the contents of the initial lists.

@bsmedberg Do you know the answer to this?

Comment 19

[Benjamin Smedberg]

The expectation is that these lists will start empty and be added to over time. I'm in a discussion to seed the 3rd-party list from disconnect, but that isn't finalized and shouldn't block initial rollout.

What system (repository/PR) is correct for making changes to these lists over time?

Comment 20

[Luke Crouch [:groovecoder]]

We're using shavar-server-list-config [1] to store config file for all the lists shavar should serve. The config file for each list includes a source = s3+file:// directive telling shavar where it can find the source list file.

For example, for the tracking protection lists, we have a couple of Jenkins jobs that:

1. Pull JSON files down from shavar-prod-lists [2]
2. Run lists2safebrowsing.py from shavar-list-creation [3]
3. Upload the resulting list file to the appropriate s3+file:// location specified in shavar-server-list-config

Then shavar can serve the list.

So, you need to:

1. Create an equivalent 1-3 process for these lists
2. Send a pull request to shavar-server-list-config [1] so shavar knows where your process is uploading these lists

[1] https://github.com/mozilla-services/shavar-server-list-config
[2] https://github.com/mozilla-services/shavar-prod-lists
[3] https://github.com/mozilla-services/shavar-list-creation

Comment 21

[François Marier [:francois]]

The upstream lists (currently all empty) are here:

  https://github.com/mozilla-services/shavar-plugin-blocklist

They are not JSON files, they're just text files with one URL per line, just like https://github.com/mozilla-services/shavar-plugin-blocklist/blob/master/mozplugin-block.txt

The lists names will be:

- block-flash-digest256 (flash.txt)
- except-flash-digest256 (flashexceptions.txt)
- allow-flashallow-digest256 (flashallow.txt)
- except-flashallow-digest256 (flashallowexceptions.txt)
- block-flashsubdoc-digest256 (flashsubdoc.txt)
- except-flashsubdoc-digest256 (flashsubdocexceptions.txt)

Luke, are you able to own this and take this config all the way to production?

It will be a good test of the dynamic list config we recently added.

Comment 22

[Luke Crouch [:groovecoder]]

:bsmedberg, :bytesized - can either of you add content to the list files in https://github.com/mozilla-services/shavar-plugin-blocklist/

Comment 23

[Robin Steuber (she/her) [:bytesized]]

I'll let bsmedberg confirm, but he earlier said "The expectation is that these lists will start empty and be added to over time", leading me to believe that there is currently no content to add to the lists.

Comment 24

[Chris Peterson [:cpeterson]]

(In reply to Kirk Steuber [:bytesized] from comment #23)
> I'll let bsmedberg confirm, but he earlier said "The expectation is that
> these lists will start empty and be added to over time", leading me to
> believe that there is currently no content to add to the lists.

The initial lists should have at least one entry so QA can confirm that the client is downloading the lists and blocking Flash correctly.

Comment 25

[Benjamin Smedberg]

If they should be non-empty, I will leave Kirk to figure out what values make sense for QA. Filling out these lists with "real" content (primarily the "flashsubdoc" list for advertising and the "flash" list for topsites) is a separate bug that will happen later.

Comment 26

[Robin Steuber (she/her) [:bytesized]]

@felipe There was some discussion of having "special" entries in order to determine if the lists have been updated. Are we still doing that? Do we know what we want the "special" entries to be?

Comment 27

[Robin Steuber (she/her) [:bytesized]]

Do all the initial lists need to have an entry? Is adding a single entry to a single list sufficient? It sounds like this would be sufficient for Felipe's needs.

Comment 28

[Chris Peterson [:cpeterson]]

(In reply to Kirk Steuber [:bytesized] from comment #27)
> Do all the initial lists need to have an entry? Is adding a single entry to
> a single list sufficient?

One entry in one list is probably adequate for smoke testing the feature, but ideally we would test an entry from each list. A Shavar configuration bug might break or overlook one list but not another. But that is probably unlikely.

Comment 29

[Benjamin Smedberg]

Maybe put it another way: how is Stefan going to test that the lists are working if there isn't at least one artificial entry in each one?

Comment 30

[Robin Steuber (she/her) [:bytesized]]

It sounds to me like everyone's needs will be met by adding a dummy URL to every list, so I will submit a pull request to do that. The test URL I discussed with Felipe, which I will add to every list, is: week0.flashstudy.example.com/

Comment 31

[Benjamin Smedberg]

That isn't a visitable URL, so I'm skeptical of its testing value. ISTR a loadable site we use for testing with various domains/subdomains blocked and not-blocked.

Comment 32

[Robin Steuber (she/her) [:bytesized]]

@bsmedberg My understanding was that the presence of these URLs was to verify the list updating mechanism rather than the blocking mechanism. The blocking mechanism can already be tested via the existing, hardcoded URLs [1]. Is there a reason we need to integrate these two testing mechanisms or is it ok for them to remain separate like this?

[1] http://searchfox.org/mozilla-central/rev/546a05fec017cb785fca62a5199d42812a6a1fd3/toolkit/components/url-classifier/SafeBrowsing.jsm#381

Comment 33

[Benjamin Smedberg]

I think you should discuss this with Abe. I figured you would have some end-to-end testing that list updates were working and in fact blocking Flash as part of your testing?

Comment 34

[Robin Steuber (she/her) [:bytesized]]

Abe, what is your opinion on this? Do the test URLs distributed by Shavar need to be visitable URLs with test Flash content, or is it sufficient to have "dummy" test URLs distributed by Shavar and visitable URLs hardcoded into the browser?

Comment 35

[Chris Peterson [:cpeterson]]

(In reply to Kirk Steuber [:bytesized] from comment #34)
> Abe, what is your opinion on this? Do the test URLs distributed by Shavar
> need to be visitable URLs with test Flash content, or is it sufficient to
> have "dummy" test URLs distributed by Shavar and visitable URLs hardcoded
> into the browser?

AFAIK users cannot inspect the Shavar list contents on the client, so I believe the URLs must be visitable so QA can confirm that the URLs are blocked (and thus confirm the client has downloaded the expected lists).

We can probably add some new resources to be blocked on itisatrap.org or people.mozilla.org.

Comment 36

[Robin Steuber (she/her) [:bytesized]]

@cpeterson From the browser toolbox, the following code will allow you to determine which lists a URL is on:

let service = Cc["@mozilla.org/url-classifier/dbservice;1"].getService(Ci.nsIURIClassifier);
let ios = Cc["@mozilla.org/network/io-service;1"].getService(Ci.nsIIOService);
let uri = ios.newURI("http://flashblock.itisatrap.org/")
let tables = "comma_separated_list_of_tables_to_check"
service.classifyLocal(uri, tables)

This will return a list of tables that the URI occurs in.

Comment 37

[Abe - QA (:Abe_LV)]

Kirk, any steps to run the code in comment 36 so that we can populate the list? 
What channel should this be tested? 
What is the target milestone for this?

Comment 38

[Robin Steuber (she/her) [:bytesized]]

If you wanted to test for the existence of the itisatrap.org domains, no steps are necessary to populate the list. It is done automatically as those URLs are hardcoded.

To test for the existence of the domains on the Shavar lists, you would need to wait for the lists to be updated. I am afraid that information about when exactly the lists get updated naturally or how to force a list update is a question for the Shavar/SafeBrowsing team. 

I do know a preference that is useful here: |browser.safebrowsing.provider.mozilla.nextupdatetime|. Before the lists have ever been updated, the preference will have a value of |1|. By putting an observer on this pref, you can also schedule a function to be run when the lists are updated.

We probably want to test this in Firefox 53 (Aurora channel).

@francois When do the lists get updated? Especially, when does the first list update occur? Is there a way to trigger an immediate update?

Comment 39

[François Marier [:francois]]

(In reply to Kirk Steuber [:bytesized] from comment #38)
> @francois When do the lists get updated? Especially, when does the first
> list update occur? Is there a way to trigger an immediate update?

It should happen within 30 seconds of browser startup.

You can set "browser.safebrowsing.provider.mozilla.nextupdatetime = 1" to force a list download at startup time.

Comment 40

[Luke Crouch [:groovecoder]]

tl;dr: When we merge new contents into the master branch of shavar-plugin-blocklist repo on GitHub, it can take up to 30m before Firefox clients will start to see it.

On the server side, the ShavarListCreationProd job runs every 30m - typically at HH:11 and HH:41 GMT. It pulls files down from https://github.com/mozilla-services/shavar-plugin-blocklist and creates the SafeBrowsing-compatible versions of those files, which are then served by shavar. Currently, we still need to restart the shavar servers to make them serve *new* lists, but once shavar is serving a list, its contents will auto-update.

Comment 41

[Abe - QA (:Abe_LV)]

Discussed with Kirk on irc. We will create test cases and update the test plan accordingly.

Comment 42

[Robin Steuber (she/her) [:bytesized]]

After discussing with Abe, it seems that dummy URLs are not sufficient. I am looking into what "live" URLs could be used for this.

Comment 43

[Luke Crouch [:groovecoder]]

The shavar stage server is now serving the new flash lists:

curl -k 'https://shavar.stage.mozaws.net/list'
allow-flashallow-digest256
base-track-digest256
baseeff-track-digest256
basew3c-track-digest256
block-flash-digest256
block-flashsubdoc-digest256
content-track-digest256
contenteff-track-digest256
contentw3c-track-digest256
except-flash-digest256
except-flashallow-digest256
except-flashsubdoc-digest256
mozfull-track-digest256
mozfullstaging-track-digest256
mozplugin-block-digest256
mozpub-track-digest256
mozstd-track-digest256
mozstd-trackwhite-digest256
mozstdstaging-track-digest256
mozstdstaging-trackwhite-digest256
moztestpub-track-digest256
moztestpub-trackwhite-digest256

When you have URLs to add, you may send a pull request here:

https://github.com/mozilla-services/shavar-plugin-blocklist

And within 30 minutes of merge, the URLs will start showing up on the stage server.

Would you like to do an end-to-end test with the stage server before we add these lists to production?

Comment 44

[Luke Crouch [:groovecoder]]

:bytesized - so, I'm waiting to hear from you and/or :rpapa and/or :rbillings that the lists served by the shavar stage server are verified before I make the same changes to put the lists on the production server.

Comment 45

[Robin Steuber (she/her) [:bytesized]]

I do not think I understand what you are asking for. If you are requesting verification that the staging servers are serving the correct lists for Flash Blocking/testing Flash Blocking, I cannot answer this because we still have not decided what the contents of those lists should be.

Comment 46

[Benjamin Smedberg]

Kirk, didn't you already decide that the correct contents of those lists right now is either a) empty lists or b) contains one testing domain?

Comment 47

[Robin Steuber (she/her) [:bytesized]]

From the most recent emails, it sounds like the lists should contain the same values that are already hardcoded in [1].

@Abe_LV Can you confirm that these are the URLs that should be added and that these are all that will be needed for testing?

[1] http://searchfox.org/mozilla-central/rev/546a05fec017cb785fca62a5199d42812a6a1fd3/toolkit/components/url-classifier/SafeBrowsing.jsm#381

Comment 48

[Rebecca Billings [:rbillings]]

I need an updated ini file* & set of prefs for the new flash blocking list in order to complete e2e testing.

* https://github.com/mozilla-services/services-test/blob/master/shavar/e2e-test/prefs.ini

Comment 49

[Abe - QA (:Abe_LV)]

(In reply to Kirk Steuber [:bytesized] from comment #47)
> From the most recent emails, it sounds like the lists should contain the
> same values that are already hardcoded in [1].
> 
> @Abe_LV Can you confirm that these are the URLs that should be added and
> that these are all that will be needed for testing?
> 
> [1]
> http://searchfox.org/mozilla-central/rev/
> 546a05fec017cb785fca62a5199d42812a6a1fd3/toolkit/components/url-classifier/
> SafeBrowsing.jsm#381
Yes, we only need these URLs [1] to test. We are creating test cases based that. BSmedberg said updating lists to include other sites will be follow-up work that happens before and during the experiment(s).

Comment 50

[Luke Crouch [:groovecoder]]

We have an opportunity now to verify "updating lists to include other sites ... before ... the experiment" works on the shavar stage server before we do it on prod "for realz".

Do we want to take that opportunity now, or should we go ahead and push these empty lists all the way to the production server?

Comment 51

[Robin Steuber (she/her) [:bytesized]]

I believe that we want to add just the itisatrap.org URLs. I will create a pull request today.

Comment 52

[Robin Steuber (she/her) [:bytesized]]

Pull request created: https://github.com/mozilla-services/shavar-plugin-blocklist/pull/5

Is there any more information required, or can we cancel the needinfos?

Comment 53

[Luke Crouch [:groovecoder]]

Merged the PR and the ShavarListCreationStage job shows expected output:

>Plugin blocklist(flash-blocklist): publishing 1 items; file size 51
>Plugin blocklist(flash-exceptions): publishing 1 items; file size 51
>Plugin blocklist(flash-allow): publishing 1 items; file size 51
>Plugin blocklist(flash-allow-exceptions): publishing 1 items; file size 51
>Plugin blocklist(flash-subdoc): publishing 1 items; file size 51
>Plugin blocklist(flash-subdoc-exceptions): publishing 1 items; file size 51
>...
>Uploaded to s3: flash-blocklist
>Uploaded to s3: flash-exceptions
>Uploaded to s3: flash-allow
>Uploaded to s3: flash-allow-exceptions
>Uploaded to s3: flash-subdoc
>Uploaded to s3: flash-subdoc-exceptions

So, the new URLs should be in the list files served by the shavar stage server. Now you can use the shavar stage server to test expected behavior of/on flashblock.itisatrap.org, flashallow.itisatrap.org, except.flashblock.itisatrap.org, except.flashallow.itisatrap.org, flashsubdoc.itisatrap.org, and except.flashsubdoc.itisatrap.org.

Whoever is doing that testing can start now.

Comment 54

[Robin Steuber (she/her) [:bytesized]]

It sounds like you are saying that no more information is required and the needinfos can be cancelled.

Comment 55

[Luke Crouch [:groovecoder]]

Without needinfo? flags or assignee on this bug, I'm afraid it will fall off anyone's workload.

So to be clear:

    1. We need to test both generating these lists, and verifying the browser behaves as expected from the content of these lists
    2. The list generation code is creating and deploying flash files in STAGE ONLY. These files are available at the stage endpoint at https://shavar.stage.mozaws.net
    3. Without QA testing on stage, we can't deploy these lists to production

If this is going to be a problem for the client side, we need to coordinate a testing plan that includes tests of actual working content-blocking configuration for these lists.

Comment 56

[:Felipe Gomes (needinfo for replies!)]

So I think the next steps should be:

- Make a custom try build for QA that doesn't contain these same URLs built in
- Ask them to use this build and configure the shavar pref to point to the staging server
- Force a list update and verify through itisatrap.org that the lists' contents are working as expected

Comment 57

[Abe - QA (:Abe_LV)]

(In reply to :Felipe Gomes (needinfo me!) from comment #56)
> So I think the next steps should be:
> 
> - Make a custom try build for QA that doesn't contain these same URLs built
> in
> - Ask them to use this build and configure the shavar pref to point to the
> staging server
> - Force a list update and verify through itisatrap.org that the lists'
> contents are working as expected

We have created test cases to test on official aurora build [1] and comments are appreciated. 
We can also make the testing on a staging server. Do we need a custom trybuild? If yes, who could assist us in making this build. We also need the list of prefs that needs to be set. 

[1] https://docs.google.com/a/mozilla.com/spreadsheets/d/1vCtxeAaKfwrbigc7fAk9bYhEb5E4DN6ODTOIcHC47qs/edit?usp=sharing

Comment 58

[:Felipe Gomes (needinfo for replies!)]

That looks good, Abe! Could we do two versions of this testing:

- [1] test with a current build of Nightly and Aurora  (this will test the built-in URLs)

- [2] and also with a custom try build that we'll generate (this build will have the built-in URLs removed, in order to test the list update mechanism)


For [1], you don't need to worry about setting browser.safebrowser.provider.mozilla.nextupdatetime.

For [2], you'll need that + another pref change on browser.safebrowsing.provider.mozilla.updateURL to point it to the staging server. (change the domain to to https://shavar.stage.mozaws.net while preserving the rest of the URL)

Comment 59

[Luke Crouch [:groovecoder]]

:Felipe's right - now that the flash-blocking lists are served at shavar.stage.mozaws.net, set those browser.safebrowsing.provider.mozilla prefs on that custom try build and you should see the new flash-blocking behavior.

Comment 60

[:Felipe Gomes (needinfo for replies!)]

(In reply to :Felipe Gomes (needinfo me!) from comment #58)
> That looks good, Abe! Could we do two versions of this testing:
> 
> - [1] test with a current build of Nightly and Aurora  (this will test the
> built-in URLs)

A small correction here: the best will be to test with a build of Beta 53

Comment 61

[Abe - QA (:Abe_LV)]

(In reply to :Felipe Gomes (needinfo me!) from comment #60)
> > - [1] test with a current build of Nightly and Aurora  (this will test the
> > built-in URLs)
> 
> A small correction here: the best will be to test with a build of Beta 53
Is this for built-in URLs only or for the custom build (staging)as well? 

In addition, we are planning to test on Windows only with the following combinations: 
What do you think?

1-Windows-10  x64 with 64-bits of Firefox
2-Windows-7   x86 with 32-bits of Firefox
3-Windows-8.1 x64 with 32-bits of Firefox

Comment 62

[:Felipe Gomes (needinfo for replies!)]

(In reply to Abe - QA (:Abe_LV) from comment #61)
> Is this for built-in URLs only or for the custom build (staging)as well? 

For built-in only.  I mean, the custom build will be based off beta too, but it's not something that you'll need to worry about (since it's one build)

> 
> In addition, we are planning to test on Windows only with the following
> combinations: 
> What do you think?
> 
> 1-Windows-10  x64 with 64-bits of Firefox
> 2-Windows-7   x86 with 32-bits of Firefox
> 3-Windows-8.1 x64 with 32-bits of Firefox

LGTM

Comment 63

[Abe - QA (:Abe_LV)]

We have completed testing using built-in URLs on beta 53.0b1 and custom on Nightly 55.0a1 and aurora 54.0a2.
Test cases executions show all green [1]. Link to test cases is available in comment 57.
Please needinfo me for any additional testing or about the report. 

[1] https://drive.google.com/a/mozilla.com/file/d/1ljj7NWNzAqpqeypNAYtDL8JN1HzBZNWQjAQJr4OXR2c3C5772_LzFHMR1cwIKGzseLettk_CBDH0L7FE/view?usp=sharing

Comment 64

[Rebecca Billings [:rbillings]]

As I noted in Comment 48: I need an updated ini file* & set of prefs for the new flash blocking list in order to complete e2e testing.

* https://github.com/mozilla-services/services-test/blob/master/shavar/e2e-test/prefs.ini

As I never heard back I made an attempt to put them together myself- but this needs review:
user_pref("browser.startup.homepage", "http://itisatrap.org/firefox/its-a-tracker.html");
user_pref("browser.safebrowsing.debug", true);
user_pref("browser.safebrowsing.phishing.enabled", false);
user_pref("browser.safebrowsing.malware.enabled", false);
user_pref("privacy.trackingprotection.enabled", true);
user_pref("browser.safebrowsing.provider.mozilla.nextupdatetime", 1);
user_pref("plugins.flashBlock.enabled", true);
user_pref("browser.safebrowsing.provider.mozilla.lists", "block-flash-digest256, except-flash-digest256, allow-flashallow-digest256, except-flashallow-digest256, block-flashsubdoc-digest256, 
except-flashsubdoc-digest256");
user_pref("urlclassifier.disallow_completions", "block-flash-digest256, except-flash-digest256, allow-flashallow-digest256, except-flashallow-digest256, block-flashsubdoc-digest256, 
except-flashsubdoc-digest256");
user_pref("browser.safebrowsing.provider.mozilla.gethashURL", "https://shavar.stage.mozaws.net/gethash?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");
user_pref("browser.safebrowsing.provider.mozilla.updateURL", "https://shavar.stage.mozaws.net/downloads?client=SAFEBROWSING_ID&appver=%VERSION%&pver=2.2");

Comment 65

[Robin Steuber (she/her) [:bytesized]]

@rbillings I do not really understand what you are asking for because I have never seen an .ini file like this. It seems to be a list of Firefox preferences. Do you need a .ini file AND a list of preferences or are they the same thing?

I also do not really understand what you are trying to test. Are you just testing list update functionality specifically from the staging server? Are you testing Flash Blocking from list update to blocking functionality?

Furthermore, because I cannot really tell what you are requesting, I also cannot tell who the request is being directed towards. Are you expecting a response from me?

If you do want a response from me, could you please provide a more detailed description of what you are trying to accomplish and how you intend to do it? And if you want any answers specifically relating to a .ini file, I will need additional details regarding what the .ini file is and what is consumed by.

Comment 66

[Rebecca Billings [:rbillings]]

@bytesized the ini.file needs to be updated - in stage for testing, and eventually also in prod. I also need the list of preferences so that I can create a FF profile with the correct prefs in place.

Our tests cover e2e testing: verifying cache file sizes, that Tracking Protection displays correctly on approved/disapproved sizes, itsatrap etc. 

These updates in the past were done by francois. He has indicated in email that he has not worked on Flash Blocking, so someone else should do it. I wish I could tell you who the new person is- everyone I have asked has not taken it on, and yet this is where testing is blocked - until the ini. file is updated & the prefs list is confirmed.
 
I'm going to set a number of needinfo's on this until I find someone who can make this happen.

Comment 67

[Robin Steuber (she/her) [:bytesized]]

:ckolos has informed me that you wish to test all the way from an addition to the list to a Flash object being blocked. These are descriptions of the prefs that are relevant this:

This preference should be set to |true| to enable updating of the Flash blocking lists and to enable Flash blocking functionality.
    plugins.flashBlock.enabled

This preference should contain a comma-separated list of table names indicating all tables that should be updated from Mozilla's list server.
    browser.safebrowsing.provider.mozilla.lists

These preferences should contain comma-separated lists of table names. In order for these lists to be updated by Mozilla's list server, these lists should correspond to lists that are named by |browser.safebrowsing.provider.mozilla.lists|. It should be noted that the existing lists stored in these preferences do not update online and are hardcoded to contain subdomains of itisatrap.org. For more specifics, see the code that hardcodes these values [1].
    urlclassifier.flashAllowTable  (defines the whitelist)
    urlclassifier.flashAllowExceptTable  (defines exceptions to the whitelist)
    urlclassifier.flashTable  (defines the blacklist)
    urlclassifier.flashExceptTable  (defines exceptions to the blacklist)
    urlclassifier.flashSubDocTable  (defines the third-party blacklist)
    urlclassifier.flashSubDocExceptTable  (defines exceptions to the third-party blacklist)

These URLs control where list updates are requested for lists that are served by Mozilla:
    browser.safebrowsing.provider.mozilla.updateURL
    browser.safebrowsing.provider.mozilla.gethashURL

This value controls when the next update will be requested from Mozilla's list server. By setting its value to |1|, you can force the lists to be updated as soon as possible:
    browser.safebrowsing.provider.mozilla.nextupdatetime

This preference controls which tables may not receive partial results (which could be completed by an online query). I am not sure whether you need this or not. I imagine it depends on whether the server you are using supports partial hashes and completions.
    urlclassifier.disallow_completions

In summary, I believe that these are the prefs you will want to set:
 - |plugins.flashBlock.enabled| should be set to |true|
 - The above prefs of the form |urlclassifier.*| should either be set to the list names to be updated, or those list names should be added to the existing value (depending on whether or not you want the hardcoded tables).
 - The tables added to the |urlclassifier.*| prefs should also be added to |browser.safebrowsing.provider.mozilla.lists|
 - |browser.safebrowsing.provider.mozilla.updateURL| and |browser.safebrowsing.provider.mozilla.gethashURL| should point to the stage server (assuming that is what you want to use).
 - |browser.safebrowsing.provider.mozilla.nextupdatetime| should be set to 1.

I am afraid that I still do not know enough about this INI file to be able to address that in any way. It sounds to me like it probably needs to be updated by someone that knows more about it.

[1] http://searchfox.org/mozilla-central/rev/a5c2b278897272497e14a8481513fee34bbc7e2c/toolkit/components/url-classifier/SafeBrowsing.jsm#372-377,400-417

Comment 68

[:Felipe Gomes (needinfo for replies!)]

It looks like there's some overlap here. Abe from QA is already testing the entire process (through the staging server) and is right now testing the updating of the lists through the shavar staging server to Flash being blocked/whitelisted in the browser.

Comment 69

[Chris Kolosiwsky [:ckolos] (ckolos has left the building)]

There may be some overlap in this case, but Services QA (rbillings/rpapa) have their own testing to be done against the server-side of things. I'm not really surprised that there is some overlap. Adding new lists to the shavar service hasn't been done much, so I think some of the confusion here is due to the lack of a well-defined process for accomplishing that.

Comment 70

[Robin Steuber (she/her) [:bytesized]]

I would like to add the Shavar table names to the default prefs for Flash blocking (urlclassifier.flash*Table). It looks like these will be the names of the lists:

 - block-flash-digest256
 - except-flash-digest256
 - allow-flashallow-digest256
 - except-flashallow-digest256
 - block-flashsubdoc-digest256
 - except-flashsubdoc-digest256

Could somebody (maybe :groovecoder or :francois) please confirm that these are the table names that should be added?

Comment 71

[François Marier [:francois]]

Looks fine to me. Luke?

Comment 72

[Luke Crouch [:groovecoder]]

Those list names match what we have here:

https://github.com/mozilla-services/shavar-list-creation-config/blob/master/stage.ini#L76-L104

I'm not sure what "table names" mean, as compared to "list names"

Comment 73

[Robin Steuber (she/her) [:bytesized]]

Attachment: Bug 1314094 - Add Shavar lists to Flash blocking pref defaults

This change adds the names of the lists that Shavar will be serving to the Flash blocking prefs. This will allow those lists to control what domains are blocked or whitelisted.

Review commit: https://reviewboard.mozilla.org/r/120606/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/120606/

Comment 74

[François Marier [:francois]]

Comment on attachment 8847687 [details]
Bug 1314094 - Add Shavar lists to Flash blocking pref defaults

https://reviewboard.mozilla.org/r/120606/#review122608

I assume that you are going to wait until the new lists are in production before landing this?

::: modules/libpref/init/all.js:5164
(Diff revision 1)
>  
>  #else
>  pref("urlclassifier.downloadAllowTable", "");
>  #endif // XP_WIN
>  
>  pref("urlclassifier.disallow_completions", "test-malware-simple,test-phish-simple,test-unwanted-simple,test-track-simple,test-trackwhite-simple,test-block-simple,test-flashallow-simple,testexcept-flashallow-simple,test-flash-simple,testexcept-flash-simple,test-flashsubdoc-simple,testexcept-flashsubdoc-simple,goog-downloadwhite-digest256,base-track-digest256,mozstd-trackwhite-digest256,content-track-digest256,mozplugin-block-digest256,mozplugin2-block-digest256");

You need to add the new lists to this pref as well because we don't want the browser to call the non-existent gethash endpoint on the shavar server.

Comment 75

[Robin Steuber (she/her) [:bytesized]]

(In reply to François Marier [:francois] from comment #74)
> I assume that you are going to wait until the new lists are in production
> before landing this?

I wanted to land it now. We need this for the Shield study in Firefox 53, so it will be need to be uplifted to beta.

I had thought that updating non-existant lists would fail silently, allowing them to do nothing until they are served. Is that not the case?

I feel that I should also point out that these lists will continue to do nothing (the browser will not update or use them) until the pref |plugins.flashBlock.enabled| is set to |true|.

So can I add these lists to the prefs now and uplift them to beta? If not, how do I add them to Firefox 53 when Shield study takes place (at which point, Firefox 53 will be Release, not Beta)?

Comment 76

[Robin Steuber (she/her) [:bytesized]]

Comment on attachment 8847687 [details]
Bug 1314094 - Add Shavar lists to Flash blocking pref defaults

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/120606/diff/1-2/

Comment 77

[François Marier [:francois]]

(In reply to Kirk Steuber [:bytesized] from comment #75)
> So can I add these lists to the prefs now and uplift them to beta? If not,
> how do I add them to Firefox 53 when Shield study takes place (at which
> point, Firefox 53 will be Release, not Beta)?

Yes, it's OK to add them now, they just won't work until they're present in prod.

Comment 78

[François Marier [:francois]]

Comment on attachment 8847687 [details]
Bug 1314094 - Add Shavar lists to Flash blocking pref defaults

https://reviewboard.mozilla.org/r/120606/#review122618

Comment 79

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/27455a258006
Add Shavar lists to Flash blocking pref defaults r=francois

Comment 80

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/27455a258006

Comment 81

[François Marier [:francois]]

Reopening since the lists aren't yet in production.

Comment 82

[Robin Steuber (she/her) [:bytesized]]

Comment on attachment 8847687 [details]
Bug 1314094 - Add Shavar lists to Flash blocking pref defaults

Approval Request Comment
[Feature/Bug causing the regression]: These pref changes are needed for the Shield study to be run on Release 53 (bug 1335232), which we'll use to study the effect of making flash click-to-play by default.
[User impact if declined]: Study's allow and block lists will not be update-able.
[Is this code covered by automated tests?]: N/A - This is just a pref change.
[Has the fix been verified in Nightly?]: Yes
[Needs manual test from QE? If yes, steps to reproduce]: Manual testing of the added lists is being conducted by :Abe_LV and :rbillings.
[List of other uplifts needed for the feature/fix]: None
[Is the change risky?]: Minimal risk
[Why is the change risky/not risky?]: This changes the behavior of code that is currently pref-ed off and will only be exposed to users who are part of the study.
[String changes made/needed]: none

Comment 83

[Abe - QA (:Abe_LV)]

(In reply to Abe - QA (:Abe_LV) from comment #63)
> We have completed testing using built-in URLs on beta 53.0b1 and custom on
> Nightly 55.0a1 and aurora 54.0a2.
> Test cases executions show all green [1]. Link to test cases is available in
> comment 57.
> Please needinfo me for any additional testing or about the report. 

I want to make a correction on my comment:
We tested using built-in URLs and it is all green. We did not test on a staging server. Felipe is trying to give us custom build and prefs to test that.

Comment 84

[Abe - QA (:Abe_LV)]

We have tested again by making pref changes on the latest nightly to get URLs from shavar services.
So, test cases [1] run shows all pass [2]. Please let us know if you have questions about those docs.

[1]https://docs.google.com/a/mozilla.com/spreadsheets/d/1vCtxeAaKfwrbigc7fAk9bYhEb5E4DN6ODTOIcHC47qs/edit?usp=sharing
[2]https://drive.google.com/a/mozilla.com/file/d/17D3MfCnqD3_dVcsLomM6usoigaOAufYkmmwmPrr8tlIFzCANca5gKcwH1EvIgNHxZNNrXCjUM3MaSlZL/view?usp=sharing

Comment 85

[Liz Henry (:lizzard) (relman/hg->git project)]

Comment on attachment 8847687 [details]
Bug 1314094 - Add Shavar lists to Flash blocking pref defaults

Putting new blocklists in place for the upcoming Shield study for 53. 
This code should all be behind a pref (plugins.flashBlock.enabled) when we ship, so this change should only affect the users who will be part of the Shield study.

Comment 86

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/0496e366a8f4

Comment 87

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/92d8d94bfed5

Comment 88

[Rebecca Billings [:rbillings]]

Verified flashblock lists are being served on prod, and that flashblock pref is correctly disabled in about:config.

Comment 89

[Luke Crouch [:groovecoder]]

Thanks :rbillings.

:bytesized - when you want to add/update content in the lists, send a pull request to https://github.com/mozilla-services/shavar-plugin-blocklist. When it's merged, clients should receive the new list contents within 30m.

Do we call this bug closed now and move future work to other bugs?

Comment 90

[Robin Steuber (she/her) [:bytesized]]

Sounds good to me.

Thank you everyone!

Comment 91

[Abe - QA (:Abe_LV)]

QA verified as fixed.