Closed Bug 1478195 Opened 6 years ago Closed 6 years ago

Deploy the fastblock lists to prod

Categories

(Cloud Services :: Operations: Shavar, task, P1)

Product:
Cloud Services
Component:
Operations: Shavar
Type:
task

Tracking

(Not tracked)

Status:
VERIFIED FIXED

People

(Reporter: francois, Assigned: ckolos)

References

Details

Now that bug 1474412 is complete and has been tested, we can go to prod: 14:11 < rbillings> francois: all tests look good 14:16 < francois> rbillings: great, so we have your OK to move to prod? 14:24 < rbillings> francois: absolutely
Depends on: 1474412
done
Status: ASSIGNED → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
I used the scripts in the safebrowsing-tools repo (https://github.com/fmarier/safebrowsing-tools) to confirm that all of the new lists are advertised by the shavar server: $ ./get-lists.py | grep fastblock fastblock1-track-digest256 fastblock1-trackwhite-digest256 fastblock2-track-digest256 fastblock2-trackwhite-digest256 fastblock3-track-digest256 and then requested them all: $ ./download-list.py n:3600 i:fastblock1-trackwhite-digest256 u:shavar.services.mozilla.com/fastblock1-trackwhite-digest256/1532636914 i:fastblock2-track-digest256 u:shavar.services.mozilla.com/fastblock2-track-digest256/1532636914 i:fastblock1-track-digest256 u:shavar.services.mozilla.com/fastblock1-track-digest256/1532636914 i:fastblock3-track-digest256 u:shavar.services.mozilla.com/fastblock3-track-digest256/1532636914 i:fastblock2-trackwhite-digest256 u:shavar.services.mozilla.com/fastblock2-trackwhite-digest256/1532636914 However, when I follow these redirects, they all return a 404: $ curl https://shavar.services.mozilla.com/fastblock1-track-digest256/1532636914 The requested page was not found. $ curl https://shavar.services.mozilla.com/fastblock1-trackwhite-digest256/1532636914 The requested page was not found. $ curl https://shavar.services.mozilla.com/fastblock2-track-digest256/1532636914 The requested page was not found. $ curl https://shavar.services.mozilla.com/fastblock2-trackwhite-digest256/1532636914 The requested page was not found. $ curl https://shavar.services.mozilla.com/fastblock3-track-digest256/1532636914 The requested page was not found. The interesting thing is that this domain is different from when I request the tracking protection list: $ ./download-list.py n:3600 i:base-track-digest256 u:tracking-protection.cdn.mozilla.net/base-track-digest256/1512580265 That list works fine: $ curl --head https://tracking-protection.cdn.mozilla.net/base-track-digest256/1512580265 HTTP/1.1 200 OK Content-Type: application/octet-stream Content-Length: 57526 Connection: keep-alive Date: Thu, 26 Jul 2018 13:01:41 GMT Last-Modified: Wed, 06 Dec 2017 17:11:10 GMT ETag: "91c9f9b9b6e2b5c7312e0df739362f85" Accept-Ranges: bytes Server: AmazonS3 Age: 29447 X-Cache: Hit from cloudfront Via: 1.1 affec1a0de50d1f690b08b468f7d2156.cloudfront.net (CloudFront) X-Amz-Cf-Id: gWlNhBSZiimBauIcftKQwtcGOnVO6iB-RnR2kSXGs7Bqx1pseQU0SQ== and if I use the same hostname for the fastblock lists, it also seems to work: $ curl --head https://tracking-protection.cdn.mozilla.net/fastblock1-track-digest256/1532636914 HTTP/1.1 200 OK Content-Type: application/octet-stream Content-Length: 55830 Connection: keep-alive Date: Mon, 30 Jul 2018 23:45:30 GMT Last-Modified: Thu, 26 Jul 2018 20:29:01 GMT ETag: "1128bd872f4b2685ac13ba81905bcdfd" Accept-Ranges: bytes Server: AmazonS3 X-Cache: Miss from cloudfront Via: 1.1 b63769e2d89c89274acd908e4bfcb9f4.cloudfront.net (CloudFront) X-Amz-Cf-Id: uXqb45dwDKlF2m8jjT4_Z5Hz4-Z9AFYJd8gMnMWcLXBgF2-ZT33ijg== Could it be that there's some missing CDN config?
Status: RESOLVED → REOPENED
Flags: needinfo?(ckolos)
Resolution: FIXED → ---
That was indeed the case. Since we add lists so often (note, this is sarcasm :) ), I forgot that in stage the config is different than in prod. In prod, we serve lists only from the CDN. I've changed the configs, re-started all servers and now see the following (using your tools): ./download-list.py n:3600 i:fastblock1-trackwhite-digest256 u:tracking-protection.cdn.mozilla.net/fastblock1-trackwhite-digest256/1532636914 i:fastblock2-track-digest256 u:tracking-protection.cdn.mozilla.net/fastblock2-track-digest256/1532636914 i:fastblock1-track-digest256 u:tracking-protection.cdn.mozilla.net/fastblock1-track-digest256/1532636914 i:fastblock3-track-digest256 u:tracking-protection.cdn.mozilla.net/fastblock3-track-digest256/1532636914 i:fastblock2-trackwhite-digest256 u:tracking-protection.cdn.mozilla.net/fastblock2-trackwhite-digest256/1532636914
Status: REOPENED → RESOLVED
Closed: 6 years ago → 6 years ago
Flags: needinfo?(ckolos)
Resolution: --- → FIXED
I ran my tests again: $ ./download-list.py n:3600 i:fastblock1-trackwhite-digest256 u:tracking-protection.cdn.mozilla.net/fastblock1-trackwhite-digest256/1532636914 i:fastblock2-track-digest256 u:tracking-protection.cdn.mozilla.net/fastblock2-track-digest256/1532636914 i:fastblock1-track-digest256 u:tracking-protection.cdn.mozilla.net/fastblock1-track-digest256/1532636914 i:fastblock3-track-digest256 u:tracking-protection.cdn.mozilla.net/fastblock3-track-digest256/1532636914 i:fastblock2-trackwhite-digest256 u:tracking-protection.cdn.mozilla.net/fastblock2-trackwhite-digest256/1532636914 $ for u in "https://tracking-protection.cdn.mozilla.net/fastblock1-trackwhite-digest256/1532636914 https://tracking-protection.cdn.mozilla.net/fastblock2-track-digest256/1532636914 https://tracking-protection.cdn.mozilla.net/fastblock1-track-digest256/1532636914 https://tracking-protection.cdn.mozilla.net/fastblock3-track-digest256/1532636914 https://tracking-protection.cdn.mozilla.net/fastblock2-trackwhite-digest256/1532636914" ; do curl --head $u ; done HTTP/1.1 200 OK Content-Type: application/octet-stream Content-Length: 19446 Connection: keep-alive Date: Tue, 31 Jul 2018 19:29:23 GMT Last-Modified: Thu, 26 Jul 2018 20:29:03 GMT ETag: "fdb36954b5f1c8a59e53f5a414293865" Accept-Ranges: bytes Server: AmazonS3 X-Cache: Miss from cloudfront Via: 1.1 5b51f6e8f38342d63beb93a0db7a392b.cloudfront.net (CloudFront) X-Amz-Cf-Id: vAg98D0jZAafczh_e5gHA1hTvrmqMxo96vArG6yvsD2nwPfpykJ0xw== HTTP/1.1 200 OK Content-Type: application/octet-stream Content-Length: 54614 Connection: keep-alive Date: Tue, 31 Jul 2018 19:29:23 GMT Last-Modified: Thu, 26 Jul 2018 20:29:04 GMT ETag: "20c158258c9acb6f8d831b826c3c5c6c" Accept-Ranges: bytes Server: AmazonS3 X-Cache: Miss from cloudfront Via: 1.1 5b51f6e8f38342d63beb93a0db7a392b.cloudfront.net (CloudFront) X-Amz-Cf-Id: NbfCS-mcWWg_PWnzSK_k_Y3p-KKJxu0TWMZ1tCndcD8164YFK7Q8tA== HTTP/1.1 200 OK Content-Type: application/octet-stream Content-Length: 55830 Connection: keep-alive Date: Tue, 31 Jul 2018 19:29:24 GMT Last-Modified: Thu, 26 Jul 2018 20:29:01 GMT ETag: "1128bd872f4b2685ac13ba81905bcdfd" Accept-Ranges: bytes Server: AmazonS3 X-Cache: Miss from cloudfront Via: 1.1 5b51f6e8f38342d63beb93a0db7a392b.cloudfront.net (CloudFront) X-Amz-Cf-Id: ED1tYTceJac3aXaZ4I-zUlVO6Py-SJVKcCUkYWSyiiT6KE5kzRMwWg== HTTP/1.1 200 OK Content-Type: application/octet-stream Content-Length: 57494 Connection: keep-alive Date: Tue, 31 Jul 2018 19:29:24 GMT Last-Modified: Thu, 26 Jul 2018 20:29:07 GMT ETag: "3f5efa2a4c6f385cd5ebd32556cee4d3" Accept-Ranges: bytes Server: AmazonS3 X-Cache: Miss from cloudfront Via: 1.1 5b51f6e8f38342d63beb93a0db7a392b.cloudfront.net (CloudFront) X-Amz-Cf-Id: 0C-CdPlTnfOxBwgGFdI9UnnKDOq8gp2LaqHWSQ5ezeQs7VOJzjqO2w== HTTP/1.1 200 OK Content-Type: application/octet-stream Content-Length: 2037 Connection: keep-alive Date: Tue, 31 Jul 2018 19:29:24 GMT Last-Modified: Thu, 26 Jul 2018 20:29:06 GMT ETag: "369dec062c3353ed8647c9a419472062" Accept-Ranges: bytes Server: AmazonS3 X-Cache: Miss from cloudfront Via: 1.1 5b51f6e8f38342d63beb93a0db7a392b.cloudfront.net (CloudFront) X-Amz-Cf-Id: 3hpB2fu36J5a9iqd-q06yZulcYFsfKhtCVirmosMnnhBX8W_gAsyXg== Then I created three separate profiles and tested https://mozilla.github.io/tracking-test/fastblock.html successfully in each of them. I'll let Rebecca mark this as VERIFIED once she's had a chance to run through her tests, but from my point of view it all looks good now.
Production tests pass, new lists are returned, all 3 Fastblock list profiles work as expected.
Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.