991843 - Set up blocklist.addons.mozilla.org

Status: RESOLVED FIXED

(Cloud Services :: Operations: Marketplace, task, P2)

Comment 1

[Anurag Phadke[:[e-mail]]]

oremj - current BLP uses addons.mozilla.org as URL. Changing the URL involves modification on the client side. It's impossible to push updates to our entire userbase, changing the end point will mean we will need to look at 2 different locations to get ADIs and thus complicate the process.

Please correct me if I am wrong, happy to chat over vidyo.

-anurag

Comment 2

[Jeremy Orem [:oremj]]

This is correct. All old versions will continue to use addons.mozilla.org and all new versions would use blocklist.addons.mozilla.org.

Comment 3

[Jason Thomas [:jason]]

Can the client handle 301 redirect? We can configure addons.mozilla.org blocklist to redirect to the domain.

Comment 4

[Anurag Phadke[:[e-mail]]]

jason - the client maybe able to handle 301's, but let's avoid if we can.

oremj - this complicates the pipeline, any specific reason to move to blocklist.a.m.o?

Comment 5

[Jeremy Orem [:oremj]]

Yes. It is a big source of load and we'd like to move it to its own cluster and we'd like to move it behind zeus so we have more logging options and stability.

Comment 6

[Robert Kaiser]

For one thing, I'm pretty sure the blocklist request can handle a 301, but we should of course QA that to be sure.
That said, we can actually send an add-on hotfix to the vast majority of Firefox installations out there (IIRC Fx 10+) and switch the extensions.blocklist.url pref with that.

Even with that, we should make sure that the 301 works, as I'm not sure if Thunderbird and Firefox for Android have this capability, I'm pretty sure that SeaMonkey doesn't - and not sure what else is still requesting add-on blocklists and therefore sending the ping.

Comment 7

[Mike Connor [:mconnor]]

I'd be surprised if any HTTP lib doesn't handle 301s cleanly.

I do like the hotfix idea, since we've got a _lot_ of lagged users out there.  We still wouldn't get 100%, but we'd get a good idea of how effective that option is.

Comment 8

[Anurag Phadke[:[e-mail]]]

jason/mconnor - how do we plan to release this? Push the 301 out on nightly channel first followed by other channels?

Comment 9

[Mark Mayo [:mmayo]]

Let's also make sure the 301 actually works, too.  :)

Comment 10

[Mike Connor [:mconnor]]

Is there a bug filed on updating clients to use the new hostname once it's available?  It's probably too late for Fx29, but we can get it on branches so it'll hit Fx30.

Comment 11

[Anurag Phadke[:[e-mail]]]

one wacky suggestion - why not just stick to 301's and not do any client change? That way, we will only have one data source to look at....

-anurag

Comment 12

[Jeremy Orem [:oremj]]

(In reply to Mike Connor [:mconnor] from comment #10)
> Is there a bug filed on updating clients to use the new hostname once it's
> available?  It's probably too late for Fx29, but we can get it on branches
> so it'll hit Fx30.

We'll get this filed once blocklist.addons... is set up properly.

(In reply to Anurag Phadke[:[e-mail]] from comment #11)
> one wacky suggestion - why not just stick to 301's and not do any client
> change? That way, we will only have one data source to look at....
> 
> -anurag

You should only have to look at "blocklist.addons" logs in either set up. With 301s only, we'd be serving double the requests for all clients, which isn't desirable.

Comment 13

[John Jensen]

Hi all,

As per our meeting a week or so:

1) I have asked Anthony Hughes' team to do some basic testing of the Fx Desktop client, using both recent and old browsers. I will create a ticket for this.

Comment 14

[Jason Thomas [:jason]]

https://blocklist.addons.mozilla.org is up. 

https://blocklist.addons.mozilla.org/blocklist/3/x/x/

Comment 15

[Anurag Phadke[:[e-mail]]]

jason - when will it start receiving traffic?

Comment 16

[Jason Thomas [:jason]]

(In reply to Anurag Phadke[:[e-mail]] from comment #15)
> jason - when will it start receiving traffic?

We probably want to make sure 301 work first (bug 999784) before we start sending traffic. I've setup https://addons-blocklist-redirect.allizom.org for testing purposes. 

ex. https://addons-blocklist-redirect.allizom.org/blocklist/3/x/x/

Comment 17

[Jason Thomas [:jason]]

Monitoring added, documentation https://mana.mozilla.org/wiki/display/websites/blocklist.addons.mozilla.org.

Comment 18

[Mike Connor [:mconnor]]

We should file a bug to change the client (obviously with a dependency on jjensen's testing bug) if we want to get this landed in the current cycle.  It's super easy to lose track of this sort of thing...

Comment 19

[Jason Thomas [:jason]]

https://bugzilla.mozilla.org/show_bug.cgi?id=1006615

Comment 20

[Robert Kaiser]

As bug 970406 comment #3 describes, something is wrong with logging.
I verified at least on my machine that we call the full URL but in the logs we get a cut-off version of it.

Comment 21

[Jason Thomas [:jason]]

Updated custom log format to:

%h %{Host}i %u %t "%{realpath}d" %s %b "%{Referer}i" "%{User-Agent}i" "%{Cookie}i" "DNT:%{DNT}i"

Please test and verify

Comment 22

[Anurag Phadke[:[e-mail]]]

jason - the log lines aren't in the correct format:

Actual:
122.163.208.154 blocklist.addons.mozilla.org - [25/May/2014:00:00:05 -0700] "/blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/32.0a1/Firefox/20140516030204/WINNT_x86-msvc/en-US/nightly/Windows_NT%206.1/default/default/17/79/1/" 200 117800 "-" "Mozilla/5.0 (Windows NT 6.1; rv:32.0; WUID=99560722a36bb0cb89c4fc9dbde5a873; WTB=6724) Gecko/20100101 Firefox/32.0" "-" "DNT:-"

Expected:
93.215.144.48 addons.mozilla.org - [25/May/2014:00:00:00 +0000] "GET /blocklist/3/%7Bec8030f7-c20a-464f-9b0e-13a3a9e97384%7D/17.0.1/Firefox/20121128204232/WINNT_x86-msvc/de/release/Windows_NT%205.1/default/default/2/2/514/ HTTP/1.1" 200 117646 "-" "Mozilla/5.0 (Windows NT 5.1; rv:17.0) Gecko/20100101 Firefox/17.0" "-" "DNT:-"

The type of request (GET/POST) is missing in the current blocklist.addons.mozilla.org format.

Also, what's the timezone for the zeus files? Is it PST or UTC? IIRC, nginx is UTC.

-anurag

Comment 23

[Robert Kaiser]

We should see that times are UTC, if possible, yes. Can we make sure of that? Given that the cited log entry says "-0700" as time zone, I guess it's not right now.

Comment 24

[Jason Thomas [:jason]]

Updated custom log format to:

%h %{Host}i %u %t "%m %{realpath}d" %s %b "%{Referer}i" "%{User-Agent}i" "%{Cookie}i" "DNT:%{DNT}i"

Please test and verify. 

71.167.42.204 blocklist.addons.mozilla.org - [27/May/2014:07:15:15 -0700] "GET /blocklist/3/x/x/" 200 82799 "-" "curl/7.30.0" "-" "DNT:-"

Currently the Zeus LB's are configured to PDT/PST timezone. I agree they should be in UTC, filed bug 1016369 for timezone change.