Last Comment Bug 450745 - move addons.mozilla.org behind Akamai
: move addons.mozilla.org behind Akamai
Status: RESOLVED INCOMPLETE
:
Product: mozilla.org Graveyard
Classification: Graveyard
Component: Server Operations: Projects (show other bugs)
: other
: All Other
: -- minor (vote)
: ---
Assigned To: Jeremy Orem [:oremj]
: matthew zeier [:mrz]
Mentors:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2008-08-15 07:34 PDT by matthew zeier [:mrz]
Modified: 2015-03-12 08:24 PDT (History)
13 users (show)
mzeier: needs‑downtime+
See Also:
QA Whiteboard:
Iteration: ---
Points: ---


Attachments
Screenshot of successful upload on 64.245.124.43 (190.35 KB, image/png)
2008-08-22 15:34 PDT, Stephen Donner [:stephend]
no flags Details

Description matthew zeier [:mrz] 2008-08-15 07:34:28 PDT
Need CSR.
Comment 1 Frank Hecker 2008-08-15 11:13:25 PDT
Per my earlier comments in bug 449610, I think we should seriously consider getting an EV SSL certificate for the addons.mozilla.org site, especially if the plan is to get a new SSL certificate for the site anyway (i.e., it wouldn't continue to use the existing *.mozilla.org wildcard). AMO is an end-user site, the default mode of interacting with the site is via SSL (http URLs being redirected to https), and IMO we have an interest in assuring end users that it is in fact an official Mozilla site.

Also, if we do get an EV cert for AMO arguably the site owner should be listed as "Mozilla" (not "Mozilla Foundation" or "Mozilla Corporation") assuming the CA is willing to do this. This is consistent with the text on the site itself.
Comment 2 Reed Loden [:reed] (use needinfo?) 2008-08-15 11:51:54 PDT
http://www.geotrust.com/products/ssl_certificates/true_businessid_ev.asp is GeoTrust's EV "product". I'll generate the CSR with an O of Mozilla.
Comment 3 Johnathan Nightingale [:johnath] 2008-08-15 12:05:41 PDT
I can't add much to what Frank's said here - even though AMO has third party software all over it, we try very hard throughout the site to ensure that they are getting the real goods (hence SSL everywhere) and that they know it's a mozilla-maintained site - our trust relationship is on the line: that's where identity UI is intended to be used.

Obviously you guys need to make the call on whether you're willing to go through the more extended vetting, and I've never been through one, so maybe it's reasonably painless, maybe not, but in principle if I could just make an EV cert magically appear, AMO is probably the Mozilla site I'd choose.
Comment 4 Reed Loden [:reed] (use needinfo?) 2008-08-15 12:17:31 PDT
Subject: C=US, ST=California, L=Mountain View, O=Mozilla, OU=Add-ons CDN, CN=addons.mozilla.org/emailAddress=hostmaster@mozilla.com

-----BEGIN CERTIFICATE REQUEST-----
MIIDHjCCAgYCAQAwgaYxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlh
MRYwFAYDVQQHEw1Nb3VudGFpbiBWaWV3MRAwDgYDVQQKEwdNb3ppbGxhMRQwEgYD
VQQLEwtBZGQtb25zIENETjEbMBkGA1UEAxMSYWRkb25zLm1vemlsbGEub3JnMSUw
IwYJKoZIhvcNAQkBFhZob3N0bWFzdGVyQG1vemlsbGEuY29tMIIBIjANBgkqhkiG
9w0BAQEFAAOCAQ8AMIIBCgKCAQEAt3ZlNEGSfTA6V8SD+OL3+Y0kSzGDivuQm0ah
1c9Lv7gild2H218mkkYol2G2XvGoDkPI8eUDEZ1e7G401Zwp9fR3UehCAo4CLfSS
tUlVp1/729CDt+NlMni0wnOY8P85Mic9JeWrfy4gNhI8C3udroLTbvtwpfC4Y7Z/
fQVxUJDjvY1/jkRsoJvA8e4bRkiCsyp9vt7THlWlwKbAiEmeLy0mr13n2ysOnxdl
auOPCr1hGbu8tH/C7LDcwL95vuKv3hm1/WTlDC6ZkGazpAnEa2A3gCWq/le4Z1at
oaIZBM6eDoWzT3hvBza8gJ1gmaWEIMX8Y4buBepEVqSuCflhuwIDAQABoDIwMAYJ
KoZIhvcNAQkOMSMwITARBglghkgBhvhCAQEEBAMCBkAwDAYDVR0TAQH/BAIwADAN
BgkqhkiG9w0BAQUFAAOCAQEAZK72YA7zM4ICNg50Hoq7hxaOqczDuEa4QZGHlNPA
jtAYNuGBKEvnZ8YpL4xbYjJ4XGkgqEDEwlB11fdMELXCslitUWIGNFl5Lbd25ROt
85ahsi31K3yOBZOyvD+qC4CIcMcjs/PSozN6BlPCfp4nlKgRYqLGrfFnlOgxdia3
NCase26vgD7nx5tLtpGtxlPuY6DWh0sl2zVrBUjWktddSQFbMhrl70wnucX8QRdi
zC5F0l1tX8mAJt3jXkClwqA/eHSQFpldI5iLTkZ4C0LQcCvSYl+wfFuH6Arkie0J
TztjzaZZh5G3fToQ1NXuEcCmzK/X11jRcrsnRfa1cF0m0g==
-----END CERTIFICATE REQUEST-----
Comment 5 matthew zeier [:mrz] 2008-08-18 10:48:25 PDT
Don't have a login, over to justin.  This needs to be an EV cert.
Comment 6 Justin Fitzhugh 2008-08-18 14:43:37 PDT
no on the EV cert per discussions.  on order.
Comment 7 matthew zeier [:mrz] 2008-08-19 08:55:05 PDT
-----BEGIN CERTIFICATE-----
MIIDhTCCAu6gAwIBAgIDCa+2MA0GCSqGSIb3DQEBBQUAME4xCzAJBgNVBAYTAlVT
MRAwDgYDVQQKEwdFcXVpZmF4MS0wKwYDVQQLEyRFcXVpZmF4IFNlY3VyZSBDZXJ0
aWZpY2F0ZSBBdXRob3JpdHkwHhcNMDgwODE5MTM0NDM5WhcNMTAwODIwMTM0NDM5
WjCBizELMAkGA1UEBhMCVVMxEzARBgNVBAgTCkNhbGlmb3JuaWExFjAUBgNVBAcT
DU1vdW50YWluIFZpZXcxHDAaBgNVBAoTE01vemlsbGEgQ29ycG9yYXRpb24xFDAS
BgNVBAsTC0FkZC1vbnMgQ0ROMRswGQYDVQQDExJhZGRvbnMubW96aWxsYS5vcmcw
ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC3dmU0QZJ9MDpXxIP44vf5
jSRLMYOK+5CbRqHVz0u/uCKV3YfbXyaSRiiXYbZe8agOQ8jx5QMRnV7sbjTVnCn1
9HdR6EICjgIt9JK1SVWnX/vb0IO342UyeLTCc5jw/zkyJz0l5at/LiA2EjwLe52u
gtNu+3Cl8Lhjtn99BXFQkOO9jX+ORGygm8Dx7htGSIKzKn2+3tMeVaXApsCISZ4v
LSavXefbKw6fF2Vq448KvWEZu7y0f8LssNzAv3m+4q/eGbX9ZOUMLpmQZrOkCcRr
YDeAJar+V7hnVq2hohkEzp4OhbNPeG8HNryAnWCZpYQgxfxjhu4F6kRWpK4J+WG7
AgMBAAGjga4wgaswDgYDVR0PAQH/BAQDAgTwMB0GA1UdDgQWBBTWol0U4Jm4Gb0q
OtJqA0cy9QtBYTA6BgNVHR8EMzAxMC+gLaArhilodHRwOi8vY3JsLmdlb3RydXN0
LmNvbS9jcmxzL3NlY3VyZWNhLmNybDAfBgNVHSMEGDAWgBRI5mj5K9KylddH2CMg
EE8zmJCf1DAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDQYJKoZIhvcN
AQEFBQADgYEAY1mIQBnED44I+Mtsswe4Iz79jD19yCBIQJGvxBPXTyeaX+nhJXCh
Z7w7IfGdN0UcMBA5n6AgqIG0ft+jclKctJImdSQuXvSzNbv1IGowPyYUJNRG/MOa
OBX++vEm18UANh6uul7pvBqWqr6PyuBTbTmdQqff/EUw5MAOndtyp2w=
-----END CERTIFICATE-----
Comment 8 Jeremy Orem [:oremj] 2008-08-19 13:15:04 PDT
Certs have been handed off to CDNetworks.
Comment 9 matthew zeier [:mrz] 2008-08-20 09:33:07 PDT
Thinking this should be extended to include versioncheck.addons.mozilla.org and services.addons.mozilla.org.  Haven't a clue how much bandwidth that that would equate to (and probably more as folks upgrade to Fx3?).  

Any reason to no do that?  How dependent are we on logs for those?  
Comment 10 matthew zeier [:mrz] 2008-08-20 09:55:29 PDT
AMO gets 20x the traffic of either services or versioncheck.  Likely to increase of course.
Comment 11 matthew zeier [:mrz] 2008-08-20 10:05:18 PDT
For numbers - versioncheck & services are each about 4Mbps.
Comment 12 Reed Loden [:reed] (use needinfo?) 2008-08-20 12:59:32 PDT
(In reply to comment #9)
> Thinking this should be extended to include versioncheck.addons.mozilla.org and
> services.addons.mozilla.org.  Haven't a clue how much bandwidth that that would
> equate to (and probably more as folks upgrade to Fx3?).  
> 
> Any reason to no do that?  How dependent are we on logs for those?  

We're extremely dependent on logs from versioncheck for AMO. I don't know about SAMO.
Comment 13 Basil Hashem [:baz] 2008-08-20 14:22:39 PDT
Don't want to hijack this bug but Justin can you explain the rationale of why NOT to get an EV cert (as you said in comment 6)? I must have missed that discussion.
Comment 14 matthew zeier [:mrz] 2008-08-22 08:51:03 PDT
Site up for testing @ 

root@mrz-mp (/Users/mrz/) 1> host addons.mozilla.org.usgscac.cdnetworks.net
addons.mozilla.org.usgscac.cdnetworks.net has address 64.245.124.40
addons.mozilla.org.usgscac.cdnetworks.net has address 64.245.124.43
Comment 15 Stephen Donner [:stephend] 2008-08-22 14:56:30 PDT
mrz: r=stephend for these two instances; tested quite liberally, with no issues found.

* Made sure to sign up on both boxes, change password, log in, etc.
* Also tested changes to my test add-on at https://addons.mozilla.org/en-US/developers/edit/7431, (author addition, preview image swapping, etc.), from both this site -> production, and ensured that changes in production -> these boxes took.
Comment 16 Justin Scott [:fligtar] 2008-08-22 15:18:58 PDT
I can't access the sites for some reason (tried editing my hosts file and everything), so Stephen - can you or did you try uploading a new version of your test add-on?
Comment 17 Stephen Donner [:stephend] 2008-08-22 15:34:11 PDT
Created attachment 335117 [details]
Screenshot of successful upload on 64.245.124.43
Comment 18 Justin Scott [:fligtar] 2008-08-24 13:37:55 PDT
(In reply to comment #12)
> (In reply to comment #9)
> > Thinking this should be extended to include versioncheck.addons.mozilla.org and
> > services.addons.mozilla.org.  Haven't a clue how much bandwidth that that would
> > equate to (and probably more as folks upgrade to Fx3?).  
> > 
> > Any reason to no do that?  How dependent are we on logs for those?  
> 
> We're extremely dependent on logs from versioncheck for AMO. I don't know about
> SAMO.
> 

Log loss is definitely not okay for a number of URLs with AMO and all of versioncheck.AMO. There are no current uses of logs for SAMO that I know of, but there will be in the coming months.

Can someone explain how this setup with CDNetworks works?
Comment 19 matthew zeier [:mrz] 2008-08-24 16:01:46 PDT
> Log loss is definitely not okay for a number of URLs with AMO and all of
> versioncheck.AMO. There are no current uses of logs for SAMO that I know of,
> but there will be in the coming months.

Log loss would be temporary and not even a full log loss.  The current log format from CDNetworks doesn't match what we're used to but we are getting logs.  The are working on matching our log format, however.

This is somewhat moot though since they aren't even configured for *.addons.mozilla.org and probably won't until after we've confirmed the log format for AMO works and then decide if that offloads enough traffic to make it worthwhile.

> Can someone explain how this setup with CDNetworks works?

Think of them as a giant distributed Netscaler - the proxy back to the origin server which in this case is the production San Jose Netscaler.  Otherwise they aren't much different than Netscaler (I took some liberty in that statement).

Did that answer your question?
Comment 20 Justin Scott [:fligtar] 2008-08-24 18:06:23 PDT
Ok, some additional questions:

- Are we going to be using them all the time or only in high load when we can't handle traffic ourselves?

- How does .NL play into this? is it useless now?

- Can we flush their caches, and how long do they cache for?

- Are all of the AMO rules of the netscaler in effect on their side?

(In reply to comment #19)
> Log loss would be temporary and not even a full log loss.  The current log
> format from CDNetworks doesn't match what we're used to but we are getting
> logs.  The are working on matching our log format, however.
It would need to match pretty much exactly for the stats script to parse it.

> This is somewhat moot though since they aren't even configured for
> *.addons.mozilla.org and probably won't until after we've confirmed the log
> format for AMO works and then decide if that offloads enough traffic to make it
> worthwhile.
Firefox 2 and lower is still using addons.mozilla.org for versioncheck, so the majority of our active user stats are still hitting addons.mozilla.org, not versioncheck.amo.
Comment 21 Jeremy Orem [:oremj] 2008-08-24 18:32:10 PDT
(In reply to comment #20)
> Ok, some additional questions:
> 
> - Are we going to be using them all the time or only in high load when we can't
> handle traffic ourselves?
> 
All the time.
> - How does .NL play into this? is it useless now?
We will no longer need the .NL cache.
> 
> - Can we flush their caches, and how long do they cache for?
We can flush their caches and they cache as long as cache headers tell them to.
> 
> - Are all of the AMO rules of the netscaler in effect on their side?
They will be.
> 
> (In reply to comment #19)
> > Log loss would be temporary and not even a full log loss.  The current log
> > format from CDNetworks doesn't match what we're used to but we are getting
> > logs.  The are working on matching our log format, however.
> It would need to match pretty much exactly for the stats script to parse it.
They are working on matching our format.

Comment 22 Justin Scott [:fligtar] 2008-08-25 08:51:59 PDT
Ok, everything sounds good to me then, as long as they match our format before we switch.

My only other question is the one that Basil already asked in comment #13 about why we're not getting an EV cert. My pizza place has an EV cert. ( https://www.papajohnsonline.com ) Mozilla has enough problems with people trying to pass off unofficial versions of Firefox as the real thing. AMO's source is available for anyone to use, which means I can make something that looks *exactly* like AMO on my own website in about 5 minutes and distribute virus add-ons from it. I think we should be as secure and identifiable as possible when distributing add-ons, which right now means a green "Mozilla" in the location bar.
Comment 23 Justin Scott [:fligtar] 2008-09-04 14:23:17 PDT
Since this is going live tonight, have they been able to match our log format? And what, if any, will be the delay before we get their logs? Also, what directory are the logs going to be stored in?
Comment 24 matthew zeier [:mrz] 2008-09-04 14:37:49 PDT
Postponing till next Tuesday.
Comment 25 matthew zeier [:mrz] 2008-09-04 16:28:46 PDT
(In reply to comment #23)
> Since this is going live tonight, have they been able to match our log format?

Yes of course, this was a requirement.

> And what, if any, will be the delay before we get their logs? 

Quoting, "logs will be avaialbe a day after the service in GMT time."  Logs are pulled from each node and pushed into a central repository for us to rsync over.

What do you have that uses AMO logs?  Can that just be integrated into stats processing so it's not something you have to deal with?
Comment 26 Justin Scott [:fligtar] 2008-09-04 16:46:39 PDT
Our download/active user counter parses the logs every day for developer stats. The system was almost done when oremj's addons stats stuff was done, so the plan was to switch our system over to using his somehow, but that's probably not going to happen soon (especially since I'm not in webdev anymore).

So any new directories of logs to be parsed have to be added to the scripts on dm-stats01.

Having a day delay is gonna cause some stats abnormalities that I guess we'll have to deal with. For example, instead of seeing the number of downloads of an add-on from Tuesday on Wednesday, we'll see a very small number of downloads that hit our netscaler, and then Thursday we'll see the rest of Tuesday show up.

It's not as much of a problem with downloads, but the active user counter only runs on Thursdays to count Wednesday's logs for Wednesdays only. So I guess we'd need to move that to Friday to count Wednesdays.
Comment 27 matthew zeier [:mrz] 2008-09-04 16:50:18 PDT
> So any new directories of logs to be parsed have to be added to the scripts on
> dm-stats01.

Is this process documented somewhere?  I feel like you're the only one that knows anything about it. 

> Having a day delay is gonna cause some stats abnormalities that I guess we'll
> have to deal with. For example, instead of seeing the number of downloads of an
> add-on from Tuesday on Wednesday, we'll see a very small number of downloads
> that hit our netscaler, and then Thursday we'll see the rest of Tuesday show
> up.

You won't even count the Netscaler logs - only thing hitting it will be CDNetworks and what does make it through to the Netscaler (in other words, wasn't already cached) will be counted twice when the CDNetworks logs are imported.
Comment 28 Daniel Veditz [:dveditz] 2008-09-04 17:31:11 PDT
(In reply to comment #13)
> Justin can you explain the rationale of why NOT to get an EV cert (as you
> said in comment 6)? I must have missed that discussion.

Ditto -- where's the discussion?
Comment 29 :Gavin Sharp [email: gavin@gavinsharp.com] 2008-09-04 17:37:53 PDT
EV cert decision seems to have been reversed - see bug 452314. Not sure how that relates to this bug.
Comment 30 matthew zeier [:mrz] 2008-09-04 18:50:28 PDT
(In reply to comment #29)
> EV cert decision seems to have been reversed - see bug 452314. Not sure how
> that relates to this bug.

It related to this bug in the sense that I'm holding off on the EV cert until I decide if CDNetworks is right or not.  I don't want to get an EV cert for a trial.
Comment 31 matthew zeier [:mrz] 2008-09-10 12:53:45 PDT
Unless there are objections with the Subject, you can use this CSR:

Subject: C=US, ST=California, L=Mountain View, O=Mozilla, OU=Add-ons CDN, CN=addons.mozilla.org/emailAddress=hostmaster@mozilla.com

-----BEGIN CERTIFICATE REQUEST-----
MIIDHjCCAgYCAQAwgaYxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlh
MRYwFAYDVQQHEw1Nb3VudGFpbiBWaWV3MRAwDgYDVQQKEwdNb3ppbGxhMRQwEgYD
VQQLEwtBZGQtb25zIENETjEbMBkGA1UEAxMSYWRkb25zLm1vemlsbGEub3JnMSUw
IwYJKoZIhvcNAQkBFhZob3N0bWFzdGVyQG1vemlsbGEuY29tMIIBIjANBgkqhkiG
9w0BAQEFAAOCAQ8AMIIBCgKCAQEAt3ZlNEGSfTA6V8SD+OL3+Y0kSzGDivuQm0ah
1c9Lv7gild2H218mkkYol2G2XvGoDkPI8eUDEZ1e7G401Zwp9fR3UehCAo4CLfSS
tUlVp1/729CDt+NlMni0wnOY8P85Mic9JeWrfy4gNhI8C3udroLTbvtwpfC4Y7Z/
fQVxUJDjvY1/jkRsoJvA8e4bRkiCsyp9vt7THlWlwKbAiEmeLy0mr13n2ysOnxdl
auOPCr1hGbu8tH/C7LDcwL95vuKv3hm1/WTlDC6ZkGazpAnEa2A3gCWq/le4Z1at
oaIZBM6eDoWzT3hvBza8gJ1gmaWEIMX8Y4buBepEVqSuCflhuwIDAQABoDIwMAYJ
KoZIhvcNAQkOMSMwITARBglghkgBhvhCAQEEBAMCBkAwDAYDVR0TAQH/BAIwADAN
BgkqhkiG9w0BAQUFAAOCAQEAZK72YA7zM4ICNg50Hoq7hxaOqczDuEa4QZGHlNPA
jtAYNuGBKEvnZ8YpL4xbYjJ4XGkgqEDEwlB11fdMELXCslitUWIGNFl5Lbd25ROt
85ahsi31K3yOBZOyvD+qC4CIcMcjs/PSozN6BlPCfp4nlKgRYqLGrfFnlOgxdia3
NCase26vgD7nx5tLtpGtxlPuY6DWh0sl2zVrBUjWktddSQFbMhrl70wnucX8QRdi
zC5F0l1tX8mAJt3jXkClwqA/eHSQFpldI5iLTkZ4C0LQcCvSYl+wfFuH6Arkie0J
TztjzaZZh5G3fToQ1NXuEcCmzK/X11jRcrsnRfa1cF0m0g==
-----END CERTIFICATE REQUEST-----
Comment 32 matthew zeier [:mrz] 2008-09-11 16:01:05 PDT
Summary notes regarding this change over - 

1. Currently during the trial logs will be delayed ~24 hours.  CDNetworks is currently pushing all logs from their systems to a common place at the end of the day GMT.  

2. Log location for AMO is im-log02:/data/stats/logs/cdnetworks/addons.mozilla.org
Comment 33 matthew zeier [:mrz] 2008-09-11 19:05:36 PDT
Moved.  

;addons         60 IN CNAME addons.glb.mozilla.com.
addons          60 IN CNAME addons.mozilla.org.usgscac.cdnetworks.net.
Comment 34 Jeremy Orem [:oremj] 2008-09-11 19:28:29 PDT
Their servers stopped responding, for some reason, reverted.
Comment 35 matthew zeier [:mrz] 2008-09-11 20:55:02 PDT
fyi, testing load once more after some config changes on cdnetworks side.
Comment 36 matthew zeier [:mrz] 2008-09-22 11:44:09 PDT
Trial done and everything has been pulled back from CDNetworks.
Comment 37 Daniel Einspanjer [:dre] [:deinspanjer] 2008-10-14 11:02:03 PDT
If we move forward with a CDN solution, we need to evaluate the effect this will have on GeoIP resolution.  Unless we are willing to give up GeoIP resolution, we will need the CDN provider to pass through the actual IP address of the request in the logs we receive.
Comment 38 Jeremy Orem [:oremj] 2008-10-14 11:04:30 PDT
We should receive logs from the CDN provider that match up with our current log format.  Also, all proxies should pass through the X-forwarded-for header, which will contain the actual IP.
Comment 39 Daniel Einspanjer [:dre] [:deinspanjer] 2008-10-14 11:10:29 PDT
While the logs matched up fine, the IP address in the access logs did not seem to be the end user IP address, but rather the IP of the CDN node.  Maybe there could be some logging configuration that would take the address in teh X-forwarded-for header and place it into the access log instead?
Comment 40 matthew zeier [:mrz] 2008-11-07 09:49:14 PST
New summary to reflect new CDN trial, more details to come later.
Comment 41 Jeremy Orem [:oremj] 2008-12-11 14:19:39 PST
Closing these until there is something to do.

Note You need to log in before you can comment on or make changes to this bug.