Closed Bug 450745 Opened 17 years ago Closed 17 years ago

move addons.mozilla.org behind Akamai

Categories

(mozilla.org Graveyard :: Server Operations: Projects, task)

Product:
mozilla.org Graveyard
Component:
Server Operations: Projects
Platform:
All
Other
Type:
task
Priority:
Not set
Severity:
minor

Tracking

(Not tracked)

Status:
RESOLVED INCOMPLETE

People

(Reporter: mrz, Assigned: oremj)

Details

Attachments

(1 file)

Screenshot of successful upload on 64.245.124.43
190.35 KB, image/png
Details
Need CSR.
Assignee: server-ops → reed
Per my earlier comments in bug 449610, I think we should seriously consider getting an EV SSL certificate for the addons.mozilla.org site, especially if the plan is to get a new SSL certificate for the site anyway (i.e., it wouldn't continue to use the existing *.mozilla.org wildcard). AMO is an end-user site, the default mode of interacting with the site is via SSL (http URLs being redirected to https), and IMO we have an interest in assuring end users that it is in fact an official Mozilla site. Also, if we do get an EV cert for AMO arguably the site owner should be listed as "Mozilla" (not "Mozilla Foundation" or "Mozilla Corporation") assuming the CA is willing to do this. This is consistent with the text on the site itself.
http://www.geotrust.com/products/ssl_certificates/true_businessid_ev.asp is GeoTrust's EV "product". I'll generate the CSR with an O of Mozilla.
I can't add much to what Frank's said here - even though AMO has third party software all over it, we try very hard throughout the site to ensure that they are getting the real goods (hence SSL everywhere) and that they know it's a mozilla-maintained site - our trust relationship is on the line: that's where identity UI is intended to be used. Obviously you guys need to make the call on whether you're willing to go through the more extended vetting, and I've never been through one, so maybe it's reasonably painless, maybe not, but in principle if I could just make an EV cert magically appear, AMO is probably the Mozilla site I'd choose.
Subject: C=US, ST=California, L=Mountain View, O=Mozilla, OU=Add-ons CDN, CN=addons.mozilla.org/emailAddress=hostmaster@mozilla.com -----BEGIN CERTIFICATE REQUEST----- MIIDHjCCAgYCAQAwgaYxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlh MRYwFAYDVQQHEw1Nb3VudGFpbiBWaWV3MRAwDgYDVQQKEwdNb3ppbGxhMRQwEgYD VQQLEwtBZGQtb25zIENETjEbMBkGA1UEAxMSYWRkb25zLm1vemlsbGEub3JnMSUw IwYJKoZIhvcNAQkBFhZob3N0bWFzdGVyQG1vemlsbGEuY29tMIIBIjANBgkqhkiG 9w0BAQEFAAOCAQ8AMIIBCgKCAQEAt3ZlNEGSfTA6V8SD+OL3+Y0kSzGDivuQm0ah 1c9Lv7gild2H218mkkYol2G2XvGoDkPI8eUDEZ1e7G401Zwp9fR3UehCAo4CLfSS tUlVp1/729CDt+NlMni0wnOY8P85Mic9JeWrfy4gNhI8C3udroLTbvtwpfC4Y7Z/ fQVxUJDjvY1/jkRsoJvA8e4bRkiCsyp9vt7THlWlwKbAiEmeLy0mr13n2ysOnxdl auOPCr1hGbu8tH/C7LDcwL95vuKv3hm1/WTlDC6ZkGazpAnEa2A3gCWq/le4Z1at oaIZBM6eDoWzT3hvBza8gJ1gmaWEIMX8Y4buBepEVqSuCflhuwIDAQABoDIwMAYJ KoZIhvcNAQkOMSMwITARBglghkgBhvhCAQEEBAMCBkAwDAYDVR0TAQH/BAIwADAN BgkqhkiG9w0BAQUFAAOCAQEAZK72YA7zM4ICNg50Hoq7hxaOqczDuEa4QZGHlNPA jtAYNuGBKEvnZ8YpL4xbYjJ4XGkgqEDEwlB11fdMELXCslitUWIGNFl5Lbd25ROt 85ahsi31K3yOBZOyvD+qC4CIcMcjs/PSozN6BlPCfp4nlKgRYqLGrfFnlOgxdia3 NCase26vgD7nx5tLtpGtxlPuY6DWh0sl2zVrBUjWktddSQFbMhrl70wnucX8QRdi zC5F0l1tX8mAJt3jXkClwqA/eHSQFpldI5iLTkZ4C0LQcCvSYl+wfFuH6Arkie0J TztjzaZZh5G3fToQ1NXuEcCmzK/X11jRcrsnRfa1cF0m0g== -----END CERTIFICATE REQUEST-----
Assignee: reed → mrz
Don't have a login, over to justin. This needs to be an EV cert.
Assignee: mrz → justin
no on the EV cert per discussions. on order.
-----BEGIN CERTIFICATE----- MIIDhTCCAu6gAwIBAgIDCa+2MA0GCSqGSIb3DQEBBQUAME4xCzAJBgNVBAYTAlVT MRAwDgYDVQQKEwdFcXVpZmF4MS0wKwYDVQQLEyRFcXVpZmF4IFNlY3VyZSBDZXJ0 aWZpY2F0ZSBBdXRob3JpdHkwHhcNMDgwODE5MTM0NDM5WhcNMTAwODIwMTM0NDM5 WjCBizELMAkGA1UEBhMCVVMxEzARBgNVBAgTCkNhbGlmb3JuaWExFjAUBgNVBAcT DU1vdW50YWluIFZpZXcxHDAaBgNVBAoTE01vemlsbGEgQ29ycG9yYXRpb24xFDAS BgNVBAsTC0FkZC1vbnMgQ0ROMRswGQYDVQQDExJhZGRvbnMubW96aWxsYS5vcmcw ggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQC3dmU0QZJ9MDpXxIP44vf5 jSRLMYOK+5CbRqHVz0u/uCKV3YfbXyaSRiiXYbZe8agOQ8jx5QMRnV7sbjTVnCn1 9HdR6EICjgIt9JK1SVWnX/vb0IO342UyeLTCc5jw/zkyJz0l5at/LiA2EjwLe52u gtNu+3Cl8Lhjtn99BXFQkOO9jX+ORGygm8Dx7htGSIKzKn2+3tMeVaXApsCISZ4v LSavXefbKw6fF2Vq448KvWEZu7y0f8LssNzAv3m+4q/eGbX9ZOUMLpmQZrOkCcRr YDeAJar+V7hnVq2hohkEzp4OhbNPeG8HNryAnWCZpYQgxfxjhu4F6kRWpK4J+WG7 AgMBAAGjga4wgaswDgYDVR0PAQH/BAQDAgTwMB0GA1UdDgQWBBTWol0U4Jm4Gb0q OtJqA0cy9QtBYTA6BgNVHR8EMzAxMC+gLaArhilodHRwOi8vY3JsLmdlb3RydXN0 LmNvbS9jcmxzL3NlY3VyZWNhLmNybDAfBgNVHSMEGDAWgBRI5mj5K9KylddH2CMg EE8zmJCf1DAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDQYJKoZIhvcN AQEFBQADgYEAY1mIQBnED44I+Mtsswe4Iz79jD19yCBIQJGvxBPXTyeaX+nhJXCh Z7w7IfGdN0UcMBA5n6AgqIG0ft+jclKctJImdSQuXvSzNbv1IGowPyYUJNRG/MOa OBX++vEm18UANh6uul7pvBqWqr6PyuBTbTmdQqff/EUw5MAOndtyp2w= -----END CERTIFICATE-----
Assignee: justin → oremj
Certs have been handed off to CDNetworks.
Thinking this should be extended to include versioncheck.addons.mozilla.org and services.addons.mozilla.org. Haven't a clue how much bandwidth that that would equate to (and probably more as folks upgrade to Fx3?). Any reason to no do that? How dependent are we on logs for those?
AMO gets 20x the traffic of either services or versioncheck. Likely to increase of course.
For numbers - versioncheck & services are each about 4Mbps.
(In reply to comment #9) > Thinking this should be extended to include versioncheck.addons.mozilla.org and > services.addons.mozilla.org. Haven't a clue how much bandwidth that that would > equate to (and probably more as folks upgrade to Fx3?). > > Any reason to no do that? How dependent are we on logs for those? We're extremely dependent on logs from versioncheck for AMO. I don't know about SAMO.
Don't want to hijack this bug but Justin can you explain the rationale of why NOT to get an EV cert (as you said in comment 6)? I must have missed that discussion.
Site up for testing @ root@mrz-mp (/Users/mrz/) 1> host addons.mozilla.org.usgscac.cdnetworks.net addons.mozilla.org.usgscac.cdnetworks.net has address 64.245.124.40 addons.mozilla.org.usgscac.cdnetworks.net has address 64.245.124.43
Assignee: oremj → stephen.donner
mrz: r=stephend for these two instances; tested quite liberally, with no issues found. * Made sure to sign up on both boxes, change password, log in, etc. * Also tested changes to my test add-on at https://addons.mozilla.org/en-US/developers/edit/7431, (author addition, preview image swapping, etc.), from both this site -> production, and ensured that changes in production -> these boxes took.
I can't access the sites for some reason (tried editing my hosts file and everything), so Stephen - can you or did you try uploading a new version of your test add-on?
(In reply to comment #12) > (In reply to comment #9) > > Thinking this should be extended to include versioncheck.addons.mozilla.org and > > services.addons.mozilla.org. Haven't a clue how much bandwidth that that would > > equate to (and probably more as folks upgrade to Fx3?). > > > > Any reason to no do that? How dependent are we on logs for those? > > We're extremely dependent on logs from versioncheck for AMO. I don't know about > SAMO. > Log loss is definitely not okay for a number of URLs with AMO and all of versioncheck.AMO. There are no current uses of logs for SAMO that I know of, but there will be in the coming months. Can someone explain how this setup with CDNetworks works?
> Log loss is definitely not okay for a number of URLs with AMO and all of > versioncheck.AMO. There are no current uses of logs for SAMO that I know of, > but there will be in the coming months. Log loss would be temporary and not even a full log loss. The current log format from CDNetworks doesn't match what we're used to but we are getting logs. The are working on matching our log format, however. This is somewhat moot though since they aren't even configured for *.addons.mozilla.org and probably won't until after we've confirmed the log format for AMO works and then decide if that offloads enough traffic to make it worthwhile. > Can someone explain how this setup with CDNetworks works? Think of them as a giant distributed Netscaler - the proxy back to the origin server which in this case is the production San Jose Netscaler. Otherwise they aren't much different than Netscaler (I took some liberty in that statement). Did that answer your question?
Assignee: stephen.donner → mrz
Ok, some additional questions: - Are we going to be using them all the time or only in high load when we can't handle traffic ourselves? - How does .NL play into this? is it useless now? - Can we flush their caches, and how long do they cache for? - Are all of the AMO rules of the netscaler in effect on their side? (In reply to comment #19) > Log loss would be temporary and not even a full log loss. The current log > format from CDNetworks doesn't match what we're used to but we are getting > logs. The are working on matching our log format, however. It would need to match pretty much exactly for the stats script to parse it. > This is somewhat moot though since they aren't even configured for > *.addons.mozilla.org and probably won't until after we've confirmed the log > format for AMO works and then decide if that offloads enough traffic to make it > worthwhile. Firefox 2 and lower is still using addons.mozilla.org for versioncheck, so the majority of our active user stats are still hitting addons.mozilla.org, not versioncheck.amo.
(In reply to comment #20) > Ok, some additional questions: > > - Are we going to be using them all the time or only in high load when we can't > handle traffic ourselves? > All the time. > - How does .NL play into this? is it useless now? We will no longer need the .NL cache. > > - Can we flush their caches, and how long do they cache for? We can flush their caches and they cache as long as cache headers tell them to. > > - Are all of the AMO rules of the netscaler in effect on their side? They will be. > > (In reply to comment #19) > > Log loss would be temporary and not even a full log loss. The current log > > format from CDNetworks doesn't match what we're used to but we are getting > > logs. The are working on matching our log format, however. > It would need to match pretty much exactly for the stats script to parse it. They are working on matching our format.
Ok, everything sounds good to me then, as long as they match our format before we switch. My only other question is the one that Basil already asked in comment #13 about why we're not getting an EV cert. My pizza place has an EV cert. ( https://www.papajohnsonline.com ) Mozilla has enough problems with people trying to pass off unofficial versions of Firefox as the real thing. AMO's source is available for anyone to use, which means I can make something that looks *exactly* like AMO on my own website in about 5 minutes and distribute virus add-ons from it. I think we should be as secure and identifiable as possible when distributing add-ons, which right now means a green "Mozilla" in the location bar.
Since this is going live tonight, have they been able to match our log format? And what, if any, will be the delay before we get their logs? Also, what directory are the logs going to be stored in?
Postponing till next Tuesday.
Flags: needs-downtime+
(In reply to comment #23) > Since this is going live tonight, have they been able to match our log format? Yes of course, this was a requirement. > And what, if any, will be the delay before we get their logs? Quoting, "logs will be avaialbe a day after the service in GMT time." Logs are pulled from each node and pushed into a central repository for us to rsync over. What do you have that uses AMO logs? Can that just be integrated into stats processing so it's not something you have to deal with?
Our download/active user counter parses the logs every day for developer stats. The system was almost done when oremj's addons stats stuff was done, so the plan was to switch our system over to using his somehow, but that's probably not going to happen soon (especially since I'm not in webdev anymore). So any new directories of logs to be parsed have to be added to the scripts on dm-stats01. Having a day delay is gonna cause some stats abnormalities that I guess we'll have to deal with. For example, instead of seeing the number of downloads of an add-on from Tuesday on Wednesday, we'll see a very small number of downloads that hit our netscaler, and then Thursday we'll see the rest of Tuesday show up. It's not as much of a problem with downloads, but the active user counter only runs on Thursdays to count Wednesday's logs for Wednesdays only. So I guess we'd need to move that to Friday to count Wednesdays.
> So any new directories of logs to be parsed have to be added to the scripts on > dm-stats01. Is this process documented somewhere? I feel like you're the only one that knows anything about it. > Having a day delay is gonna cause some stats abnormalities that I guess we'll > have to deal with. For example, instead of seeing the number of downloads of an > add-on from Tuesday on Wednesday, we'll see a very small number of downloads > that hit our netscaler, and then Thursday we'll see the rest of Tuesday show > up. You won't even count the Netscaler logs - only thing hitting it will be CDNetworks and what does make it through to the Netscaler (in other words, wasn't already cached) will be counted twice when the CDNetworks logs are imported.
Depends on: 453729
(In reply to comment #13) > Justin can you explain the rationale of why NOT to get an EV cert (as you > said in comment 6)? I must have missed that discussion. Ditto -- where's the discussion?
EV cert decision seems to have been reversed - see bug 452314. Not sure how that relates to this bug.
(In reply to comment #29) > EV cert decision seems to have been reversed - see bug 452314. Not sure how > that relates to this bug. It related to this bug in the sense that I'm holding off on the EV cert until I decide if CDNetworks is right or not. I don't want to get an EV cert for a trial.
Unless there are objections with the Subject, you can use this CSR: Subject: C=US, ST=California, L=Mountain View, O=Mozilla, OU=Add-ons CDN, CN=addons.mozilla.org/emailAddress=hostmaster@mozilla.com -----BEGIN CERTIFICATE REQUEST----- MIIDHjCCAgYCAQAwgaYxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlh MRYwFAYDVQQHEw1Nb3VudGFpbiBWaWV3MRAwDgYDVQQKEwdNb3ppbGxhMRQwEgYD VQQLEwtBZGQtb25zIENETjEbMBkGA1UEAxMSYWRkb25zLm1vemlsbGEub3JnMSUw IwYJKoZIhvcNAQkBFhZob3N0bWFzdGVyQG1vemlsbGEuY29tMIIBIjANBgkqhkiG 9w0BAQEFAAOCAQ8AMIIBCgKCAQEAt3ZlNEGSfTA6V8SD+OL3+Y0kSzGDivuQm0ah 1c9Lv7gild2H218mkkYol2G2XvGoDkPI8eUDEZ1e7G401Zwp9fR3UehCAo4CLfSS tUlVp1/729CDt+NlMni0wnOY8P85Mic9JeWrfy4gNhI8C3udroLTbvtwpfC4Y7Z/ fQVxUJDjvY1/jkRsoJvA8e4bRkiCsyp9vt7THlWlwKbAiEmeLy0mr13n2ysOnxdl auOPCr1hGbu8tH/C7LDcwL95vuKv3hm1/WTlDC6ZkGazpAnEa2A3gCWq/le4Z1at oaIZBM6eDoWzT3hvBza8gJ1gmaWEIMX8Y4buBepEVqSuCflhuwIDAQABoDIwMAYJ KoZIhvcNAQkOMSMwITARBglghkgBhvhCAQEEBAMCBkAwDAYDVR0TAQH/BAIwADAN BgkqhkiG9w0BAQUFAAOCAQEAZK72YA7zM4ICNg50Hoq7hxaOqczDuEa4QZGHlNPA jtAYNuGBKEvnZ8YpL4xbYjJ4XGkgqEDEwlB11fdMELXCslitUWIGNFl5Lbd25ROt 85ahsi31K3yOBZOyvD+qC4CIcMcjs/PSozN6BlPCfp4nlKgRYqLGrfFnlOgxdia3 NCase26vgD7nx5tLtpGtxlPuY6DWh0sl2zVrBUjWktddSQFbMhrl70wnucX8QRdi zC5F0l1tX8mAJt3jXkClwqA/eHSQFpldI5iLTkZ4C0LQcCvSYl+wfFuH6Arkie0J TztjzaZZh5G3fToQ1NXuEcCmzK/X11jRcrsnRfa1cF0m0g== -----END CERTIFICATE REQUEST-----
Summary notes regarding this change over - 1. Currently during the trial logs will be delayed ~24 hours. CDNetworks is currently pushing all logs from their systems to a common place at the end of the day GMT. 2. Log location for AMO is im-log02:/data/stats/logs/cdnetworks/addons.mozilla.org
Moved. ;addons 60 IN CNAME addons.glb.mozilla.com. addons 60 IN CNAME addons.mozilla.org.usgscac.cdnetworks.net.
Their servers stopped responding, for some reason, reverted.
fyi, testing load once more after some config changes on cdnetworks side.
Trial done and everything has been pulled back from CDNetworks.
Summary: move addons.mozilla.org behind CDNetworks → move addons.mozilla.org behind CDN
Component: Server Operations → Server Operations: Projects
If we move forward with a CDN solution, we need to evaluate the effect this will have on GeoIP resolution. Unless we are willing to give up GeoIP resolution, we will need the CDN provider to pass through the actual IP address of the request in the logs we receive.
We should receive logs from the CDN provider that match up with our current log format. Also, all proxies should pass through the X-forwarded-for header, which will contain the actual IP.
While the logs matched up fine, the IP address in the access logs did not seem to be the end user IP address, but rather the IP of the CDN node. Maybe there could be some logging configuration that would take the address in teh X-forwarded-for header and place it into the access log instead?
New summary to reflect new CDN trial, more details to come later.
Assignee: mrz → oremj
No longer depends on: 453729
Summary: move addons.mozilla.org behind CDN → move addons.mozilla.org behind Akamai
Closing these until there is something to do.
Status: NEW → RESOLVED
Closed: 17 years ago
Resolution: --- → INCOMPLETE
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: