We have a vm trial at an .ar isp, could spin off into physical rackspace. We need to setup a remote zlb cache there as soon as mrz has the vm details. cc'ing Jake, please work him in to this process.
Jake, late me know when Zeus has been installed on these servers.
Temp stealing this so I can keep track if it until hand-off to Jeremy. Nodes are added to inventory and DNS (ar-zlb0.mozilla.org), rhn_registered, yum updated. Attempting to puppetize right now, messing with mrepo's allowed IPs list.
Assignee: jeremy.orem+bugs → nmaul
It appears that puppet is not going to work- at present, we don't have any good way to puppetize nodes outside of our network, or keep them up-to-date. Instead, we're going to just install Zeus and run with it, outside of puppet. I've downloaded ZTM 7.3r0 into /usr/src and installed it on both. ar-zlb01 is completed, and has joined the cluster with no traffic groups assigned to it (note that it still needs a license at some point). ar-zlb02 is throwing this error during configuration: Machine ar-zlb01.mozilla.org was uncontactable using multicast messages. This will mean fault tolerance problems, and I think we will need to contact the hosting provider to have them look into this. Somehow, they have no problems reaching the other servers, but cannot multicast with each other. Some type of VMware config issue, perhaps... In the meantime, giving this back to Jeremy as it's more or less g2g except this. To (re)configure ar-zlb02, just do /usr/local/zeus/zxtm/configure.
Assignee: nmaul → jeremy.orem+bugs
Might not work at all with multicast - don't know what sort of VM infra they run. I'm okay running with just one node or two in a non-cluster environment. I've asked for two demo licenses.
ar-zlb01 is up and in the cluster. We need to decide now how this is actually going to work- ar-zlb01 and 02 are not on the backend network, so they cannot act as normal load balancers like the others. I see a few possibilities: 1) Get them on the backend network somehow (OpenVPN or similar). I think this is the most direct solution, given our environment, and the assumptions made by our existing configs. 2) Make some of the nodes directly accessible (give them public IPs w/ ACLs) so that AR nodes can reach them directly. This would be the obvious choice if our environment didn't make heavy use of NAT and private IPs. In 10 years we should be doing it this way, with IPv6. 3) Remove the AR nodes from the Zeus cluster and set them up standalone with the same external Zeus services that the main cluster has. Set up their "nodes" to be the existing Zeus cluster. Use 3crowd to direct traffic to either the .ar nodes or the main nodes. Effectively this is a 2-tiered design: ar->mpt->servers. Zeus is irrelevant here, any caching proxy would work. 3crowd is doing all the work. With either 1 or 2, we would want to use Zeus's Global Load Balancing. With 3, 3crowd does that.
jake - this would be just like amsterdam. The "backend" server is the frontend VIP. Set it up just like that. When I did Singapore, I literally exported the config from ams-zlb01 and imported. oremj had .sg in MSM I thought at one point. You can use the other VM with Zeus GLB and have it join the existing GLB cluster. Guess I'll find you on irc to chat more?
(In reply to comment #5) > ar-zlb01 is up and in the cluster. > > We need to decide now how this is actually going to work- ar-zlb01 and 02 > are not on the backend network, so they cannot act as normal load balancers > like the others. I see a few possibilities: We can just use single hosted IPs. As long as Zeus says they can talk to each other I think this will work. Zeus allows for per location settings. So, say you wanted to set up AMO on these: 1) Pick an IP in .ar 2) Set up a traffic group in .ar 3) Create a new pool that with addons public IP as the backend 4) Edit the AMO virtual server and set the .ar pool to the one created above 5) Make sure all the rules should be run in both locations (usually they should) 6) Use 3crowd or some sort of glb to send some traffic to the ip in step 1. I think that is everything. This was all sort of from memory, so I can clarify if needed.
This is largely configured, except for the actual GLB part: All of the pools / vservers on the AMO ZLB cluster have been updated so that they do not throw errors about hosts being down or the like, because ar-zlb01 can't reach them. All pools have been changed so that the AR location points to the IP in the Traffic Group bound to the MPT servers. All vservers except AMO are bound to no IPs in AR, because there's only 1 IP *in* AR to use. addons.mozilla.org uses it. The 2 AMO (SSL and non-SSL) vservers are bound to "All IPs" in AR, which is basically the one public IP plus a handful of private ones that are of no significance. FYI, it is not possible to make a Traffic Group encompassing the 1 IP on the server, because the LB is already bound to it. That's why I'm using "All IPs" instead of a Traffic Group. If you add this to your /etc/hosts file, you can test it out: 18.104.22.168 addons.mozilla.org I have asked Stephen to give it a run-through, and if it looks good we'll see about getting it set up for some traffic somehow.
Does GLB require you to have a local GLB install at each location? You may have to install that on the other VM to get it into GLB. Or just test this with static-cdn.addons.mozilla.org ?
please let metrics know the server names as they will appear in im-log02/3 and the site names that will have traffic hosted by those sites. we need the files to appear on the filers just like other zeus nodes. we can enable passive tracking of those logs as soon as the directories exist. when we are notivied that the sites are actively serving traffic, we can flip it to active tracking (which will notify if logs are missing for any hour). we should also confirm that our blacklist ips wiki page has these ips listed so we dont double count traffic from this site. pedro, when it supplies all this info, can you coordinate getting these into our processing?
dre, I think we decided over the weekend we're just going to test static-cdn.addons.mozilla.net. Do you care about metrics for that site?
Talked to dre offline, nothing for Metrics here if we're just testing static-cdn.addons.mozilla.net.
Created attachment 534957 [details] static-ar.addons.mozilla.net.csr This is ready for testing, however we don't have a usable cert for it. Attached is a CSR that should do the trick.
Dear Matthew Zeier, Congratulations! GeoTrust has approved your request for a Enterprise SSL certificate. Your certificate is included at the end of this email. INSTALLATION INSTRUCTIONS 1. INSTALL CERTIFICATE: Install the X.509 version of your certificate included at the end of this e-mail. For installation instructions for your SSL Certificate, go to: http://www.geotrust.com/support/installation-instructions/index.html 2. INTERMEDIATE CERTIFICATE ADVISORY: You MUST install the GeoTrust intermediate Certificate included at end of this e-mail on your server together with your Certificate or it may not operate correctly You can also get your GeoTrust intermediate Certificates at: https://knowledge.geotrust.com/support/knowledge-base/index?page=content&actp=CROSSLINK&id=AR1423 3. CHECK INSTALLATION: Ensure you have installed your certificate correctly at: https://knowledge.geotrust.com/support/knowledge-base/index?page=content&id=SO9557&actp=LIST 4. INSTALL THE GEOTRUST TRUE SITE SEAL: Additionally, as part of your SSL Certificate Service, you are entitled to display the GeoTrust True Site Seal - recognized across the Internet and around the world as a symbol of authenticity, security, and trust - to build consumer confidence in your Web site. Installation instructions for the GeoTrust True Site Seal can be found on the following link: https://www.geotrust.com/support/true-businessid/true-site-seal/ Visit the GeoTrust Support Web site, where you will find a range of support tools to help you: http://www.geotrust.com/support Best regards, GeoTrust Customer Support http://www.geotrust.com/support Hours of Operation: Mon - Fri 05:00 - 17:00 (PST) Email: firstname.lastname@example.org Web: http://www.geotrust.com Phone: 1-866-436-8787 or 1-678-366-8399 Live Chat: http://www.geotrust.com/support ** MICROSOFT IIS and TOMCAT USERS Microsoft and Tomcat users are advised to download a PKCS #7 formatted certificate from the GeoTrust User Portal: https://products.geotrust.com/orders/orderinformation/authentication.do. PKCS #7 is the default format used by these vendors during installation and includes the intermediate CA certificate, you may also install the below web server certificate and intermediate CA certificate individually. Web Server CERTIFICATE ----------------- -----BEGIN CERTIFICATE----- MIIEpTCCA42gAwIBAgIDAJ97MA0GCSqGSIb3DQEBBQUAMEAxCzAJBgNVBAYTAlVT MRcwFQYDVQQKEw5HZW9UcnVzdCwgSW5jLjEYMBYGA1UEAxMPR2VvVHJ1c3QgU1NM IENBMB4XDTExMDUyMzE0MzMwMFoXDTEyMDUyNTA0MTAyN1owgbcxKTAnBgNVBAUT IGZvWW5ZTkNVZDRjWWc2L3R4NC8xMm5BZzZTU0h4NUJ1MQswCQYDVQQGEwJVUzET MBEGA1UECBMKQ2FsaWZvcm5pYTEWMBQGA1UEBxMNTW91bnRhaW4gVmlldzEcMBoG A1UEChMTTW96aWxsYSBDb3Jwb3JhdGlvbjELMAkGA1UECxMCSVQxJTAjBgNVBAMT HHN0YXRpYy1hci5hZGRvbnMubW96aWxsYS5uZXQwggEiMA0GCSqGSIb3DQEBAQUA A4IBDwAwggEKAoIBAQCscb2APt5o0U8uCr3WrpBPpGlZnH3UH/QdNukFKewea7C4 69kf/03xSwYvPxwuD5cie+6j0LGmQrFREA52xS3w7PNg2VRpuygDuMuYkhhWPq2v //Xc9ELkiv47u9RV2uv1quZ23Km9nCTFEaR3a2zhUBsG+px0TbILh9P0QJoqfSr1 Glb2s6Y6RATNgz6tuE76uts5ulfKmiSOON98UXZadE5eTmgPCcSlPLLKVrQOvyJ1 YnxPha37TDJa1q95r1VI67CuQDmzX5JkUDznXNybBXRpEyBaluYaGt2vlr3ivfa1 VKEgVwvvyUSrskxZcYZGEa5As9Od7Wzr0K2mpUvTAgMBAAGjggEuMIIBKjAfBgNV HSMEGDAWgBRCeVQbYc1VKz5j1TxIV/Wf+0XOSjAOBgNVHQ8BAf8EBAMCBaAwHQYD VR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMCcGA1UdEQQgMB6CHHN0YXRpYy1h ci5hZGRvbnMubW96aWxsYS5uZXQwPQYDVR0fBDYwNDAyoDCgLoYsaHR0cDovL2d0 c3NsLWNybC5nZW90cnVzdC5jb20vY3Jscy9ndHNzbC5jcmwwHQYDVR0OBBYEFDVO /jCsFB85Ak3FLm6Wo5h2bvbDMAwGA1UdEwEB/wQCMAAwQwYIKwYBBQUHAQEENzA1 MDMGCCsGAQUFBzAChidodHRwOi8vZ3Rzc2wtYWlhLmdlb3RydXN0LmNvbS9ndHNz bC5jcnQwDQYJKoZIhvcNAQEFBQADggEBAG03/iD+pxLOEtLsh5JUbq41plcinbrz YJYeXkXu7K/SDfIdv8upTDVS52VOgjXqIMSxvFoIluxc9fm2FS/C6WmjkHhVteO5 toZ9RkioW/ELEP7KVwjGddxNXFgrpADigFVHQDiw4fe2wNyKRJdXkRIPb9rCtBsi Ad8zNtVnAW6P7Q5LWoggn4GXtoTHItJh/B+R7wquIkqb65ur+SbItw+XWZRDaORu C1+Dkuj0WWLX6NPms0MXcpbKM6yI6v/qEXbZj6VXIQoaZgke9J8Zi6BuecO+Xs3K ekIq/02rc0QuW2/GjYV5AAvZSaBRrctqm14LK5ln3k9piCsIgcMDQJU= -----END CERTIFICATE----- INTERMEDIATE CA: --------------------------------------- -----BEGIN CERTIFICATE----- MIID2TCCAsGgAwIBAgIDAjbQMA0GCSqGSIb3DQEBBQUAMEIxCzAJBgNVBAYTAlVT MRYwFAYDVQQKEw1HZW9UcnVzdCBJbmMuMRswGQYDVQQDExJHZW9UcnVzdCBHbG9i YWwgQ0EwHhcNMTAwMjE5MjIzOTI2WhcNMjAwMjE4MjIzOTI2WjBAMQswCQYDVQQG EwJVUzEXMBUGA1UEChMOR2VvVHJ1c3QsIEluYy4xGDAWBgNVBAMTD0dlb1RydXN0 IFNTTCBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAJCzgMHk5Uat cGA9uuUU3Z6KXot1WubKbUGlI+g5hSZ6p1V3mkihkn46HhrxJ6ujTDnMyz1Hr4Gu FmpcN+9FQf37mpc8oEOdxt8XIdGKolbCA0mEEoE+yQpUYGa5jFTk+eb5lPHgX3UR 8im55IaisYmtph6DKWOy8FQchQt65+EuDa+kvc3nsVrXjAVaDktzKIt1XTTYdwvh dGLicTBi2LyKBeUxY0pUiWozeKdOVSQdl+8a5BLGDzAYtDRN4dgjOyFbLTAZJQ50 96QhS6CkIMlszZhWwPKoXz4mdaAN+DaIiixafWcwqQ/RmXAueOFRJq9VeiS+jDkN d53eAsMMvR8CAwEAAaOB2TCB1jAOBgNVHQ8BAf8EBAMCAQYwHQYDVR0OBBYEFEJ5 VBthzVUrPmPVPEhX9Z/7Rc5KMB8GA1UdIwQYMBaAFMB6mGiNifurBWQMEX2qfWW4 ysxOMBIGA1UdEwEB/wQIMAYBAf8CAQAwOgYDVR0fBDMwMTAvoC2gK4YpaHR0cDov L2NybC5nZW90cnVzdC5jb20vY3Jscy9ndGdsb2JhbC5jcmwwNAYIKwYBBQUHAQEE KDAmMCQGCCsGAQUFBzABhhhodHRwOi8vb2NzcC5nZW90cnVzdC5jb20wDQYJKoZI hvcNAQEFBQADggEBANTvU4ToGr2hiwTAqfVfoRB4RV2yV2pOJMtlTjGXkZrUJPji J2ZwMZzBYlQG55cdOprApClICq8kx6jEmlTBfEx4TCtoLF0XplR4TEbigMMfOHES 0tdT41SFULgCy+5jOvhWiU1Vuy7AyBh3hjELC3DwfjWDpCoTZFZnNF0WX3OsewYk 2k9QbSqr0E1TQcKOu3EDSSmGGM8hQkx0YlEVxW+o78Qn5Rsz3VqI138S0adhJR/V 4NwdzxoQ2KDLX4z6DOW/cf/lXUQdpj6HR/oaToODEj+IZpWYeZqF6wJHzSXj8gYE TpnKXKBuervdo5AaRTPvvz7SBMS24CqFZUE+ENQ= -----END CERTIFICATE-----
Assignee: mrz → nmaul
This is online. More specifically, the Firefox Addons Manager traffic in AR is being directed to static-ar.addons.mozilla.net, instead of static-cdn. Note that this *only* affects the addons manager... visiting addons.mozilla.org still generates links to static-cdn.addons.mozilla.net, regardless of location. However at this point all the groundwork has been laid- it would be fairly simple to set this up for the main site as well.
Status: NEW → ASSIGNED
AR, BR, and CL have been set up for this now. Overall it seems to be working well. 2 minor problems: 1) Some traffic is not being redirected as expected. In particular, Gomez testing is not being redirected, even though their nodes are detected as being in BR and AR, respectively. I believe this may be an issue with how Gomez works, rather than a problem on our end. 2) Seeing a bit higher volume of hits from non-redirected countries than would have been expected. In particular, we're getting a decent amount of traffic from Spain and Mexico. Best guess on this is that this is due to some odd client-side proxying, causing some users in these locations to be sent to the wrong place. Investigating possible solutions for either problem.
Jake - before we go into the weekend, let's tear this down. It's not HA and we don't have any monitoring. I think for the sake of testing, we're done right?
Aye, I think we're done. The only thing I don't have is solid evidence of a speed improvement, since Gomez isn't testing it usefully. Did you hear back from anyone in AR? I will remove the rules this afternoon.
Nothing from Gomez yet, still have a case open.
These rules have been disabled and the cache for services.addons.mozilla.org has been flushed. AR/BR/CL traffic is back to normal channels (static-cdn). Assigning to mrz for further direction (probably close out until we get a more long-term plan).
Assignee: nmaul → mrz
Nothing left to do here.
Status: ASSIGNED → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.