Spin up zlb caching node in .ar

RESOLVED FIXED

Status

mozilla.org Graveyard
Server Operations
RESOLVED FIXED
7 years ago
3 years ago

People

(Reporter: cshields, Assigned: mrz)

Tracking

Details

Attachments

(1 attachment)

(Reporter)

Description

7 years ago
We have a vm trial at an .ar isp, could spin off into physical rackspace.  We need to setup a remote zlb cache there as soon as mrz has the vm details.  cc'ing Jake, please work him in to this process.
(Assignee)

Updated

7 years ago
Group: infra

Comment 1

7 years ago
Jake, late me know when Zeus has been installed on these servers.

Comment 2

7 years ago
Temp stealing this so I can keep track if it until hand-off to Jeremy.

Nodes are added to inventory and DNS (ar-zlb0[12].mozilla.org), rhn_registered, yum updated. Attempting to puppetize right now, messing with mrepo's allowed IPs list.
Assignee: jeremy.orem+bugs → nmaul

Comment 3

7 years ago
It appears that puppet is not going to work- at present, we don't have any good way to puppetize nodes outside of our network, or keep them up-to-date.

Instead, we're going to just install Zeus and run with it, outside of puppet.

I've downloaded ZTM 7.3r0 into /usr/src and installed it on both.

ar-zlb01 is completed, and has joined the cluster with no traffic groups assigned to it (note that it still needs a license at some point).

ar-zlb02 is throwing this error during configuration:
Machine ar-zlb01.mozilla.org was uncontactable using multicast messages.

This will mean fault tolerance problems, and I think we will need to contact the hosting provider to have them look into this. Somehow, they have no problems reaching the other servers, but cannot multicast with each other. Some type of VMware config issue, perhaps...

In the meantime, giving this back to Jeremy as it's more or less g2g except this. To (re)configure ar-zlb02, just do /usr/local/zeus/zxtm/configure.
Assignee: nmaul → jeremy.orem+bugs
(Assignee)

Comment 4

7 years ago
Might not work at all with multicast - don't know what sort of VM infra they run.  I'm okay running with just one node or two in a non-cluster environment.

I've asked for two demo licenses.
(Assignee)

Updated

7 years ago
Whiteboard: [waiting for vm details]

Comment 5

7 years ago
ar-zlb01 is up and in the cluster.

We need to decide now how this is actually going to work- ar-zlb01 and 02 are not on the backend network, so they cannot act as normal load balancers like the others. I see a few possibilities:


1) Get them on the backend network somehow (OpenVPN or similar). I think this is the most direct solution, given our environment, and the assumptions made by our existing configs.

2) Make some of the nodes directly accessible (give them public IPs w/ ACLs) so that AR nodes can reach them directly. This would be the obvious choice if our environment didn't make heavy use of NAT and private IPs. In 10 years we should be doing it this way, with IPv6.

3) Remove the AR nodes from the Zeus cluster and set them up standalone with the same external Zeus services that the main cluster has. Set up their "nodes" to be the existing Zeus cluster. Use 3crowd to direct traffic to either the .ar nodes or the main nodes. Effectively this is a 2-tiered design: ar->mpt->servers. Zeus is irrelevant here, any caching proxy would work. 3crowd is doing all the work.


With either 1 or 2, we would want to use Zeus's Global Load Balancing. With 3, 3crowd does that.
(Assignee)

Comment 6

7 years ago
jake - this would be just like amsterdam.  The "backend" server is the frontend VIP.  Set it up just like that.

When I did Singapore, I literally exported the config from ams-zlb01 and imported.  oremj had .sg in MSM I thought at one point. 

You can use the other VM with Zeus GLB and have it join the existing GLB cluster.  Guess I'll find you on irc to chat more?

Comment 7

7 years ago
(In reply to comment #5)
> ar-zlb01 is up and in the cluster.
> 
> We need to decide now how this is actually going to work- ar-zlb01 and 02
> are not on the backend network, so they cannot act as normal load balancers
> like the others. I see a few possibilities:
We can just use single hosted IPs. As long as Zeus says they can talk to each other I think this will work.


Zeus allows for per location settings. So, say you wanted to set up AMO on these:

1) Pick an IP in .ar
2) Set up a traffic group in .ar
3) Create a new pool that with addons public IP as the backend
4) Edit the AMO virtual server and set the .ar pool to the one created above
5) Make sure all the rules should be run in both locations (usually they should)
6) Use 3crowd or some sort of glb to send some traffic to the ip in step 1.


I think that is everything. This was all sort of from memory, so I can clarify if needed.

Comment 8

7 years ago
This is largely configured, except for the actual GLB part:

All of the pools / vservers on the AMO ZLB cluster have been updated so that they do not throw errors about hosts being down or the like, because ar-zlb01 can't reach them.

All pools have been changed so that the AR location points to the IP in the Traffic Group bound to the MPT servers.

All vservers except AMO are bound to no IPs in AR, because there's only 1 IP *in* AR to use. addons.mozilla.org uses it.

The 2 AMO (SSL and non-SSL) vservers are bound to "All IPs" in AR, which is basically the one public IP plus a handful of private ones that are of no significance.


FYI, it is not possible to make a Traffic Group encompassing the 1 IP on the server, because the LB is already bound to it. That's why I'm using "All IPs" instead of a Traffic Group.


If you add this to your /etc/hosts file, you can test it out:
190.210.142.92	addons.mozilla.org

I have asked Stephen to give it a run-through, and if it looks good we'll see about getting it set up for some traffic somehow.
(Assignee)

Comment 9

7 years ago
Does GLB require you to have a local GLB install at each location?  You may have to install that on the other VM to get it into GLB.

Or just test this with static-cdn.addons.mozilla.org ?
(Reporter)

Updated

7 years ago
Assignee: jeremy.orem+bugs → nmaul
please let metrics know the server names as they will appear in im-log02/3 and the site names that will have traffic hosted by those sites.

we need the files to appear on the filers just like other zeus nodes. we can enable passive tracking of those logs as soon as the directories exist. when we are notivied that the sites are actively serving traffic, we can flip it to active tracking (which will notify if logs are missing for any hour).

we should also confirm that our blacklist ips wiki page has these ips listed so we dont double count traffic from this site.

pedro, when it supplies all this info, can you coordinate getting these into our processing?
(Assignee)

Comment 11

7 years ago
dre, I think we decided over the weekend we're just going to test static-cdn.addons.mozilla.net.  Do you care about metrics for that site?
(Assignee)

Comment 12

7 years ago
Talked to dre offline, nothing for Metrics here if we're just testing static-cdn.addons.mozilla.net.

Comment 13

7 years ago
Created attachment 534957 [details]
static-ar.addons.mozilla.net.csr

This is ready for testing, however we don't have a usable cert for it. Attached is a CSR that should do the trick.

Updated

7 years ago
Assignee: nmaul → mrz
(Assignee)

Comment 14

7 years ago
Dear Matthew Zeier,

Congratulations! GeoTrust has approved your request for a Enterprise SSL certificate. Your certificate is included at the end of this email.

INSTALLATION INSTRUCTIONS

1. INSTALL CERTIFICATE:
Install the X.509 version of your certificate included at the end of this e-mail.
For installation instructions for your SSL Certificate, go to:
http://www.geotrust.com/support/installation-instructions/index.html

2. INTERMEDIATE CERTIFICATE ADVISORY:
You MUST install the GeoTrust intermediate Certificate included at end of this e-mail on your server together with your Certificate or it may not operate correctly

You can also get your GeoTrust intermediate Certificates at:
https://knowledge.geotrust.com/support/knowledge-base/index?page=content&actp=CROSSLINK&id=AR1423

3. CHECK INSTALLATION:
Ensure you have installed your certificate correctly at:
https://knowledge.geotrust.com/support/knowledge-base/index?page=content&id=SO9557&actp=LIST

4. INSTALL THE GEOTRUST TRUE SITE SEAL:
Additionally, as part of your SSL Certificate Service, you are entitled to display the GeoTrust True Site Seal - recognized across the Internet and around the world as a symbol of authenticity, security, and trust - to build consumer confidence in your Web site.

Installation instructions for the GeoTrust True Site Seal can be found on the following link:
https://www.geotrust.com/support/true-businessid/true-site-seal/

Visit the GeoTrust Support Web site, where you will find a range of support tools to help you:
http://www.geotrust.com/support

Best regards,

GeoTrust Customer Support
http://www.geotrust.com/support
Hours of Operation: Mon - Fri 05:00 - 17:00 (PST)
Email:     esslorders@geotrust.com
Web:       http://www.geotrust.com
Phone:     1-866-436-8787 or 1-678-366-8399
Live Chat: http://www.geotrust.com/support


** MICROSOFT IIS and TOMCAT USERS
Microsoft and Tomcat users are advised to download a PKCS #7 formatted certificate from the GeoTrust User Portal:
https://products.geotrust.com/orders/orderinformation/authentication.do. PKCS #7 is the default format used by these vendors during installation and includes the intermediate CA certificate, you may also install the below web server certificate and intermediate CA certificate individually.

Web Server CERTIFICATE
-----------------

-----BEGIN CERTIFICATE-----
MIIEpTCCA42gAwIBAgIDAJ97MA0GCSqGSIb3DQEBBQUAMEAxCzAJBgNVBAYTAlVT
MRcwFQYDVQQKEw5HZW9UcnVzdCwgSW5jLjEYMBYGA1UEAxMPR2VvVHJ1c3QgU1NM
IENBMB4XDTExMDUyMzE0MzMwMFoXDTEyMDUyNTA0MTAyN1owgbcxKTAnBgNVBAUT
IGZvWW5ZTkNVZDRjWWc2L3R4NC8xMm5BZzZTU0h4NUJ1MQswCQYDVQQGEwJVUzET
MBEGA1UECBMKQ2FsaWZvcm5pYTEWMBQGA1UEBxMNTW91bnRhaW4gVmlldzEcMBoG
A1UEChMTTW96aWxsYSBDb3Jwb3JhdGlvbjELMAkGA1UECxMCSVQxJTAjBgNVBAMT
HHN0YXRpYy1hci5hZGRvbnMubW96aWxsYS5uZXQwggEiMA0GCSqGSIb3DQEBAQUA
A4IBDwAwggEKAoIBAQCscb2APt5o0U8uCr3WrpBPpGlZnH3UH/QdNukFKewea7C4
69kf/03xSwYvPxwuD5cie+6j0LGmQrFREA52xS3w7PNg2VRpuygDuMuYkhhWPq2v
//Xc9ELkiv47u9RV2uv1quZ23Km9nCTFEaR3a2zhUBsG+px0TbILh9P0QJoqfSr1
Glb2s6Y6RATNgz6tuE76uts5ulfKmiSOON98UXZadE5eTmgPCcSlPLLKVrQOvyJ1
YnxPha37TDJa1q95r1VI67CuQDmzX5JkUDznXNybBXRpEyBaluYaGt2vlr3ivfa1
VKEgVwvvyUSrskxZcYZGEa5As9Od7Wzr0K2mpUvTAgMBAAGjggEuMIIBKjAfBgNV
HSMEGDAWgBRCeVQbYc1VKz5j1TxIV/Wf+0XOSjAOBgNVHQ8BAf8EBAMCBaAwHQYD
VR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMCMCcGA1UdEQQgMB6CHHN0YXRpYy1h
ci5hZGRvbnMubW96aWxsYS5uZXQwPQYDVR0fBDYwNDAyoDCgLoYsaHR0cDovL2d0
c3NsLWNybC5nZW90cnVzdC5jb20vY3Jscy9ndHNzbC5jcmwwHQYDVR0OBBYEFDVO
/jCsFB85Ak3FLm6Wo5h2bvbDMAwGA1UdEwEB/wQCMAAwQwYIKwYBBQUHAQEENzA1
MDMGCCsGAQUFBzAChidodHRwOi8vZ3Rzc2wtYWlhLmdlb3RydXN0LmNvbS9ndHNz
bC5jcnQwDQYJKoZIhvcNAQEFBQADggEBAG03/iD+pxLOEtLsh5JUbq41plcinbrz
YJYeXkXu7K/SDfIdv8upTDVS52VOgjXqIMSxvFoIluxc9fm2FS/C6WmjkHhVteO5
toZ9RkioW/ELEP7KVwjGddxNXFgrpADigFVHQDiw4fe2wNyKRJdXkRIPb9rCtBsi
Ad8zNtVnAW6P7Q5LWoggn4GXtoTHItJh/B+R7wquIkqb65ur+SbItw+XWZRDaORu
C1+Dkuj0WWLX6NPms0MXcpbKM6yI6v/qEXbZj6VXIQoaZgke9J8Zi6BuecO+Xs3K
ekIq/02rc0QuW2/GjYV5AAvZSaBRrctqm14LK5ln3k9piCsIgcMDQJU=
-----END CERTIFICATE-----


INTERMEDIATE CA:
---------------------------------------

-----BEGIN CERTIFICATE-----
MIID2TCCAsGgAwIBAgIDAjbQMA0GCSqGSIb3DQEBBQUAMEIxCzAJBgNVBAYTAlVT
MRYwFAYDVQQKEw1HZW9UcnVzdCBJbmMuMRswGQYDVQQDExJHZW9UcnVzdCBHbG9i
YWwgQ0EwHhcNMTAwMjE5MjIzOTI2WhcNMjAwMjE4MjIzOTI2WjBAMQswCQYDVQQG
EwJVUzEXMBUGA1UEChMOR2VvVHJ1c3QsIEluYy4xGDAWBgNVBAMTD0dlb1RydXN0
IFNTTCBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAJCzgMHk5Uat
cGA9uuUU3Z6KXot1WubKbUGlI+g5hSZ6p1V3mkihkn46HhrxJ6ujTDnMyz1Hr4Gu
FmpcN+9FQf37mpc8oEOdxt8XIdGKolbCA0mEEoE+yQpUYGa5jFTk+eb5lPHgX3UR
8im55IaisYmtph6DKWOy8FQchQt65+EuDa+kvc3nsVrXjAVaDktzKIt1XTTYdwvh
dGLicTBi2LyKBeUxY0pUiWozeKdOVSQdl+8a5BLGDzAYtDRN4dgjOyFbLTAZJQ50
96QhS6CkIMlszZhWwPKoXz4mdaAN+DaIiixafWcwqQ/RmXAueOFRJq9VeiS+jDkN
d53eAsMMvR8CAwEAAaOB2TCB1jAOBgNVHQ8BAf8EBAMCAQYwHQYDVR0OBBYEFEJ5
VBthzVUrPmPVPEhX9Z/7Rc5KMB8GA1UdIwQYMBaAFMB6mGiNifurBWQMEX2qfWW4
ysxOMBIGA1UdEwEB/wQIMAYBAf8CAQAwOgYDVR0fBDMwMTAvoC2gK4YpaHR0cDov
L2NybC5nZW90cnVzdC5jb20vY3Jscy9ndGdsb2JhbC5jcmwwNAYIKwYBBQUHAQEE
KDAmMCQGCCsGAQUFBzABhhhodHRwOi8vb2NzcC5nZW90cnVzdC5jb20wDQYJKoZI
hvcNAQEFBQADggEBANTvU4ToGr2hiwTAqfVfoRB4RV2yV2pOJMtlTjGXkZrUJPji
J2ZwMZzBYlQG55cdOprApClICq8kx6jEmlTBfEx4TCtoLF0XplR4TEbigMMfOHES
0tdT41SFULgCy+5jOvhWiU1Vuy7AyBh3hjELC3DwfjWDpCoTZFZnNF0WX3OsewYk
2k9QbSqr0E1TQcKOu3EDSSmGGM8hQkx0YlEVxW+o78Qn5Rsz3VqI138S0adhJR/V
4NwdzxoQ2KDLX4z6DOW/cf/lXUQdpj6HR/oaToODEj+IZpWYeZqF6wJHzSXj8gYE
TpnKXKBuervdo5AaRTPvvz7SBMS24CqFZUE+ENQ=
-----END CERTIFICATE-----
Assignee: mrz → nmaul

Comment 15

7 years ago
This is online.

More specifically, the Firefox Addons Manager traffic in AR is being directed to static-ar.addons.mozilla.net, instead of static-cdn. Note that this *only* affects the addons manager... visiting addons.mozilla.org still generates links to static-cdn.addons.mozilla.net, regardless of location. However at this point all the groundwork has been laid- it would be fairly simple to set this up for the main site as well.
Status: NEW → ASSIGNED

Comment 16

7 years ago
AR, BR, and CL have been set up for this now. Overall it seems to be working well.

2 minor problems:

1) Some traffic is not being redirected as expected. In particular, Gomez testing is not being redirected, even though their nodes are detected as being in BR and AR, respectively. I believe this may be an issue with how Gomez works, rather than a problem on our end.

2) Seeing a bit higher volume of hits from non-redirected countries than would have been expected. In particular, we're getting a decent amount of traffic from Spain and Mexico. Best guess on this is that this is due to some odd client-side proxying, causing some users in these locations to be sent to the wrong place.

Investigating possible solutions for either problem.
(Assignee)

Comment 17

7 years ago
Jake - before we go into the weekend, let's tear this down.  It's not HA and we don't have any monitoring.  I think for the sake of testing, we're done right?

Comment 18

7 years ago
Aye, I think we're done. The only thing I don't have is solid evidence of a speed improvement, since Gomez isn't testing it usefully. Did you hear back from anyone in AR?

I will remove the rules this afternoon.
(Assignee)

Comment 19

7 years ago
Nothing from Gomez yet, still have a case open.

Comment 20

7 years ago
These rules have been disabled and the cache for services.addons.mozilla.org has been flushed. AR/BR/CL traffic is back to normal channels (static-cdn).

Assigning to mrz for further direction (probably close out until we get a more long-term plan).
Assignee: nmaul → mrz
(Assignee)

Comment 21

7 years ago
Nothing left to do here.
Status: ASSIGNED → RESOLVED
Last Resolved: 7 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.