Distribute VAMO traffic better



Infrastructure & Operations
WebOps: Other
6 years ago
4 years ago


(Reporter: jakem, Assigned: jakem)





6 years ago
We want to better distribute our versions.addons.mozilla.org traffic.

For starters, let's get a vhost set up in AMS and SJC (and maybe those AR and RU nodes?), with PHX1 set as their backend pool. Then we can move DNS for this to Cedexis and using some simple geographic LB.

I don't see a Cedexis template right off the bat for complete geo-LB'ing, just simple "this country goes here, that country goes there" load balancing. That's good enough for starters. We can send a few key EU countries to AMS, and some NA/SA countries to SJC1, and let everything else fall back to PHX1. If/when we get a better Cedexis template that does this right, we can drop that in. I will fire off an email to Cedexis about this.

(FWIW I think 3crowd is the same way, but we're trying to move to a single provider in this space.)

Performance isn't really the main concern here. The main concern is offloading some of the load from PHX1 to other datacenters... if we get some performance along the way with intelligent GeoLB choices, so much the better.

Comment 1

6 years ago
Another option for this is to look into using an SSL CDN for this. There's a longer turnaround on this, and more expense, and also apparently in the past there has been some security concerns about this. I think we can get past those and this would be a worthwhile endeavor, for the future.

Also: apparently this is already migrated to 3crowd, so we're going to do it there for now and look at migrating to Cedexis in the future.

Comment 2

6 years ago
Let's just set up proxies in MPT/AR/AMS for now and if we can get approval for SSL CDN later that'd be great, but it probably won't happen any time soon.
Assignee: server-ops → oremj

Comment 3

6 years ago
I'm working on AMS now.

Comment 4

6 years ago
AMS, MPT (pm-zlb-amo*), and PHX1 are all set up for this in 3crowd. This is using a simple weighted-round-robin DNS balancing, no geolocation info used.

This is also largely set up and good-to-go in Cedexis, also with a simple WRR scheme. We can change over to this at any time. Going to ride this out on 3crowd for a while though, to avoid making too many changes at once (also waiting on Legal to approve the Cedexis agreement, and don't want to make them sweat over huge amounts of traffic unnecessarily).

A good next step while waiting would be to set up some better rules in Cedexis as far as load balancing, instead of just WRR. EMEA can go to AMS, Asia to SJC, and everything else to PHX1. Seems like a reasonable start. If we need more offloading we have those 2 RU nodes and 2 AR nodes... the AR ones should be good-to-go already, but the RU ones are not- they need Zeus or some other caching engine set up.

Comment 5

6 years ago
We are distributing this to AMS/MPT/PHX. I'll call this bug fixed, since we can easily spin up more proxies in other datacenters.

Right now DNS is in 3crowd, but we can move it to cedexis whenever.
Last Resolved: 6 years ago
Resolution: --- → FIXED

Comment 6

6 years ago
I'm going to steal this and re-open... using it as a tracking bug to get this migrated to Cedexis. Leaving as minor prio for now.
Assignee: oremj → nmaul
Resolution: FIXED → ---

Comment 7

6 years ago
Honestly I'm not sure this is worth trying to balance geographically. It will be faster, but might also cause servers to become overloaded more easily at different times of the day. Like when morning rolls around in some particular region, they'll flood a single site rather than being evenly distributed. This simple WRR scheme avoids that, and I don't think we're considering VAMO to be a performance-sensitive service. Closing this back out.
Last Resolved: 6 years ago6 years ago
Resolution: --- → FIXED
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.