1141812 - Check number of ZLBs assigned to ftp/ftp-ssl/cdn origin

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Product Delivery, task)

Description

[Nick Thomas [:nthomas] (UTC+12)]

Bug 1140070 was one of a few recent examples of CDN traffic affecting the development trees by slowing down the download of tests. I would like to revisit the number of ZLBs allocated to several domains, since they may be optimized for older traffic levels. Based on DNS we have

* 1 for download-origin.cdn.mozilla.net - CDN origin, I think
* 3 for ftp-ssl.mozilla.org - test traffic
* 2 for ftp.mozilla.org - previously for test traffic

So please check if traffic justifies 2 zlb's on ftp.m.o, if we can add another zlb to download-origin, and reduce overlap between ftp-ssl and download-origin. I realize these there are a lot of other properties which also use these load balancers. FWIW there should be very little overlap between download-origin and ftp-ssl in terms of caching, but they probably use a lot of cache.

Could we do this before 37.0 ships on March 31, if not before ?

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Also download-installer-origin.cdn.mozilla.net

Comment 2

[Shyam Mani [:fox2mike]]

Going to grab this for now. Because of recent outages, we will be trying out some stuff over the next couple of days that might help solve this.

Comment 3

[Shyam Mani [:fox2mike]]

Nick,

Because of our new temporary licenses, I believe we'll be ok for March 31st. I'm going to give this back to Eric and we'll work on this post March 31st.

Comment 4

[Eric Ziegenhorn :ericz]

We're making some changes to Zeus licensing in bug 1133912 which should alleviate these concerns.  We'll make sure everything is reasonably balanced after that process finishes.

Comment 5

[Eric Ziegenhorn :ericz]

download-installer-origin.cdn.mozilla.net points through dynect to ftp1-zlb.vips.scl3.mozilla.com so it's the same as ftp.mozilla.org

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

FWIW, I'm not as worried about this any more. A lot of the traffic on ftp-ssl has already moved away, and ftp.m.o will go away with the move to S3.

Comment 7

[Eric Ziegenhorn :ericz]

Ah, well I'll leave the answers I've already dug up and as nothing looks to be big problem area, we'll leave it as is for now.

Addressing the questions in comment 1:

1) Does traffic justify 2 zlb's on ftp.m.o?
    ftp.mozilla.org has two ZLBs now, zlb5 and zlb6, which have a 10Gbps and 2Gbps license respectively.  Traffic for ftp.mozilla.org is 50-200Mbps typically with spikes around 300Mbps, so we appear to have plenty of headroom.

2) Can we add another zlb to download-origin?
    download-origin.cdn.mozilla.net points to download-origin-zlb.vips.scl3.mozilla.com which is hosted on zlb1.  Traffic is usually 75-250Mbps, with occasional spikes around 1500Mbps.  zlb1's license is 2000Mbps.  Another zlb does not appear to be necessary though it could have more headroom.

3) Can we reduce overlap between ftp-ssl and download-origin?
    ftp-ssl uses zlb1, zlb3 and zlb5.  That only overlaps with download-origin on zlb1.  Given ftp-ssl's traffic is typically 0-200Mbps with spikes to around 400Mbps, it looks low enough to just take it off of zlb1 if desired.