757656 - Add IPv6 VIPs to the PHX Zeus cluster

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Arzhel Younsi [:XioNoX]]

Now that the Zeus nodes have v6 connectivity, could you add the VIPs for at least those websites:

www.mozilla.org 2620:101:8008:5::2:1
addons.mozilla.org 2620:101:8008:5::2:2
support.mozilla.org 2620:101:8008:5::2:3
wiki.mozilla.org 2620:101:8008:5::2:4

And pick any IP in the 2620:101:8008:5::2:X range for more sites.

Make sure to add that IP to the reverse DNS (external/db.2620-0101-8008.ip6.arpa) and forward (AAAA), you can also test it with your "hosts" file before adding the AAAA.

Monitoring will only be with gomez and watchmouse until it's ready on Nagios (probably 680828).

Comment 1

[Jake Maul [:jakem]]

Mine!

I don't know if Catchpoint does IPv6 monitoring... I'll find out.

Comment 3

[Jake Maul [:jakem]]

2 of these hosts should be relatively easy, and 2 will be... less so.

wiki.mozilla.org and addons.mozilla.org are (currently) simple A records on our end. Those should be the easy ones. I'm starting with wiki.

support.mozilla.org goes through Cedexis. They work *only* with CNAMEs. This *should* be fine... we just need to add AAAA records beside the existing A records. The downside is that we'll have IPv6 support in North America (PHX1), but not in the EU and some parts of Asia (hosted by AMS1).

www.mozilla.org goes through 3crowd. Currently they return an A record directly. We can change this to return a CNAME instead, like Cedexis does. This is another layer of DNS queries users have to wade through, and slightly degrades performance. Also, this is served in PHX1 *and* SCL3, and cached in AMS1. We will have only partial IPv6 support in North America, as people who are directed to the SCL3 CNAME will not be able to resolve a AAAA record.

Comment 4

[Jake Maul [:jakem]]

I've set up the forward records for wiki.mozilla.org (and wikimo.zlb.phx.mozilla.net, because that made sense to me) and added this IP to the Zeus traffic group- it works.

However, the rDNS record does not resolve. I'm not sure what's wrong, I've bumped the SOA twice.

; 2620:0101:8008:0005::/64
; VLAN 5 - VIPs go here!
$ORIGIN 5.0.0.0.8.0.0.8.1.0.1.0.0.2.6.2.ip6.arpa.
4.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0         IN  PTR wikimo.zlb.phx.mozilla.net.

The ones right above it (pp-zlb08 through 12) resolve fine, but this one doesn't. Can you take a look and see if I've done something wrong, or... ?

Comment 5

[Jake Maul [:jakem]]

support.mozilla.org is done now as well, with the caveat from comment 3 (it only works if you're directed to PHX1... so it doesn't work in the EU and parts of Asia).

I've also set up blog.mozilla.org... tested with curl and working.

I figured out the rDNS problem. It was... rather obvious with fresh eyes. It helps a lot to put in the *right* records:

$ORIGIN 5.0.0.0.8.0.0.8.1.0.1.0.0.2.6.2.ip6.arpa.
3.0.0.0.2.0.0.0.0.0.0.0.0.0.0.0         IN  PTR sumo.zlb.phx.mozilla.net.
4.0.0.0.2.0.0.0.0.0.0.0.0.0.0.0         IN  PTR wikimo.zlb.phx.mozilla.net.
5.0.0.0.2.0.0.0.0.0.0.0.0.0.0.0         IN  PTR blog.zlb.phx.mozilla.net.

Comment 6

[Jake Maul [:jakem]]

www.mozilla.org is done also. Again only in PHX1, so the majority of traffic isn't eligible for it. SCL3 and PHX1 are split 50/50 on NA/SA traffic, and AMS1 gets all EU traffic.

I've also added a v6 VIP for the "generic" VIP in PHX1. However, virtually everything hitting that is an A record, not a CNAME to the VIP name, so this won't have much effect until/unless all those individual records get updated one way or the other. To avoid "doubling up" on all those records, it might be cleaner to make them CNAMEs to generic.zlb.phx.mozilla.net instead.


I will wait until tomorrow to do addons.mozilla.org, so oremj, jthomas, and the developers will be around "just in case". So far though, the 4 I've done have all gone very smoothly. AMO and MDN are the last "big" sites, and MDN lives in SCL3... hopefully we can have that up and running by World IPv6 Day on June 6.


The process is simple:

1) add the v6 IP to the Traffic Group that already contains the matching v4 IP
2) test w/ curl from natasha: curl -v -6 -g -H 'Host: <site>' 'http://[v6 IP]/'
3) add the rDNS record in external/db.2620-0101-8008.ip6.arpa
4) add the forward DNS record in the appropriate place
5) If necessary, change 3crowd to point to the *name* rather than the v4 IP
6) Watch the Zeus real-time activity graph to see the hits!

Comment 7

[Jake Maul [:jakem]]

AMO is done.

We moved most of the generic cluster domains over to use the generic VIP name (A->CNAME), so they're all IPv6 now too. This includes quality.mozilla.org and hacks.mozilla.org, for example.

I've also added v6 IP's to the static cluster. This includes nightly.mozilla.org and planet.mozilla.org, among others.

Comment 8

[Jake Maul [:jakem]]

Added to engagement cluster VIP (ex: affiliates.mozilla.org et al).

Added to input cluster VIP (input.mozilla.org).

Comment 9

[Jake Maul [:jakem]]

Nothing big left to do here, so I'm closing this out. If there's anything big that I missed (that doesn't already have another bug, like ns2 IIRC), please re-open or make a new bug.

Thanks! And let me know when we can do the same in SCL3. :) :) :)