522923 - sumo / GLB deployment in Amsterdam/Singapore

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[matthew zeier [:mrz]]

About ready to add sumo into GLB and serve from Amsterdam.

It's staged @ 63.245.213.88 & 63.245.213.89.  Would like some QA eyes on this.

Possible to target Tuesday (10/20) or Thursday (10/21)?

Comment 1

[matthew zeier [:mrz]]

And staged @ 180.92.184.28 in Singapore for those in that region.

Comment 2

[Shyam Mani [:fox2mike]]

(In reply to comment #1)
> And staged @ 180.92.184.28 in Singapore for those in that region.

Works fine. I clicked around a bit, searches etc seem to fine :)

Comment 3

[Stephen Donner [:stephend] Not actively reading bugmail]

mrz, Thursday would be better for me, if that's still doable.

Comment 4

[Arzhel Younsi [:XioNoX]]

mrz, the GLB is ready for sumo, with HTTP and HTTPS checks on the 3 datacenters.

Comment 5

[Stephen Donner [:stephend] Not actively reading bugmail]

I've been getting somewhat frequent "Service Unavailable" when on 63.245.213.89; can you guys check that host?  It could also be that it's with the load balancer, and I just gut unlucky -- I'll keep looking off and on tonight.

Comment 6

[Pedro Alves]

As a side note, this won't impact metrics, as we collect data from db and not log parsing

Comment 7

[matthew zeier [:mrz]]

daniel, does this mean we're okay to do this tonight from your end?

Comment 8

[matthew zeier [:mrz]]

Pushing to Thursday.  Was unaware of support.mozilla.com's MX RRs.

Over to justdave for postfix massaging.

Comment 9

[Dave Miller [:justdave]]

CNAME's target domain added to postfix.

Comment 10

[Dave Miller [:justdave]]

I reverted the DNS changes, looks like we screwed outgoing mail, too (see bug 523533)

Comment 11

[Dave Miller [:justdave]]

The correct fix:

1) remove all records for support.mozilla.com from the mozilla.com zone.
2) add NS records for support.mozilla.com in the mozilla.com zone pointing at the glb.mozilla.net NS servers
3) tell the GLB servers to handle the entire support.mozilla.com zone
4) set up SOA and MX records in the support.mozilla.com zone on the GLB servers

Comment 12

[matthew zeier [:mrz]]

(In reply to comment #11)
> The correct fix:
> 
> 1) remove all records for support.mozilla.com from the mozilla.com zone.
> 2) add NS records for support.mozilla.com in the mozilla.com zone pointing at
> the glb.mozilla.net NS servers
> 3) tell the GLB servers to handle the entire support.mozilla.com zone
> 4) set up SOA and MX records in the support.mozilla.com zone on the GLB servers

This got more complicated.  

Over to Arzhel to stage.

Comment 13

[Dave Miller [:justdave]]

This got pushed about 40 minutes ago, DNS Trace looks good.
http://pastebin.mozilla.org/678392