Given a destination DNS entry, setup healthcheck based sendto.mozilla.org DNS

RESOLVED FIXED

Status

Infrastructure & Operations
WebOps: Other
RESOLVED FIXED
4 years ago
4 years ago

People

(Reporter: jp, Assigned: gozer)

Tracking

Details

(Whiteboard: [kanban:webops:https://kanbanize.com/ctrl_board/4/1966] )

(Reporter)

Description

4 years ago
For End-of-year fundraising, we're building a failover system of if/when BSD goes down.  

OUr plan is to use healthcheck based DNS, discussed with Gozer, to hand out either the BSD cname / address while they are up, or a Cloudfronted distribution hosted on mofosecure if BSD is down.

https://dl.dropboxusercontent.com/u/2273146/BSD%20Fundraising%20Failover.jpg

Actions yet to take:  
1) JP to setup the infra on mofosecure
2) JP to apply a resource CNAME to the cloudfront distro
3) JP to add sendto.mozilla.org to the accepted hosts on distro
4) JP to add the SSL cert, for all clients (not just SNI), to the distro endpoints
5) JP to update this bug with the desired destination CNAME for downtime situations
6) JP to specify endpoint url to test for healthcheck
7) Gozer to setup healthcheck and associated DNS records

Updated

4 years ago
Whiteboard: [kanban:webops:https://kanbanize.com/ctrl_board/4/1966]
(Reporter)

Comment 1

4 years ago
1) JP to setup the infra on mofosecure (COMPLETE)
2) JP to apply a resource CNAME to the cloudfront distro (COMPLETE) (d94ya7facqszn.cloudfront.net)
3) JP to add sendto.mozilla.org to the accepted hosts on distro (COMPLETE)
4) JP to add the SSL cert, for all clients (not just SNI), to the distro endpoints (PENDING)
5) JP to update this bug with the desired destination CNAME for downtime situations (COMPLETE) (d94ya7facqszn.cloudfront.net)
6) JP to specify endpoint url to test for healthcheck (COMPLETED, see below)
7) Gozer to setup healthcheck and associated DNS records (READY)

So, this curl does the healthcheck.  Can you pass host headers along with it?

curl -H "HOST: sendto.mozilla.org" https://mozilla.sanssl-010.bsdtools.com/page/contribute/givenow-seq

Otherwise, I'm wondering how we know to fail back.
Flags: needinfo?(gozer)
(Assignee)

Updated

4 years ago
Assignee: server-ops-webops → gozer
Flags: needinfo?(gozer)
(Assignee)

Comment 2

4 years ago
The CNAME for this failover service is :

sendto.dynect.mozilla.net

And it's currently up and monitoring:

https://mozilla.sanssl-010.bsdtools.com/page/contribute/givenow-seq
Host: sendto.mozilla.org

for a 200 OK response.

On failure, it will return:

d94ya7facqszn.cloudfront.net

Checks are done every 60 seconds, TTL is 30 secs ( the minimums )
(Assignee)

Comment 3

4 years ago
7) Gozer to setup healthcheck and associated DNS records (COMPLETED)

And to go live, we are just missing:

8) update sendto.mozilla.org to be a CNAME for sendto.dynect.mozilla.net
(Reporter)

Comment 4

4 years ago
Gozer:
Can we please do a test of this by intentionally borking the healthcheck url to say.....bendto.mozilla.org instead of sendto.mozilla.org ?
It'd be cool to coordinate when, so we could time dns prop and be super confident in our timings for a real failover.
Flags: needinfo?(gozer)
(Assignee)

Comment 5

4 years ago
(In reply to JP Schneider :jp from comment #4)
> Gozer:
> Can we please do a test of this by intentionally borking the healthcheck url
> to say.....bendto.mozilla.org instead of sendto.mozilla.org ?

Since we are not using sendto.dynect.mozilla.net, it should be perfectly fine to break the health-check on purpose anytime you want.

> It'd be cool to coordinate when, so we could time dns prop and be super
> confident in our timings for a real failover.
Flags: needinfo?(gozer)
(Reporter)

Comment 6

4 years ago
Huzzah, testing worked!

Thu Dec 18 11:26:29 CST 2014
Server:		75.75.75.75
Address:	75.75.75.75#53

Non-authoritative answer:
sendto.dynect.mozilla.net	canonical name = mozilla.sanssl-010.bsdtools.com.
mozilla.sanssl-010.bsdtools.com	canonical name = sendto.mozilla.org.cdn.bsd.net.
sendto.mozilla.org.cdn.bsd.net	canonical name = sanssl-010.bsdtools.com.edgekey.net.
sanssl-010.bsdtools.com.edgekey.net	canonical name = e10427.g.akamaiedge.net.
Name:	e10427.g.akamaiedge.net
Address: 23.194.137.234
Thu Dec 18 11:26:30 CST 2014
Server:		75.75.75.75
Address:	75.75.75.75#53

Non-authoritative answer:
sendto.dynect.mozilla.net	canonical name = mozilla.sanssl-010.bsdtools.com.
mozilla.sanssl-010.bsdtools.com	canonical name = sendto.mozilla.org.cdn.bsd.net.
sendto.mozilla.org.cdn.bsd.net	canonical name = sanssl-010.bsdtools.com.edgekey.net.
sanssl-010.bsdtools.com.edgekey.net	canonical name = e10427.g.akamaiedge.net.
Name:	e10427.g.akamaiedge.net
Address: 23.193.164.101
Thu Dec 18 11:26:31 CST 2014
Server:		75.75.75.75
Address:	75.75.75.75#53

Non-authoritative answer:
sendto.dynect.mozilla.net	canonical name = mozilla.sanssl-010.bsdtools.com.
mozilla.sanssl-010.bsdtools.com	canonical name = sendto.mozilla.org.cdn.bsd.net.
sendto.mozilla.org.cdn.bsd.net	canonical name = sanssl-010.bsdtools.com.edgekey.net.
sanssl-010.bsdtools.com.edgekey.net	canonical name = e10427.g.akamaiedge.net.
Name:	e10427.g.akamaiedge.net
Address: 23.193.164.101
Thu Dec 18 11:26:32 CST 2014
Server:		75.75.75.75
Address:	75.75.75.75#53

Non-authoritative answer:
sendto.dynect.mozilla.net	canonical name = mozilla.sanssl-010.bsdtools.com.
mozilla.sanssl-010.bsdtools.com	canonical name = sendto.mozilla.org.cdn.bsd.net.
sendto.mozilla.org.cdn.bsd.net	canonical name = sanssl-010.bsdtools.com.edgekey.net.
sanssl-010.bsdtools.com.edgekey.net	canonical name = e10427.g.akamaiedge.net.
Name:	e10427.g.akamaiedge.net
Address: 23.197.26.220
Thu Dec 18 11:26:33 CST 2014
Server:		75.75.75.75
Address:	75.75.75.75#53

Non-authoritative answer:
sendto.dynect.mozilla.net	canonical name = mozilla.sanssl-010.bsdtools.com.
mozilla.sanssl-010.bsdtools.com	canonical name = sendto.mozilla.org.cdn.bsd.net.
sendto.mozilla.org.cdn.bsd.net	canonical name = sanssl-010.bsdtools.com.edgekey.net.
sanssl-010.bsdtools.com.edgekey.net	canonical name = e10427.g.akamaiedge.net.
Name:	e10427.g.akamaiedge.net
Address: 23.197.26.220
Thu Dec 18 11:26:34 CST 2014
Server:		75.75.76.76
Address:	75.75.76.76#53

Non-authoritative answer:
sendto.dynect.mozilla.net	canonical name = mozilla.sanssl-010.bsdtools.com.
mozilla.sanssl-010.bsdtools.com	canonical name = sendto.mozilla.org.cdn.bsd.net.
sendto.mozilla.org.cdn.bsd.net	canonical name = sanssl-010.bsdtools.com.edgekey.net.
sanssl-010.bsdtools.com.edgekey.net	canonical name = e10427.g.akamaiedge.net.
Name:	e10427.g.akamaiedge.net
Address: 23.193.164.101
Thu Dec 18 11:26:36 CST 2014
Server:		75.75.75.75
Address:	75.75.75.75#53

Non-authoritative answer:
sendto.dynect.mozilla.net	canonical name = mozilla.sanssl-010.bsdtools.com.
mozilla.sanssl-010.bsdtools.com	canonical name = sendto.mozilla.org.cdn.bsd.net.
sendto.mozilla.org.cdn.bsd.net	canonical name = sanssl-010.bsdtools.com.edgekey.net.
sanssl-010.bsdtools.com.edgekey.net	canonical name = e10427.g.akamaiedge.net.
Name:	e10427.g.akamaiedge.net
Address: 23.197.26.220
Thu Dec 18 11:26:37 CST 2014
Server:		75.75.75.75
Address:	75.75.75.75#53

Non-authoritative answer:
sendto.dynect.mozilla.net	canonical name = mozilla.sanssl-010.bsdtools.com.
mozilla.sanssl-010.bsdtools.com	canonical name = sendto.mozilla.org.cdn.bsd.net.
sendto.mozilla.org.cdn.bsd.net	canonical name = sanssl-010.bsdtools.com.edgekey.net.
sanssl-010.bsdtools.com.edgekey.net	canonical name = e10427.g.akamaiedge.net.
Name:	e10427.g.akamaiedge.net
Address: 23.197.26.220
Thu Dec 18 11:26:38 CST 2014
Server:		75.75.75.75
Address:	75.75.75.75#53

Non-authoritative answer:
sendto.dynect.mozilla.net	canonical name = mozilla.sanssl-010.bsdtools.com.
mozilla.sanssl-010.bsdtools.com	canonical name = sendto.mozilla.org.cdn.bsd.net.
sendto.mozilla.org.cdn.bsd.net	canonical name = sanssl-010.bsdtools.com.edgekey.net.
sanssl-010.bsdtools.com.edgekey.net	canonical name = e10427.g.akamaiedge.net.
Name:	e10427.g.akamaiedge.net
Address: 23.193.164.101
Thu Dec 18 11:26:39 CST 2014
Server:		75.75.75.75
Address:	75.75.75.75#53

Non-authoritative answer:
sendto.dynect.mozilla.net	canonical name = d94ya7facqszn.cloudfront.net.
Name:	d94ya7facqszn.cloudfront.net
Address: 54.230.90.93
Name:	d94ya7facqszn.cloudfront.net
Address: 54.230.91.70
Name:	d94ya7facqszn.cloudfront.net
Address: 54.192.91.122
Name:	d94ya7facqszn.cloudfront.net
Address: 54.192.91.149
Name:	d94ya7facqszn.cloudfront.net
Address: 54.230.90.22
Name:	d94ya7facqszn.cloudfront.net
Address: 54.192.91.92
Name:	d94ya7facqszn.cloudfront.net
Address: 54.230.91.49
Name:	d94ya7facqszn.cloudfront.net
Address: 54.230.90.109
Thu Dec 18 11:26:41 CST 2014
Server:		75.75.75.75
Address:	75.75.75.75#53

Many thanks for the help Gozer!
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
(Reporter)

Comment 7

4 years ago
We can go ahead and flip main DNS good sir Gozer!
Thanks!
Status: RESOLVED → REOPENED
Flags: needinfo?(gozer)
Resolution: FIXED → ---
(Assignee)

Comment 8

4 years ago
DNS Updated
Flags: needinfo?(gozer)
(Assignee)

Updated

4 years ago
Status: REOPENED → RESOLVED
Last Resolved: 4 years ago4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.