804995 - put symbolpush.mozilla.org on zeus

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

Per bug 742563, this CNAME currently points to a NAT IP, rather than a Zeus VIP.  I don't recall why we did that - either to duplicate the existing config, or because phx1 Zeus was acting nasty at the time.

Steps:
 * set up new VIP
 * forward tcp/22 in zeus
 * verify
 * add flows
 * switch DNS
 * switch releng to use the alias (bug 804357)
 * remove old flows

Comment 1

[Justin Wood (:Callek)]

JUST to say it here, flows will need to be added for SeaMonkey community boxes too. And we'll also need to change to the new alias before we remove the old flows.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

You're already using the new alias, but yes, we'll update the flows before changing the underlying IP.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

Web folks, is this something you can take care of?  There's docs on this in Mana, and I can try to fill in any holes from memory.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

I didn't mention a rationale here - fixing this bug is a step toward fixing a releng SPOF.  Once it's in Zeus, adding a second backend (with the same SSH key - similar to uploads1 and uploads2) would eliminate the SPOF.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

 * set up new VIP - DONE (configured to match uploads in scl3)
   symbolpush.zlb.phx.mozilla.net / 63.245.217.193
 * forward tcp/22 in zeus - DONE
 * verify - DONE
 * add flows - bug 829039
 * verify flows
 * switch DNS
 * switch releng to use the alias (bug 804357)
 * remove old flows, NAT, DNS (clean up both 63.245.216.{248,250})
 * add a duplicate VM, with a failure pool pointing to it

Comment 6

[Michal Purzynski [:michal`] (use NEEDINFO)]

Do we really need SSH to be opened for everyone?

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

That's a valid question.  It already is open, and this will get us better logging.

Some of the client systems here have dynamic IPs, as I understand it -- builders for community projects that are on SOHO-level connections.  Alternately, we could lock down what actions are possible via SSH.  Honestly, I don't know enough to answer that question.

Can we push that to another bug?  This bug is really just about eliminating a NAT and corresponding SPOF.

Comment 8

[Michal Purzynski [:michal`] (use NEEDINFO)]

Thanks for a quick answer. You're right, will follow up in another bug.

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

Callek, can you verify

$ nc -vz 63.245.217.193 22
Connection to 63.245.217.193 22 port [tcp/ssh] succeeded!

on the hosts from which you care about symbol pushes?  I'd like to change the DNS tomorrow.  It should have no effect, but verification is cool.

Comment 10

[Justin Wood (:Callek)]

(In reply to Dustin J. Mitchell [:dustin] from comment #9)
> Callek, can you verify
> 
> $ nc -vz 63.245.217.193 22
> Connection to 63.245.217.193 22 port [tcp/ssh] succeeded!

[seabld@sea-vm-linux32-1 ~]$ nc -vz 63.245.217.193 22
Connection to 63.245.217.193 22 port [tcp/ssh] succeeded!

> on the hosts from which you care about symbol pushes?  

you told me in IRC that only one host in the scl3 VLAN should be enough.

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

DNS changed:

-; bug 742563 - community entry point for uploading build symbols
-symbolpush      IN CNAME    symbols1.pub.phx1.mozilla.com.
+; bug 742563, 804995 - community entry point for uploading build symbols
+symbolpush      IN CNAME    symbolpush.zlb.phx.mozilla.net.

Remaining:
 * switch releng to use the alias (bug 804357)
 * remove old flows, NAT, DNS (clean up both 63.245.216.{248,250})
 * add a duplicate VM, with a failure pool pointing to it

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

OK, I checked every SSH source IP on this system with >100 hits in the last week.  The vast majority were from the zlb's, and the remainder were nagios, infrasec, and ssh password-guessing scanners.  So we're good to remove the old flows, NAT, and DNS.  I'll also request a new VM.

Comment 13

[Dustin J. Mitchell [:dustin] (he/him)]

bug 863216 for deleting the NAT/flows

Comment 14

[Dustin J. Mitchell [:dustin] (he/him)]

OK, symbols2.dmz.phx1 is now set up and puppetized and has the same SSH host keys as symbols1.  It has the symbols volume mounted.  What remains is to add this to a failure pool in zeus.

Comment 15

[Dustin J. Mitchell [:dustin] (he/him)]

Ted, you should have been copied here, too, sorry.  This is just bringing up a backup host for symbols1, so there's no impact, but FYI.

I didn't mention above, I configured the hosts in puppet so that the cleanup crontask only runs on symbols1.

Comment 16

[(not currently active) Ted Mielczarek]

(In reply to Michal Purzynski [:michal`] from comment #6)
> Do we really need SSH to be opened for everyone?

Yes, we allow uploads from third parties such as Adobe and Linux distributions.

Comment 17

[Dustin J. Mitchell [:dustin] (he/him)]

OK, the backup pool is in Zeus, and Corey double-checked I set it up right.  I also configured logging of connections on Zeus.

I did forget to ask for the VMs to be put on separate ESX hosts, but I'll take care of that now.  Otherwise, this is done.