1446468 - [MDC1] & [MDC2] New VM: observatory-scanner3.dmz.mdc1, observatory-scanner4.dmz.mdc2

Status: RESOLVED FIXED

(Infrastructure & Operations :: Virtualization, task)

Description

[April King [:April]]

Can I get virtual machines for the Observatory scanners, one in MDC1 and one in MDC2?

The configurations currently in use for "observatory-scanner1.dmz.scl3.mozilla.com" and "observatory-scanner2.dmz.scl3.mozilla.com" (CPU and memory) should be fine, except instead of Ubuntu 14 it would be great if they could be Ubuntu 16. This is for the SCL3 decommission.

Notably weird things with the Observatory scanners:
* They need outbound HTTP and HTTPS without the use of the proxies, due to their function
* They need to have IPv6 functionality and the ability to access IPv6-only hosts

It probably makes sense to simply call them "observatory-scanner3" and "observatory-scanner4", unless Ops has any objections. Also, could I get the exit point external IP address, so I can whitelist them in Amazon RDS?

Thanks so much!

Comment 1

[Chris Knowles [:cknowles]]

I show the existing ones at 4 core, 4GB RAM, default storage.  No problems there.

The OS is where things go a little off the rails.

Due to our puppet environment, we can't support 16.04 - we're stuck back on 14.04.  Also, due to the near complete lack of requests for Ubuntu, we've not put in templates in MDC1/2 for Ubuntu.  So.  I can't get you 16.04, and I can get you 14.04, but it'll take a couple weeks at minimum to spin that up.  

If you need something faster, I can get you CentOS 7 in short order.

Let me know and I can start moving.  

As for the flows and other network issues, that'll have to be a bug for netops.  I have no say in setting those things up.

Comment 2

[April King [:April]]

I'm not wedded to Ubuntu, although that's what I was using in SCL3 and in AWS. Given that I have time, I can probably figure it out. So go ahead with CentOS 7, I suppose.  :)

Thanks!

Comment 3

[Chris Knowles [:cknowles]]

Alright, I'll get started on that shortly.  And I'll put "get ubuntu available" into the pile of things for near future (but not crash priority).  Just in case.

Comment 4

[April King [:April]]

I'm stuck having to deal with systemctl regardless, so it's gonna be miserable either way.  :P

Comment 5

[Chris Knowles [:cknowles]]

OK, created, inventoried, puppeted, tracked.

The SCL3 versions were initial placed in nagios - but aren't there now.  I'm assuming that's intentional, and didn't add these to their respective nagios hosts - if that's in error, let me know, and I'm happy to put them in.

To be friendly to our netops brethren when they create rules:
observatory-scanner3.dmz.mdc1.mozilla.com - 10.48.74.20
observatory-scanner4.dmz.mdc2.mozilla.com - 10.50.74.20

Let me know if you need anything else.

Comment 6

[April King [:April]]

Pretty sure they were in nagios? I used to get bug reports when they went critical, but maybe they were removed because it was kind of difficult to track actual outages?

Comment 7

[Chris Knowles [:cknowles]]

Ah, yes, looks like they were removed from SCL3's nagios and into a specific EIS instance of nagios in SCL3 - those haven't been spun up in the other datacenters (bug 1445371 - they have their VMs, but it's not deployed yet)

So, I could stick then in gen-pop nagios for now, or we can wait for that bug to be resolved.  Up to you.  (Of course, right now, I'd be sticking them into preprod so it won't alert before you're done setting up.)

Let me know.

Comment 8

[April King [:April]]

Let's just wait for now, I'd say. Thanks, :cknowles. Have a great weekend!

Comment 9

[Chris Knowles [:cknowles]]

Per your comment on IRC about yum-cron - Not sure what happened to get yum-cron installed on those VMs - it's not part of our templates, as automatic updates could easily do (and has done) damage to the puppet system on there.  To prevent that. I've removed yum-cron from the systems, and verified that puppet is working.  

If you need it on there, let me know, and we can work out options.