938629 - apply network::aws in the network class, if the host is in AWS

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Puppet, task)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

This will save a good bit of noise in the node definitions.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

We could probably get rid of this class, actually:
 * add puppet.$domain and repos.$domain to every domain in aws
 * add $domain to 'search' in /etc/resolv.conf

However, I'll just move it for now.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug938629.patch

Comment 3

[Rail Aliiev [:rail]]

Comment on attachment 8367383 [details] [diff] [review]
bug938629.patch

Review of attachment 8367383 [details] [diff] [review]:
-----------------------------------------------------------------

::: modules/network/manifests/init.pp
@@ +58,5 @@
>          }
>      }
> +
> +    # enable network:aws on moco AWS nodes
> +    if $::config::org == 'moco' and $virtual == 'xenhvm' {

I'm not quite sure what is the best way to figure out whether we are in AWS or not... Using xen may introduce some issues in the future if we decide to use other cloud providers (or in-house!). Also, some of VMS report "xenhvm" ad some only "xen" depending on the virtualization type (HVM vs PV I think). Can you use something like

if $::config::org == 'moco' and $virtual =~ /^xen/ { ...

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Actually, I'll just block this on upgrading and getting the proper AWS facts working again.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

It occurs to me that this is probably causing uneven load among the AWS puppet masters.  Consider that when a puppet master is down or overloaded, nodes fall back to other puppetmasters.  When they do so, they point their 'puppet' alias to that new master, and will prefer it until *it* breaks.

We should probably just remove this altogether..

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug938629.patch

Once this is in place, we can remove the hosts manually from a few instances, and if nothing fails, add a temporary ensure => absent to make them disappear everywhere.

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

This seems to have prompted bug 938629, where mock configs are using an unqualified 'repos' to access the yum repositories:

config_opts['yum.conf'] = """
[main]
cachedir=/var/cache/yum
debuglevel=1
reposdir=/dev/null
logfile=/var/log/yum.log
retries=20
obsoletes=1
gpgcheck=0
assumeyes=1
syslog_ident=mock
syslog_device=

# repos

[centos6]
name=centos6
baseurl=http://repos/repos/yum/mirrors/centos/6/latest/os/i386
failovermethod=priority

[centos6-updates]
name=updates
baseurl=http://repos/repos/yum/mirrors/centos/6/latest/updates/i386
failovermethod=priority

[releng-centos6-i386]
name=releng-centos6-i386
baseurl=http://repos/repos/yum/releng/public/CentOS/6/i386

[releng-centos6-noarch]
name=releng-centos6-noarch
baseurl=http://repos/repos/yum/releng/public/CentOS/6/noarch
"""

Rather than revert this change, I think we should create the relevant DNS CNAMEs like we do for onsite VLANs, and also configure /etc/resolv.conf to contain a proper search path:

[[e-mail] ~]# cat /etc/resolv.conf 
; generated by /sbin/dhclient-script
search build.releng.scl3.mozilla.com. releng.scl3.mozilla.com. scl3.mozilla.com. mozilla.com. mozilla.org. mozilla.net.
nameserver 10.26.75.40
nameserver 10.26.75.41

but

[[e-mail] ~]# cat /etc/resolv.conf
; generated by /sbin/dhclient-script
nameserver 10.26.75.40
nameserver 10.26.75.41

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

I have added the necessary CNAMEs to all of the AWS subdomains.

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug938629-p1.patch

This will fight with dhclient a little bit, but it should work.  With this change, 'repos' and 'puppet' will continue to resolve, even when they're not in /etc/hosts.

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

Seems successful - I'll get a patch up to ensure 'puppet' and 'repos' aren't in /etc/hosts in a day or two.

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

The CNAMEs I added weren't in DNS, as I hadn't checked the "private" box.  I did that now, and it's propagated.

[[e-mail] ~]# host repos
repos.test.releng.usw2.mozilla.com is an alias for releng-puppet1.srv.releng.usw2.mozilla.com.
releng-puppet1.srv.releng.usw2.mozilla.com has address 10.132.48.212
[[e-mail] ~]# host puppet
puppet.test.releng.usw2.mozilla.com is an alias for releng-puppet1.srv.releng.usw2.mozilla.com.
releng-puppet1.srv.releng.usw2.mozilla.com has address 10.132.48.212

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: bug938629-p2.patch

Comment 13

[Rail Aliiev [:rail]]

Comment on attachment 8410504 [details] [diff] [review]
bug938629-p2.patch

Review of attachment 8410504 [details] [diff] [review]:
-----------------------------------------------------------------

woot!

Comment 14

[Rail Aliiev [:rail]]

Comment on attachment 8410504 [details] [diff] [review]
bug938629-p2.patch

I mean r+!

Comment 15

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8410504 [details] [diff] [review]
bug938629-p2.patch

merged to prod