1387390 - Create ansible playbook to run puppet on jumphosts

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Puppet, task)

Comment 1

[Jake Watkins [:dividehex]]

So the big problem I ran into while trying to set this up was ansible doesn't play nice with dou auth.  In order to exec the playbook, you need to setup a tunnel and duo auth to each jumphost before running ansible.  Then you pass that tunnel info to ansible.  This can probably be scripted but at that point, you might as well just skip using ansible altogether and just bash script it.

Is there a bug for the cron diff checks? That might be a bigger priority than scripting puppet runs remotely.

Comment 2

[Dragos Crisan [:dragrom]]

So, I created a short ansible play-book, to return the output for puppet agent -t --noop command. I tested the playbook on my moonshot-test machine.This is the output:

ok: [moonshot-test3.test.releng.scl3.mozilla.com] => {
    "puppet_agent.stdout_lines": [
        "\u001b[0;32mInfo: Retrieving pluginfacts\u001b[0m", 
        "\u001b[0;32mInfo: Retrieving plugin\u001b[0m", 
        "\u001b[0;32mInfo: Loading facts\u001b[0m", 
        "\u001b[0;32mInfo: Caching catalog for moonshot-test3.test.releng.scl3.mozilla.com\u001b[0m", 
        "\u001b[0;32mInfo: Applying configuration version '15bac7fca723'\u001b[0m", 
        "\u001b[mNotice: /Stage[main]/Disableservices::Release_upgrader/Package[ubuntu-release-upgrader-core]/ensure: current_value 1:16.04.21, should be absent (noop)\u001b[0m", 
        "\u001b[mNotice: Augeas[resolvconf](provider=augeas): ", 
        "--- /etc/resolv.conf\t2017-08-06 09:05:26.004000000 -0700", 
        "+++ /etc/resolv.conf.augnew\t2017-08-07 08:00:06.057322000 -0700", 
        "@@ -2,4 +2,4 @@", 
        " #     DO NOT EDIT THIS FILE BY HAND -- YOUR CHANGES WILL BE OVERWRITTEN", 
        " nameserver 10.26.75.40", 
        " nameserver 10.26.75.41", 
        "-search test.releng.scl3.mozilla.com releng.scl3.mozilla.com scl3.mozilla.com mozilla.com mozilla.org mozilla.net", 
        "+domain test.releng.scl3.mozilla.com", 
        "\u001b[0m", 
        "\u001b[mNotice: /Stage[main]/Network::Resolv/Augeas[resolvconf]/returns: current_value need_to_run, should be 0 (noop)\u001b[0m", 
        "\u001b[mNotice: Class[Network::Resolv]: Would have triggered 'refresh' from 1 events\u001b[0m", 
        "\u001b[mNotice: /Stage[main]/Disableservices::Common/Service[lightdm]/enable: current_value true, should be false (noop)\u001b[0m", 
        "\u001b[mNotice: /Stage[main]/Disableservices::Common/Service[avahi-daemon]/ensure: current_value running, should be stopped (noop)\u001b[0m", 
        "\u001b[mNotice: /Stage[main]/Taskcluster_worker/File[/etc/taskcluster-worker.yml]/content: ", 
        "--- /etc/taskcluster-worker.yml\t2017-07-13 01:48:32.404670390 -0700", 
        "+++ /tmp/puppet-file20170807-11606-sun12b\t2017-08-07 08:00:08.009322000 -0700", 
        "@@ -10,10 +10,10 @@", 
        "     $hostcredentials:", 
        "     - http://releng-puppet1.srv.releng.scl3.mozilla.com:8020/v1/credentials", 
        "     - http://releng-puppet2.srv.releng.scl3.mozilla.com:8020/v1/credentials", 
        "-    - http://releng-puppet2.srv.releng.mdc1.mozilla.com:8020/v1/credentials", 
        "-    - http://releng-puppet1.srv.releng.usw2.mozilla.com:8020/v1/credentials", 
        "-    - http://releng-puppet1.srv.releng.use1.mozilla.com:8020/v1/credentials", 
        "     - http://releng-puppet1.srv.releng.mdc1.mozilla.com:8020/v1/credentials", 
        "+    - http://releng-puppet1.srv.releng.use1.mozilla.com:8020/v1/credentials", 
        "+    - http://releng-puppet1.srv.releng.usw2.mozilla.com:8020/v1/credentials", 
        "+    - http://releng-puppet2.srv.releng.mdc1.mozilla.com:8020/v1/credentials", 
        "   engine:           native", 
        "   engines:", 
        "     native:", 
        "@@ -34,7 +34,7 @@", 
        "     logprefix:", 
        "       hostname: moonshot-test3.test.releng.scl3.mozilla.com", 
        "       workerType: gecko-t-linux-talos", 
        "-      workerGroup:  moonshot-scl3", 
        "+      workerGroup:  scl3", 
        "     env:", 
        "       extra:", 
        "         PATH: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/bin/X11", 
        "@@ -45,7 +45,7 @@", 
        "   worker:", 
        "     provisionerId:    releng-hardware", 
        "     workerType :      gecko-t-linux-talos", 
        "-    workerGroup:      moonshot-scl3", 
        "+    workerGroup:      scl3", 
        "     workerId:         moonshot-test3", 
        "     pollingInterval:  10", 
        "     reclaimOffset:    120", 
        "\u001b[0m", 
        "\u001b[mNotice: /Stage[main]/Taskcluster_worker/File[/etc/taskcluster-worker.yml]/content: current_value {md5}21fb9fcfbd68a5e74eae4c19c0aa9ae6, should be {md5}c2c41b12fdc211031223e6530b269883 (noop)\u001b[0m", 
        "\u001b[mNotice: Class[Disableservices::Release_upgrader]: Would have triggered 'refresh' from 1 events\u001b[0m", 
        "\u001b[mNotice: /Stage[main]/Puppet::Atboot/File[/etc/puppet/puppetmasters.txt]/content: ", 
        "--- /etc/puppet/puppetmasters.txt\t2017-07-13 01:48:32.572754384 -0700", 
        "+++ /tmp/puppet-file20170807-11606-pmbsaj\t2017-08-07 08:00:08.165322000 -0700", 
        "@@ -1,6 +1,6 @@", 
        " releng-puppet1.srv.releng.scl3.mozilla.com", 
        " releng-puppet2.srv.releng.scl3.mozilla.com", 
        "-releng-puppet2.srv.releng.mdc1.mozilla.com", 
        "-releng-puppet1.srv.releng.usw2.mozilla.com", 
        "-releng-puppet1.srv.releng.use1.mozilla.com", 
        " releng-puppet1.srv.releng.mdc1.mozilla.com", 
        "+releng-puppet1.srv.releng.use1.mozilla.com", 
        "+releng-puppet1.srv.releng.usw2.mozilla.com", 
        "+releng-puppet2.srv.releng.mdc1.mozilla.com", 
        "\u001b[0m", 
        "\u001b[mNotice: /Stage[main]/Puppet::Atboot/File[/etc/puppet/puppetmasters.txt]/content: current_value {md5}c067c4f962a201dbc8a21b6188f38f0b, should be {md5}3e94564f0377f4e1f92db9eb0b03c298 (noop)\u001b[0m", 
        "\u001b[mNotice: Class[Puppet::Atboot]: Would have triggered 'refresh' from 1 events\u001b[0m", 
        "\u001b[mNotice: /Stage[main]/Disableservices::Common/Service[acpid]/ensure: current_value running, should be stopped (noop)\u001b[0m", 
        "\u001b[mNotice: Class[Taskcluster_worker]: Would have triggered 'refresh' from 1 events\u001b[0m", 
        "\u001b[mNotice: Class[Disableservices::Common]: Would have triggered 'refresh' from 3 events\u001b[0m", 
        "\u001b[mNotice: Stage[main]: Would have triggered 'refresh' from 5 events\u001b[0m", 
        "\u001b[mNotice: Finished catalog run in 6.42 seconds\u001b[0m"
    ]
}

Is this output what you expect? Or want to try to refine it, and show only the nottice about new users?
If it is ok, I'll test it on mdc1 jumphosts

Comment 3

[Jake Watkins [:dividehex]]

The output looks fine but we want to exec this in a shell wrapper script from cron, not from ansible.  The wrapper should then run maybe once a day and email a notice stating there is puppet changes ready to be applied to the host.

Comment 4

[Kendall Libby [:fubar] (he/him)]

Yeah, I think this became two separate, but linked, issues: a cron job to output puppet changes, and an ansible script to force a puppet run on the jumphosts. As Jake says, a shell script is sufficient (and the output will be a million times easier for humans to read!).

Comment 5

[Dragos Crisan [:dragrom]]

Opened Bug 1388282 for shell wrapper script

Comment 6

[Dragos Crisan [:dragrom]]

To access the rejh hosts with ansible we have 2 options:
1 - create a tunnel with the rej hosts
2 - hack the ansible code - edit /usr/lib/python2.7/dist-packages/ansible/plugins/connection/ssh.py and comment the following lines:
#                   "-o", "KbdInteractiveAuthentication=no",
#                   "-o", "PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey",

Then, ansible can be used to connect to rejh hosts without create a ssh tunnel

But this is a short term solution. I looking to use fabric and create a python script instead of using ansible

Comment 7

[Dragos Crisan [:dragrom]]

I tested Fabric (http://www.fabfile.org/) to manage puppet on jumphosts. Fabric is a python2.7 library and command-line tool for streamlining the use of SSH for application deployment or systems administration tasks.

There are some consideration to use it instead of ansible:

1. easy to install: pip install fabric
2. easy to write the management script:
vim fabfile.py

from fabric.api import run, env, sudo

env.hosts = ['rejh1.srv.releng.mdc1.mozilla.com','rejh2.srv.releng.mdc1.mozilla.com','rejh1.srv.releng.scl3.mozilla.com','rejh2.srv.releng.scl3.mozilla.com']

def run_puppet():
    sudo('puppet agent -t')

3. easy to use, just run fab -u dcrisan run_puppet -t 120

Comment 8

[Dragos Crisan [:dragrom]]

Attachment: fabfile.py

- the fabric script to run puppet on jumphosts
- to install fabric: run pip install fabric
- to run the script: from the console run fab -u <rjh_username> run_puppet -t 120

Comment 9

[Dragos Crisan [:dragrom]]

On what repo can i land this script?

Comment 10

[Kendall Libby [:fubar] (he/him)]

(In reply to Dragos Crisan [:dragrom] from comment #9)
> On what repo can i land this script?

https://github.com/mozilla-platform-ops/relops-infra

I've sent you an invite to join.

Comment 11

[Dragos Crisan [:dragrom]]

I cannot fork the repository relops-infra to my account dragoscrisan. Also master branch not exist

Comment 12

[Kendall Libby [:fubar] (he/him)]

(In reply to Dragos Crisan [:dragrom] from comment #11)
> I cannot fork the repository relops-infra to my account dragoscrisan. Also
> master branch not exist

Yes, it's empty; you'll need to:

git init
git commit -m "first commit"
git remote add origin [e-mail]:mozilla-platform-ops/relops-infra.git
git push -u origin master

You've got another invite, for write access to the above repo.

Comment 13

[Dragos Crisan [:dragrom]]

Comment on attachment 8910832 [details]
fabfile.py

https://github.com/mozilla-platform-ops/relops-infra