715029 - support configuring rabbitmq users and vhosts from puppet

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task, P4)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

This will help to standardize such things for both pulse and the releng rabbitmq instances.

There are probably puppet resources out there that do this already, which just need to be imported.

Comment 1

[Dustin J. Mitchell [:dustin] (he/him)]

In case I lose track of this, here's a Python script that will synchronize the *actual* list of vhosts with the list of vhosts in a file.  This could easily be extended to a 'vhosts.d' kind of thing.
----
#! /usr/bin/python

import sys
import subprocess

new_vhosts = set([ s.strip() for s in open(sys.argv[1]).readlines() ])

def get_old_vhosts():
    txt = subprocess.Popen(["rabbitmqctl", "list_vhosts"], stdout=subprocess.PIPE).communicate()[0]
    lines = txt.split('\n')
    vhosts = lines[1:-2]
    return vhosts
old_vhosts = set(get_old_vhosts())

for new_vhost in new_vhosts - old_vhosts:
    subprocess.check_call(['rabbitmqctl', 'add_vhost', new_vhost])

for old_vhost in old_vhosts - new_vhosts:
    subprocess.check_call(['rabbitmqctl', 'delete_vhost', old_vhost])
----

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Better version:

----
#! /usr/bin/python

import os
import sys
import subprocess

# format of each file is
#
# ..whatever..
# vhost=/foo/bar
# ..whatever..

def get_new_vhosts():
    dir = sys.argv[1]
    vhosts = []
    for filename in os.listdir(dir):
        filename = os.path.join(dir, filename)
        if not os.path.isfile(filename):
            continue
        lines = [ s.strip() for s in open(filename).readlines() ]
        vhost = None
        for line in lines:
            if line.startswith('vhost='):
                vhost = line.replace('vhost=', '')
        assert vhost
        vhosts.append(vhost)
    return vhosts
new_vhosts = set(get_new_vhosts())

def get_old_vhosts():
    txt = subprocess.Popen(["rabbitmqctl", "list_vhosts"], stdout=subprocess.PIPE).communicate()[0]
    lines = txt.split('\n')
    vhosts = lines[1:-2]
    return vhosts
old_vhosts = set(get_old_vhosts())

for new_vhost in new_vhosts - old_vhosts:
    subprocess.check_call(['rabbitmqctl', 'add_vhost', new_vhost])

for old_vhost in old_vhosts - new_vhosts:
    subprocess.check_call(['rabbitmqctl', 'delete_vhost', old_vhost])
----

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: update-rabbitmq-vhosts.py

OK, this version seems to do everything I want.  The idea is to drop config files into a .d directory that specify vhosts and users, and this script will make it so.  Note that it does *not* set users' passwords - that will need to be done manually.

This won't play perfectly with clusters, but it won't be bad, either.  Basically, it will try to set up vhosts, users, etc. on each node in the cluster.  This may fail if it tries to make changes on multiple nodes at the same time, but one operation will succeed, and on the next puppet run everything will be fine.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

OK, this has now landed.  I set up the following for rabbit2/rabbit3, representing the config I found there:

    rabbitmq::user {
        'cltbld':
            tags => [ 'administrator' ];
        'buildapi':
            tags => [ ];
    }   

    rabbitmq::vhost {
        '/':
            perms => [ 'cltbld: .* .* .*' ];
        '/buildapi':
            perms => [
                'cltbld: .* .* .*',
                'buildapi: .* .* .*',
            ];  
    }   

so (catlee) note that you won't be able to change those things via API anymore - puppet will reset them.  Please file an IT bug to handle any such changes.