997738 - Setup Puppet for QA related infrastructure in SCL3

Status: RESOLVED WONTFIX

(Mozilla QA Graveyard :: Infrastructure, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

For an easier management of our machines we want to setup Puppet. It will allow us to easily deploy changes to all of the connected machines. 

The first step will be to update nodes with updates. Later we will also cover the installation process.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Dustin, before we get started with the implementation of manifests and modules for our puppet infra, I have a couple of questions regarding the workflow. It's not that clear to me yet, and maybe we have to (re-)define it a bit, if we have special needs. So I read the Pro Puppet book, which gave some great examples for git. Those we are not able to use in our case given that a Mercurial repo is in use. What's unclear:

1. How do we separate the configuration for staging and production? In case we have to do software updates like for Flash, we want to get those tested on staging first. Only if it works as expected we want to merge the changes to production. We could have an environment on puppetmaster, which holds the latest changes from development, but this wont be in the remote repository. So what speaks strictly against a staging branch in the puppet repository?

2. How to develop and test new features by multiple users? In case of a git repository you can have a helpful post-receive hook, which dynamically can manage environments. So a new environment could be created for a new feature branch. In our case we would have to do that manually because we cannot use any additional branch in the hg repository. So each of us may work on different implementations at the same time. That's why I would not propose to use /etc/puppet/environments/%user% but an env called like the feature. This will cause a bit of mangling with the puppet config on master, which I personally don't like. How shall we do testing? Doing it locally e.g. via Vagrant? Once it is proven to work, we could test on puppetmaster, and eventually merge into default. On the other side we could make use of hg user repositories on hg.mo and switch the mq to the appropriate branch in our own user environment. Maybe that is the best option, because we have patches backed-up remotely, which we can also easily pull from.

3. As we have seen lately I pushed a broken config to our puppet repository, which let puppet stop working completely for us. Can we setup a pre-commit hook, which runs puppet lint, and only let you commit when the complete configuration doesn't have any syntax issues?

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

(In reply to Henrik Skupin (:whimboo) from comment #1)
> 1. How do we separate the configuration for staging and production? In case
> we have to do software updates like for Flash, we want to get those tested
> on staging first. Only if it works as expected we want to merge the changes
> to production. We could have an environment on puppetmaster, which holds the
> latest changes from development, but this wont be in the remote repository.
> So what speaks strictly against a staging branch in the puppet repository?

Depending on how long you want to test such a thing, you can either use your puppet environment for one run, or "pin" a host to your puppet environment for a few days.  Finally, you can write manifests that allow different versions for production and staging, using aspects.

  https://wiki.mozilla.org/ReleaseEngineering/PuppetAgain/Aspects

we've only done this for mozpool so far

  https://github.com/mozilla/build-puppet/blob/master/modules/mozpool/manifests/settings.pp
  https://github.com/mozilla/build-puppet/blob/master/manifests/moco-nodes.pp#L221

> 2. How to develop and test new features by multiple users? In case of a git
> repository you can have a helpful post-receive hook, which dynamically can
> manage environments. So a new environment could be created for a new feature
> branch. In our case we would have to do that manually because we cannot use
> any additional branch in the hg repository. So each of us may work on
> different implementations at the same time. That's why I would not propose
> to use /etc/puppet/environments/%user% but an env called like the feature.
> This will cause a bit of mangling with the puppet config on master, which I
> personally don't like. How shall we do testing? Doing it locally e.g. via
> Vagrant? Once it is proven to work, we could test on puppetmaster, and
> eventually merge into default. On the other side we could make use of hg
> user repositories on hg.mo and switch the mq to the appropriate branch in
> our own user environment. Maybe that is the best option, because we have
> patches backed-up remotely, which we can also easily pull from.

Depending on whether you're familiar with hg or git, you can set either one up in your environment on the puppet master.  I don't really understand hg workflows in general (they seem to involve a lot of different repos and manual application of files in .hg/patches with 'patch'), so I use git.  I have a git repo in my environment on the puppetmaster, along with a github repo, and I use topic branches in those two environments.  Then whenever I need to run puppet agent against my host, I just use --environment=dmitchell.  Admittedly I can't simultaneously use my environment for two projects, but in practice that's not been an issue.

> 3. As we have seen lately I pushed a broken config to our puppet repository,
> which let puppet stop working completely for us. Can we setup a pre-commit
> hook, which runs puppet lint, and only let you commit when the complete
> configuration doesn't have any syntax issues?

You could, but configuring puppet identically on the hg hosts will probably not make the dev services guys happy.  Puppetlint doesn't anything but the most basic syntax errors -- although it would have caught this particular error.  So there's a certain risk of encouraging complacency.

You might have better luck configuring Travis to run on the default branch (this would require setting up and hg -> github mirror, which hwine can help with), and taking care not to merge to production until you get a green checkmark from travis.  If you were to set this up, it's something releng would probably adopt -- but not critical enough that we're going to do the legwork to make it happen.

We've had good success keeping a weather eye on the error notification emails.  In releng's case, a serious syntax error will generate 500-1000 emails in a few minutes, so it's hard to miss.  We just have that many hosts..

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Dustin J. Mitchell [:dustin] from comment #2)
> Depending on how long you want to test such a thing, you can either use your
> puppet environment for one run, or "pin" a host to your puppet environment
> for a few days.  Finally, you can write manifests that allow different
> versions for production and staging, using aspects.

So we have dedicated hosts for staging. That means those would always be connected to staged settings. So dedicated manifests for the staging instance sounds like the best idea.

> Depending on whether you're familiar with hg or git, you can set either one
> up in your environment on the puppet master.  I don't really understand hg

The thing here is that we would need a mirror on github for the qa fork of the puppet repository. I don't wanna have to manually sync changes each time. Not sure if I want to request that right now. So we might want to try with hg.

> > 3. As we have seen lately I pushed a broken config to our puppet repository,
> > which let puppet stop working completely for us. Can we setup a pre-commit
> > hook, which runs puppet lint, and only let you commit when the complete
> > configuration doesn't have any syntax issues?
> 
> You could, but configuring puppet identically on the hg hosts will probably
> not make the dev services guys happy.  Puppetlint doesn't anything but the
> most basic syntax errors -- although it would have caught this particular
> error.  So there's a certain risk of encouraging complacency.

Oh wait. So it wont run on the local machine but the remote one? That's not something I was aware of, but yes you are right. I could have my own hook for it locally.

> You might have better luck configuring Travis to run on the default branch
> (this would require setting up and hg -> github mirror, which hwine can help
> with), and taking care not to merge to production until you get a green
> checkmark from travis.  If you were to set this up, it's something releng
> would probably adopt -- but not critical enough that we're going to do the
> legwork to make it happen.

Yeah. Something for later. More important right now is to get the first systems up. :)

> We've had good success keeping a weather eye on the error notification
> emails.  In releng's case, a serious syntax error will generate 500-1000
> emails in a few minutes, so it's hard to miss.  We just have that many
> hosts..

Ouch! So I will make sure to always run puppet lint locally, especially when I have to modify general files.

Thanks for the update on the puppet topic, Dustin!

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

You don't necessarily need a github mirror.  Just running git-remote-hg on your development environment is sufficient (if a bit fiddly).  But setting up such a mirror isn't hard.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Dustin, now that we move our testing to Taskcluster, do you see anything which we would need from Puppet? At least for Windows and OS X? If not, I would happily close this bug as wontfix.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

I don't think releng has a clear answer to that yet, either.  I'd suggest you close this for now.

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Taskcluster is the future and Puppet is no longer on our list. So marking this bug as wontfix.