1159951 - move socorro-infra/puppet into socorro repo

Status: RESOLVED WONTFIX

(Socorro :: Infra, task)

Description

[Robert Helmer [:rhelmer]]

Right now we use puppet in two places:

1) packer provisioning, which is then saved as an AMI
2) puppet is updated and run by the LC when an ASG spins up a new node

If we want to do immutable infra, where the AMI is the only thing that changes (other than configuration injected into the environment) then I think we don't want to do the "update" part in step #2.

Also, we need to have a way to trigger a deployment that produces an AMI with a single SHA - if we keep puppet in a separate repo and allow it to trigger deployments, then we have two SHAs to keep track of (the socorro repo one, and also the socorro-infra repo)

It's probably simplest to just put socorro-infra/puppet into the main socorro repo. I don't think there's anything particularly tying it to our infra, since really all it does is 1) initial provisioning 2) start up appropriate services for a given role.

This seems vaguely wrong to me, but I can't think of a better way. I think we really do want puppet baked into the AMI, and we really do want an AMI with a single SHA, though.

Comment 1

[Robert Helmer [:rhelmer]]

jp and I discussed this today and could figure out a better way than comment 0. What do you think?

Comment 2

[Daniel Maher [:phrawzty]]

I agree that the current mechanism whereby a tarball containing Puppet is downloaded and parsed at Provision-time is janky. If we make the shift to a model where the AMI is the base unit, then it makes sense to just bake everything into that AMI when it's created via Packer (including the Puppet stuff). Recall that we were initially trying to *avoid* constantly rebuilding AMIs, which is the root cause of our current workflow - if we're now OK with this practice, there is no reason not to just bake everything in from the get-go.

Tangent: we could, for example, fully templatise every config file and then fill a Hiera backend with environment-specific settings to be applied at Provision time.  Hell, if we want to tumble even further down the rabbit hole, we could explore consul-template for this as well. :)

Back on track, there is no technical reason why the socorro-infra and socorro repos couldn't be integrated.  That said, given that the plan (afaik) is to break up socorro into multiple repos, wouldn't this proposed integration constitute a step in the wrong direction?  In other words, if we're already going to have to deal with split repos, why not deal with that problem now?

Finally, I'm really not clear on what you mean by "a deployment that produces an AMI with a single SHA", so I'm unsure how to respond here.  Please explain further.

Comment 3

[JP Schneider [:jp]]

The SHA concern is the one and only reason to move at least the puppet manifests to the socorro base repo.  This is what we were talking through yesterday which led to that:

* We line up SHAs for the commit/tag to coordinate which AMI gets promoted from stage to prod.  
* We seem to want updates to puppet to be pushed to staging and prod when the change to puppet is made (right now, that'd be a new commit in our infra repo, but not a new commit in the socorro base repo.)
* Thus, if that puppet run did trigger a deploy, the SHA tag applied NewSparklyAMI would be the same exact SHA tag on the TiredOldAMI, and we would potentially land that TiredOldAMI in prod instead of the one we applied updates to.

If we were in a model where we had one master infrastructure repo, akin to mofo_system_configurations, which supplied manifests for all the apps, I'd definitely agree this is a step in the wrong direction.  As is, though, we're still splitting up infra repos to be project dependent.  For example, we wouldn't be using socorro-infra to manage sumo.  If I understand the history correctly, we don't want a centralized type of repo so we can prevent running puppetmasters, but I might not be aware of the history there.

Comment 4

[Robert Helmer [:rhelmer]]

(In reply to Daniel Maher [:phrawzty] from comment #2)
> Back on track, there is no technical reason why the socorro-infra and
> socorro repos couldn't be integrated.  That said, given that the plan
> (afaik) is to break up socorro into multiple repos, wouldn't this proposed
> integration constitute a step in the wrong direction?  In other words, if
> we're already going to have to deal with split repos, why not deal with that
> problem now?

I have the same concern. If we continue using puppet as the repos are split and continue to use this model, then we'll need to have puppet in each split-out repository.

I think we might be able to avoid this if we make our deployment model more generic, but I am OK with having to keep puppet files around if that doesn't pan out.

Comment 5

[Daniel Maher [:phrawzty]]

(In reply to Robert Helmer [:rhelmer] from comment #4)
> I have the same concern. If we continue using puppet as the repos are split
> and continue to use this model, then we'll need to have puppet in each
> split-out repository.

Maintaining identical files in a bunch of repos is crazy-town and we all know it. :)

(In reply to JP Schneider [:jp] from comment #3)
> * We line up SHAs for the commit/tag to coordinate which AMI gets promoted
> from stage to prod.

Ok, so let's attack the problem from a different angle: what if we didn't tag and track AMIs in this fashion?  Is there another way we could be doing things?
 
> If we were in a model where we had one master infrastructure repo, akin to
> mofo_system_configurations, which supplied manifests for all the apps, I'd
> definitely agree this is a step in the wrong direction.  As is, though,
> we're still splitting up infra repos to be project dependent.  For example,
> we wouldn't be using socorro-infra to manage sumo.  If I understand the
> history correctly, we don't want a centralized type of repo so we can
> prevent running puppetmasters, but I might not be aware of the history there.

Some background information: right now the Socorro repo is a monolithic structure, itself containing multiple apps and other sundry items.  Rob's intention is to split the repo out, micro-service style, into as many smaller independant (but related) repos as appropriate.  This is what I'm referring to in comment #2.  Puppetmasters and Mofo infrastructure aren't currently part of the scope of this conversation.

Comment 6

[Daniel Maher [:phrawzty]]

Imho this bug is just a symptom of a larger problem: we're not being honest with ourselves about our own methodology.  If we want to go the immutable infra route, where we deploy AMIs out of the box, that's fine - but we're not actually doing that at all.  Right now we're straddling an ill-defined line somewhere at the intersection between pre-baked AMIs, provisioned instances, and long-running mutable boxes.

We need to make some clear decisions about this.  Continuing to dance around the topic is getting us nowhere.

Comment 7

[JP Schneider [:jp]]

aws ec2 describe-images --filters Name=tag:appsha,Values=222222222222 Name=tag:puppetsha,Values=111111111 might be a workable solution.  We'll have to map out how that plays with infra during this or that situation with repo statuses/pushes. 

Definitely agree that it's hella gross managing all those same files in multiple repos, but I'm also thinking of when this expands out and we host sumo....aren't we going to be in precisely that same situation?  I am sure it will be to a lesser degree if we have only 2 or 4 puppet repos to manage in the one-infra-repo-per-"project" versus the puppet-in-every-micro-service-repo setup.

All the infra (except for the logging server and buildbox demo we're experimenting with) are fully baked AMI's, and immutable.  I feel like we're pretty solidly confirmed that anything that's going to be deployed at the service/app layer (versus the ops layer) is 100% immutable.  Well...elasticsearch to a degree.  The question is how we want that to be triggered, really.

Comment 8

[Robert Helmer [:rhelmer]]

(In reply to Daniel Maher [:phrawzty] from comment #6)
> Imho this bug is just a symptom of a larger problem: we're not being honest
> with ourselves about our own methodology.  If we want to go the immutable
> infra route, where we deploy AMIs out of the box, that's fine - but we're
> not actually doing that at all.  Right now we're straddling an ill-defined
> line somewhere at the intersection between pre-baked AMIs, provisioned
> instances, and long-running mutable boxes.
> 
> We need to make some clear decisions about this.  Continuing to dance around
> the topic is getting us nowhere.

I agree, that's why we're trying to figure out how *not* to re-deploy puppet (or any other changes) after the AMI has been pushed, hence this bug.

That said, I wonder if we shouldn't just punt on this for today and not block - "punting" in this context means:

* no longer fetch puppet tarball from S3 in launch config
* pushing changes to socorro-infra/puppet will not result in new AMI

The only thing that will result in a new AMI being generated is pushing to the app ("socorro") repo, which will pull in the socorro-infra/puppet files current at the time. If you touch socorro-infra/puppet and want to re-spin an AMI, there'll need to be a revision bump to the app repo.

Or, we could do something like comment 7.

Comment 9

[Robert Helmer [:rhelmer]]

Not currently working on this.

Comment 10

[JP Schneider [:jp]]

Nobody is.