728386 - [bedrock] Make www-dev served off the "dev" branch of bedrock

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[James Long (:jlongster)]

We're at the point where multiple devs are working on bedrock, and we'll start updating the site a lot, so we need to formalize our process with regards to git branches.

Currently we have everything on master. Two problems I have with this:

* prod should not update so easily. I'm worried that it's based off the same branch that we develop on. If someone ran an update on the site, they need to know that any changes they pull in are meant to be pulled in.

* devs need to merge to master to show stuff on the dev site

I'd like to branch off these use cases:

* Each dev gets their own branch and dev site. So bedrock/steven would be steven's branch and accessible at www-steven.allizom.org.

* Stage is based off of next-stage and prod is based off of next.

I'm open to opinions on this, but if we can't come up with any other solution I'd like to use this system for now.

Comment 1

[James Long (:jlongster)]

After discussing this with wenzel, I think we've decided on something a little different. The dev stuff stays the same (we'd like dev sites that run off a branch named by the dev), but stage and production can run off master. 

Stage will auto-update and production will be manual. Nothing should be pushed to master unless it's ready to go to stage to be QAed. Note that if something is living on stage but is not pushed yet, a simple `git pull` on the prod site will pull it in. Is that dangerous?

Comment 2

[Jake Maul [:jakem]]

Hmm... well... I'm not particularly thrilled about any of this. :)

We've kinda standardized on "*-dev.allizom.org" for dev sites. As far as I know, no other project (including AMO) has per-developer dev sites. Makes me wonder why bedrock should be different. Fortunately bedrock has its own cluster, so this is at least feasible... but I'm a little uncomfortable with it nonetheless. It becomes a bigger maintenance project to keep track of the different Apache vhosts, DocumentRoots, update cronjobs, and config files. Especially since to do it right means the django *and* PHP sides of the app. It seems virtually guaranteed that things will get out of sync. For example, managing gmap_api keys or databases for each site (need to be different, probably) vs something like basket or arecibo (common to all).

That said, I really do like the idea of a separate 'dev' branch, and then pushing to a separate branch for stage/prod. I'm just concerned about the feasibility of maintaining a separate dev branch and site for each developer. Definitely a separate branch is none of my concern, so I'm really only concerned about the feasibility of a separate site for each developer. Would it be feasible for each developer to work on their own branch, then merge to a common "dev" branch that the "www-dev.allizom.org" site uses? Possibly locally hosting a site running their own branch for their own testing... I think that's pretty common.


On stage... stage is intended not just to be a way to QA *code*, but also to QA *deployments*. That is, staging is a place for us to test that the code we're about to deploy on top of the current prod code actually deploys cleanly. If stage becomes a rolling target, we don't have this assurance anymore- stage might be working, but we can no longer guarantee that a straight 'git pull' from old-prod to new-prod will work. Generally what might happen would be things like bugs getting fixed in the stage env (settings/local.py, Apache configs, etc... things outside git), and then these get forgotten or otherwise not applied to production when the push happens.

Whether or not this is a major concern, I can't say for sure. We (web ops) intentionally decided that our staging environments should be manually deployed for this specific reason. But maybe this is overly paranoid, and problems are unlikely. Then again, this *is* a pretty important site... :)


One thing to consider is that a simple 'git pull' is usually not enough (although it might be for bedrock/django right now). Typically you want to 'git pull', update submodules (vendor), compress_assets, 'svn up' locale data and compile it, make any database schema changes, remove any random *.pyc files that might be outdated, make any relevant Apache config and settings/local.py changes, "touch" the wsgi file so the django app gets reloaded properly, etc... then deploy the whole thing at once.

That's what led us to the idea that staging should function as a test deployment... simply that "deploy" is often non-trivial, and it can be very easy to overlook or forget about things that happened along the way.

I think AMO may have solved this problem by creating a 4th environment... but I don't deal with them much, oremj or clouserw would know more on that.



I do agree that there's no obvious reason why stage and prod cannot operate off of the same branch, though. As long as prod is pushed manually, that actually makes good sense to me. Maintaining separate branches for them might actually encourage deviation between them, which would be bad.


One alternative that the SUMO folks use is to 'tag' releases with a datestamp, then deploy a particular tag during pushes ("Push tag 2012-02-16 to stage", "Push tag 2012-02-16-1 to prod", etc). This is probably better for historical and for rollback purposes. But generally speaking, we really never roll back... we roll forward with a new version. So maybe this is a theoretical benefit that never materializes in practice.

Comment 3

[Shyam Mani [:fox2mike]]

FWIW sheriffs does this. develop branch for dev, master for stage and production. Works pretty fine. I'm with Jake and I don't think we should have a per dev dev/stage site etc...that throws the spanner in the works.

Comment 4

[James Long (:jlongster)]

Thanks for the detailed response!

(In reply to Jake Maul [:jakem] from comment #2)
> Hmm... well... I'm not particularly thrilled about any of this. :)
> 
> We've kinda standardized on "*-dev.allizom.org" for dev sites. As far as I
> know, no other project (including AMO) has per-developer dev sites. Makes me
> wonder why bedrock should be different. Fortunately bedrock has its own
> cluster, so this is at least feasible... but I'm a little uncomfortable with
> it nonetheless. It becomes a bigger maintenance project to keep track of the
> different Apache vhosts, DocumentRoots, update cronjobs, and config files.
> Especially since to do it right means the django *and* PHP sides of the app.
> It seems virtually guaranteed that things will get out of sync. For example,
> managing gmap_api keys or databases for each site (need to be different,
> probably) vs something like basket or arecibo (common to all).

A lot of webdevs I've talked to have really wanted per-dev sites (a better term is per-branch sites). I think that's a great idea. However, you've made me realize that it's not really something you all should maintain.


> Would it be feasible for each developer to work on their own branch, then
> merge to a common "dev" branch that the "www-dev.allizom.org" site uses?
> Possibly locally hosting a site running their own branch for their own
> testing... I think that's pretty common.

It kind of is, bug mozilla.org is so content and engagement-heavy that devs will *always* be pushing to dev to show stuff to engagement. And then the problem is that there's a bunch of stuff on dev that can't go out yet but we're supposed to be pushing all this over stuff that has been pushed to it.

I was thinking that each dev has their own site, and when a feature is done they push it to dev to "queue" it up for release.

It comes down to this: if stuff is being pushed to dev, how and when do we freeze it and make sure that's it's all the stuff we want to be pushing out in a certain week?

> 
> 
> On stage... stage is intended not just to be a way to QA *code*, but also to
> QA *deployments*. 

That makes sense. We can keep stage manually updated.

> One alternative that the SUMO folks use is to 'tag' releases with a
> datestamp, then deploy a particular tag during pushes ("Push tag 2012-02-16
> to stage", "Push tag 2012-02-16-1 to prod", etc). This is probably better
> for historical and for rollback purposes. But generally speaking, we really
> never roll back... we roll forward with a new version. So maybe this is a
> theoretical benefit that never materializes in practice.

I like the idea of that, but not sure how it helps the problem I pointed to above. I'm concerned that devs will push stuff to dev just to show it to engagement even if it shouldn't go out in the next release.

Comment 5

[Corey Shields [:cshields]]

James,

What is our action item here?  You want to go with a similar method to sheriffs?

Comment 6

[Corey Shields [:cshields]]

Also for a brief overview of our dev, stage, and prod environments check out https://mana.mozilla.org/wiki/display/websites/Home

Comment 7

[Jake Maul [:jakem]]

James,

How does this sound: a 4th environment, called something like "next-stage". This would be the 'queue' where you push things you want to go live during the next prod push. It could actually be an auto-updated site of the master branch, with stage and prod being manually pushed versions of it... although maybe a separate branch is "safer".

My thinking here is that dev can remain a sort of wild-west, where you can push things all the time without worrying about if or when it will make it upstream. When you want to make something live, you push to next-stage and see how it looks. If it looks good (everything merged nicely, no obvious problems), we do a manual deploy on stage, QA looks it over and approves, then we push to prod.



It sounds like what you're thinking about is really "feature branches", with a site backing each one. My concern there is that on the surface this seems unmaintainable from an IT perspective... perhaps it could be made to work if we could somehow spin up Apache configs and docroot's based on git branches, but this is quite a long way from our standard setup and I don't know that it's really feasible. We're talking about this internally now.

Comment 8

[James Long (:jlongster)]

(In reply to Jake Maul [:jakem] from comment #7)
> James,
> 
> How does this sound: a 4th environment, called something like "next-stage".
> This would be the 'queue' where you push things you want to go live during
> the next prod push. It could actually be an auto-updated site of the master
> branch, with stage and prod being manually pushed versions of it... although
> maybe a separate branch is "safer".

That sounds awesome! As much as I hate to require support for a 4th site, I think that solves a real need we have. To have a real preview of what is going to be pushed the next week would be really helpful.

I just hashed this over with wenzel some, and I think we've clarified a few things:

* Like you said, dev can be a wild wild west of commits, where people push stuff just to show other Mozilla people

* All development happens on local branches/forks anyway, so they push to master when it's ready to go out (never squash stuff, since master and dev can't diverge because you'd never be able to push to dev again. You can squash stuff before you commit to dev.)

* QA happens on the "next-stage" site, which is great because it's a true representation of what's going out next. This is what we don't have right now.

"next-stage" could be an automatically updated version of master. Or a different branch. It's up to you. If stage/prod is always manually updated, it shouldn't be an issue, but if you think it's safer we can use a different branch.

> 
> My thinking here is that dev can remain a sort of wild-west, where you can
> push things all the time without worrying about if or when it will make it
> upstream. When you want to make something live, you push to next-stage and
> see how it looks. If it looks good (everything merged nicely, no obvious
> problems), we do a manual deploy on stage, QA looks it over and approves,
> then we push to prod.
> 
> 
> 
> It sounds like what you're thinking about is really "feature branches", with
> a site backing each one. My concern there is that on the surface this seems
> unmaintainable from an IT perspective... perhaps it could be made to work if
> we could somehow spin up Apache configs and docroot's based on git branches,
> but this is quite a long way from our standard setup and I don't know that
> it's really feasible. We're talking about this internally now.

Essentially what I was asking for was feature branches, yes. I wasn't going to pursue dynamically created branches (though I did create some shell scripts for that last year: https://github.com/jlongster/servable-branches), but I was hoping to get a few static ones setup.

I agree that it shouldn't be an you guys' plate though. Managing that and adding config values and such would be a pain. Don't worry about it!

Comment 9

[James Long (:jlongster)]

(In reply to Corey Shields [:cshields] from comment #6)
> Also for a brief overview of our dev, stage, and prod environments check out
> https://mana.mozilla.org/wiki/display/websites/Home

Thanks for that, it helps. mozilla.org's needs make it difficult to use a single dev site, but Jake's solution for an intermediate site between dev and stage is a nice solution to that.

Jake, let's move forward with that. It's not urgent, but hopefully by March we can have it set up?

Comment 10

[James Long (:jlongster)]

We're starting to develop on bedrock a good bit and hitting the problem that has been discussed above.

Two things that would really help us:

* Change www-dev to be based off the "dev" branch in bedrock
* Make a 4th site called something like "next-stage" that is based off of master and is auto-updated

Is there any chance this could be done soon?

Comment 11

[James Long (:jlongster)]

You know what, I may be overthinking this. Let's take this in small steps.

Can you just make www-dev based off of the "dev" branch for now? We can run the site locally and look at master ourselves. We don't need a 4th site to show that off.

Comment 12

[Jake Maul [:jakem]]

Okay, www-dev is now on the git branch 'dev'. I simply added tweaked the update script to look like this:

git pull -q
git checkout dev -q
git submodule -q sync
git submodule -q update --init
pushd vendor > /dev/null
git submodule -q sync
git submodule -q update --init
popd > /dev/null

I'm not entirely certain that this is completely ideal (perhaps checkout should be before pull, or maybe only one of the two is needed, or maybe once we're on 'dev' we no longer need to 'checkout' at all and it will stay on 'dev'), but it works.