889431 - [tracker] Set up servers for ReviewBoard

Status: RESOLVED FIXED

(bugzilla.mozilla.org :: Infrastructure, defect)

Description

[Mark Côté [:mcote]]

As we have briefly discussed with IT, the BMO team is considering replacing Splinter with ReviewBoard[1].  We are ready to start our experimentation, and to that end we would like to have the standard trio of systems (development, staging, production), as we will have to do some custom modifications to integrate ReviewBoard with BMO.

We have forked the ReviewBoard project into our own github repo[2].  We will be working with the release-1.7.x branch.  It's my understanding that the dev box, at least, can be set up to automatically update the code and that releases to staging and production are done by IT.  If that is correct, we can just use the release-1.7.x branch and tag it when it should be deployed to staging/prod.  I could also create a separate branch for dev work, whichever is easier for IT.

I'd like to start with the development box first so we can do the first basic steps necessary for working with BMO (specifically, user authentication).

We filed for security review in bug 874767.  Security doesn't have any problems with ReviewBoard itself but will review our integration strategy when we get to that (as I mentioned, it will only be authentication to start with).

[1] http://www.reviewboard.org/
[2] https://github.com/mozilla/reviewboard

Comment 1

[Jake Maul [:jakem]]

Can we put this right on the Bugzilla web and database nodes? I don't know what sort of load we expect this to generate, but the existing web nodes at least are very under-utilized.

If not it'll take a bit longer while we provision some VMs.

Comment 2

[Mark Côté [:mcote]]

I think that's fine, and actually that would probably make sense for the dev server too (alongside bugzilla-dev).  I'd like to hear :glob's opinion first, though, since he's been researching ReviewBoard for the last few weeks.  I'll check in with him tonight if he doesn't comment here first.

Comment 3

[Mark Côté [:mcote]]

Byron has some reservations about putting the systems on the webheads, at least for prod, since BMO load is fairly bursty, and we don't know what load ReviewBoard will generate.

However I think it's perfectly fine to put a development instance on bugzilla-dev.  Can you start with that?

Comment 4

[cmtalbert]

Jake, this request for a *dev* server has been open for 28 days at this point. We had planned to launch a production system for developers by the end of the quarter. We need to get a jump on this if we're going to make that date.

We definitely need this up sooner rather than later in order to get developer feedback so that we can better move forward as well as for our security review. 

The only real difference in setting this up on dev versus production is where physically does the code live and what user database does reviewboard authenticate against (either dev-bugzilla's db or production bugzilla's db). Other than that, there is no integration with bugzilla, as I understand it.

Comment 5

[:glob ✱]

(In reply to Clint Talbert ( :ctalbert ) from comment #4)
> Jake, this request for a *dev* server has been open for 28 days at this
> point.

to clarify, we're asking for it to be setup on the existing dev server (bugzilla-dev.allizom.org / web5.stage.bugs.scl3.mozilla.com), not for a new server to be deployed.

Comment 6

[Jake Maul [:jakem]]

Spent some time with :fubar today playing around with this, trying to get something going.

Operationally, it's a disaster so far. Among other things, it uses some sort of templating system to build the actual .wsgi and config files (and maybe more things we haven't discovered yet)... but everything it writes is absolute paths. When we build on the admin node, the paths are different than where it ultimately runs on the web nodes, so nothing works. We fixed some stuff up by hand, but it's still throwing a 500 ISE error... unfortunately it doesn't log any error, so we have no idea what's wrong.

Their instructions also don't use any sort of isolation (like virtualenv), so if you follow them precisely you wind up with many packages installed via easy_install at the system level, instead of containerized within the app.

Basically, they've tried to make it easy for people to deploy, and in so doing made it extremely complicated unless your environment happens to match their expectations precisely.

We'll keep banging at it. I fear it will never be very easy to update to new versions of the code. It's feeling a lot like something that we might be able to get set up and running, but won't be very maintainable.


One possibility we may be forced into is to set up a one-off server for this, and do things as close as possible to the documentation. That will probably work more easily, but gives up a lot of nice things... complete puppetization, redundant servers, rapid redeployability, etc.

Comment 7

[Julien Vehent]

From an OpSec point of view, this isn't great. Being able to upgrade is critical, so is puppetizing the servers. The full recommendation is here: https://mana.mozilla.org/wiki/display/SECURITY/System+Security+Policy

If this tool cannot match these requirements, it shouldn't, at the very least, reduce the security of other components such as bugzilla. Do we have the option to isolate it somewhere?

Comment 8

[Curtis Koenig [:curtisk-use curtis.koenig+bzATgmail.com]]]

removing my alias from the sec-review assigned as this needs to go to triage to get a proper app-sec assignment

Comment 9

[Christian Hammond]

Hi guys,

I'm one of the leads behind Review Board, and this discussion was brought to my attention. Thought I'd see if I could help and clarify parts of our installation/upgrade process.

An initial site creation does create a .wsgi file with absolute paths, and it creates a settings_local.py and Apache config. These are all meant to be starting points that can be customized as needed. In the 99% case for our customers, this all works with their installation pretty well, but more specialized setups often need to tweak these or outright replace them. An upgrade will not overwrite your changes.

When upgrading a Review Board site installation using 'rb-site upgrade', what we're mostly doing are applying database migrations and, in certain (rare) circumstances, flagging required changes for an admin to make to configuration files. That should be safe enough.

We personally have a number of Review Board servers built out for us using Chef as part of our Review Board hosting service, RBCommons, so I'd imagine it would work with Puppet. In our case, we ship/generate our own versions of settings_local.py, the .wsgi file, and the Apache configuration based on our Chef ruleset, instead of using the auto-created ones from rb-site.

The one missing piece I think you're hitting is doing all this inside a virtualenv. We have users who have done that successfully, but I'll admit that I haven't attempted it, as all our instances have their own VMs. It might take a little more work up-front, but it seems quite possible to put it in a virtualenv, and assuming an upgrade is done within that environment, I don't see upgrades being a problem there. Might be worth asking on our mailing list, [e-mail], to see if anyone with experience in doing this could respond. I'm happy to add any patches to make life easier in this way, and to otherwise help with that process.

You mentioned that after you have fixed up some paths, you still got a 500 error. One tip is to add 'DEBUG = True' to your $sitedir/conf/settings_local.py file temporarily, just to see what the error is. These errors should be logged to your Apache error log, though.

Please let me know if I can assist at all in this.

Comment 10

[:glob ✱]

(In reply to Curtis Koenig [:curtisk] from comment #8)
> removing my alias from the sec-review assigned as this needs to go to triage
> to get a proper app-sec assignment

is this different from the security review already tracked by bug 874767?

Comment 11

[Julien Vehent]

(In reply to Byron Jones ‹:glob› from comment #10)
> (In reply to Curtis Koenig [:curtisk] from comment #8)
> > removing my alias from the sec-review assigned as this needs to go to triage
> > to get a proper app-sec assignment
> 
> is this different from the security review already tracked by bug 874767?

Nope, it's the same thing. I marked the sec review bug as a blocker to this.
I can't speak for the code review, but from an infrastructure standpoint, what options do we have to make ReviewBoard match our platform requirements? In a nutshell, we don't want to deploy something that can't easily (<2 hours) be upgraded to the newest version.

Comment 12

[Daniel Veditz [:dveditz]]

We don't need a sec-review? flag when there's an explicit sec-review dependency bug.

Comment 13

[Yvan Boily [:ygjb][:yvan]]

ReviewBoard is cleared from a testing perspective, but the Security Assurance team is still working on a recommendation for this particular usage.  We will provide an answer on Monday, August 19th.

Comment 14

[Mark Côté [:mcote]]

Would like to hear if comment #9 helps at all.

In terms of isolation, it does not have to run directly alongside Bugzilla; it only accesses BMO via API calls (XMLRPC atm).  I believe it *might* have to be on the same domain as BMO for log-in calls to work, although even that can be fixed.

The first phase of this project is really only for testing & experimentation--people can use it for reviews, but there will be minimal integration for now (similar to GitHub pull requests).  If people like it, I will spend some time better integrating it with BMO, with the goal of releasing a fully usable system by year's end.  So from that perspective, the first system can be temporary, if you want to defer a proper setup until next quarter.  There's a good chance that I will need it to live closely with BMO for the next milestone, but not right now.

Comment 15

[Kendall Libby [:fubar] (he/him)]

Will put in a VM request asap; any special resource requirements (cpu/memory/disk)? I think we generally go with 1 cpu, 2gb ram, and 40gb disk.

Also, what other systems will this need to connect to? Having it on bugzilla-dev meant access to bmo stuff was pretty simple. A new vm may require some extra steps..

Comment 16

[Mark Côté [:mcote]]

It's completely independent except that I believe it has to run on the same domain so that the AJAX requests work--although there may be a workaround for that.  Everything is done, at the moment, via API calls to the Bugzilla server.  Eventually we may have tighter integration, but hopefully not much, so that other Bugzilla installations could easily use ReviewBoard.  But we can address that in a future quarter.

Comment 17

[Mark Côté [:mcote]]

Just to reiterate, since it might have gotten buried above, please deploy from the release-1.7.x branch.  It is up to date with the branch of the same name in the official ReviewBoard repo.

As also mentioned above, we want one RB server for bugzilla-dev.allizom.org and another for bugzilla.mozilla.org.  Perhaps you could set up both on the same VM?  The only change required for the production version would be, after the initial configuration step, changing the "BUGZILLA_URL" setting in settings_local.py.  We need the dev one for now, but we'd like the prod one as well by the end of Q3, assuming testing on dev goes fine.  Thanks!

Comment 18

[Kendall Libby [:fubar] (he/him)]

So, I'm working on setting this up on its own vm, but under virtualenv to try and keep it as close to possible as our other environments. I'm running into a bunch of issues with sorting out prerequisites.

First, since we're using our own fork, I can't easy_install to get everything. But there's no *full* list of all the requirements. The install docs are a start, but they're missing bits (docutils, markdown and mimeparse, offhand). 

Now it's failing on lessc, which appears to want to be installed by npm, which wants node... 

Is there an actual requirements list from ReviewBoard, so that I can skip over the frustrating time wasting?

Comment 19

[Mark Côté [:mcote]]

I assume you are following this? http://www.reviewboard.org/docs/manual/1.7/admin/installation/linux/

There is a setup.py in the root which I imagine could be used via "python setup.py install" to do most of what easy_install accomplishes.

Comment 20

[Christian Hammond]

Hi Kendall,

The reason it's needing stuff like lessc, npm, node, etc. is due to building from source. In the 99.99% case we deal with, people install from eggs, where that stuff is not needed (it's part of our build and release process), so this is a pretty custom setup.

If you can get away from using a fork, and use the eggs (you can pass egg filename to easy_install -- do not use pip, as it's incompatible), all the node-related stuff will go away.

To help with the eggs, here's what's in my development virtualenv (minus things needed solely for unit tests):

Django-1.4.6
Djblets-0.7.11
Markdown-2.3.1
MySQL_python-1.2.4
PIL-1.1.7
Pygments-1.5
bzr-2.6b2
django_evolution-0.6.9
django_pipeline-1.2.24
docutils-0.9.1
feedparser-5.1.2
hgsubversion-1.5.1
jsmin-2.0.2
mercurial-2.6.2
mimeparse-0.1.3
paramiko-1.9.0
pycrypto-2.6
python_memcached-1.48
pytz-2012h
recaptcha_client-1.0.6
setuptools-0.6c11


It's possible this will vary a bit, but it should help.

If you *do* need to build from source, you'll need node, npm, lessc, and that's probably it for this particular release. 2.0 will require uglifyjs and xgettext.

Comment 21

[Mark Côté [:mcote]]

Perhaps I could create a build from the source for the initial deployment.  Eventually I want to move this to an extension, but that requires some work on the Review Board core, and I'm not sure how long that will take.

Comment 22

[Kendall Libby [:fubar] (he/him)]

(In reply to Mark Côté ( :mcote ) from comment #19)
> I assume you are following this?
> http://www.reviewboard.org/docs/manual/1.7/admin/installation/linux/
> 
> There is a setup.py in the root which I imagine could be used via "python
> setup.py install" to do most of what easy_install accomplishes.

Yes, that's where things bomb out and I have to go back and install the missing module or software. As it happens, it looks like I'm most of the way through Christian's list, so perhaps the end is in sight. :)

Comment 23

[Mark Côté [:mcote]]

Okay I have an egg of the current source up at http://people.mozilla.com/~mcote/ReviewBoard-1.7.13.dev-py2.7.egg.  That might make your life easier. :)

Comment 24

[Kendall Libby [:fubar] (he/him)]

dev site up and running at reviewboard-dev.allizom.org, though it's currently only accessible to the ateam through the MozillaVPN. for some reason my first pass at setting up a zeus vip didn't work; will try again after I wrap up a bunch of DXR stuff.

Comment 25

[Mark Côté [:mcote]]

This bug is still open as we need a production (mozilla.org) server at some point, after the security review in bug 978976 is done.

Comment 26

[Kendall Libby [:fubar] (he/him)]

Added Bug 1021929.

Docs for prod at https://mana.mozilla.org/wiki/display/websites/reviewboard.mozilla.org

I believe the only remaining bits are updating the mysql schema (bug 1013156), nagios monitoring, backups, and actual user testing.

Comment 27

[Mark Côté [:mcote]]

I would say this blocks bug 1021929, not the other way around, no?

Comment 28

[Mark Côté [:mcote]]

Moving the sec review blocker to the main deployment bug (bug 1021929).

Comment 29

[Mark Côté [:mcote]]

(In reply to Kendall Libby [:fubar] from comment #26)
> Added Bug 1021929.
> 
> Docs for prod at
> https://mana.mozilla.org/wiki/display/websites/reviewboard.mozilla.org
> 
> I believe the only remaining bits are updating the mysql schema (bug
> 1013156), nagios monitoring, backups, and actual user testing.

We now have monitoring, backups, and on-going testing in the form of dogfooding.  The schema remains an annoying issue for new deployments but is already fixed manually in production.  I believe this bug can be closed now. \o/