512754 - Create VMs for l10n dashboard stage area

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Axel Hecht [:Pike]]

We talked to mrz about a staging setup for the l10n dashboard work. For that, we would like to have resources in San Jose to do the following:

database server - in the l10n server, we occasionally see locked up mysql daemons, so we should probably get a non-shared server that we can kick if required.

dashboard server - the dashboard server needs to run 
-- hg poller (daemon to mirror the hg.m.o repos and pushlog)
-- buildbot master
-- buildbot slave
-- django web app

All of these share disk resources, so they should be on a single VM. I'm not sure if the db server should be on a separate VM or on the same, we'll take whatever works best for you, and let's you learn the right lessons on what's needed for production at some point.

In addition, for initial testing purposes, we'd like to start the system based on a local set of hg test repositories.

django web app, buildmaster, and hg server should be accessible via http, so we'll need ports for those.

The django web app should also be accessible via https, so that we can test our ldap auth code and make sure that's https only.

The server should be setup with apache and python, and we should have shell access so that we can install the necessary python modules etc. Local installs will be good enough.

The users on there should share a group so that we can co-admin and install.

Access through mpt tunnel should be good enough.

We'll need accounts for gandalf, me, stas. Not sure about sethb, he'll need to be able to see the web pages there, but I doubt he'll actually do any code deployments or debugging.

I tried to get most questions covered, but I'm sure there are some left, which I'm happy to take a stab on.

Thanks.

Comment 1

[matthew zeier [:mrz]]

Shyam's going to run with this with you since you guys are, I think, closer in timezones.

Comment 2

[Shyam Mani [:fox2mike]]

Short status update :

So we're basically splitting this into two VMs, bm-l10n-dashboard (for the webapps) and bm-l10n-db for the Database. The VMs have been created, but I need to move them to the right network and install the stuff you need.

Comment 3

[Zibi Braniecki [:zbraniecki][:gandalf]]

are we going to have this behind VPN or will it be accessible to the world?

Comment 4

[matthew zeier [:mrz]]

Depends what you need - VPN is great unless that doesn't work for you.

Comment 5

[Shyam Mani [:fox2mike]]

(In reply to comment #3)
> are we going to have this behind VPN or will it be accessible to the world?

Based on what Axel said in comment #0, I assumed VPN will be fine.

Comment 6

[Axel Hecht [:Pike]]

Yeah, we should start with VPN. Once we get to the point of having a public site, it might make sense to think about a public stage, too.

Comment 7

[Shyam Mani [:fox2mike]]

Okay, so bm-l10n-dashboard.mozilla.org and bm-l10n-db.mozilla.org are up and running. I've installed python 2.6.2 on them (use python26 to access the same, it doesn't interfere with the system python)

Your (pike, stas and gandalf) keys are on the boxes, login as root. 

Unfortunately, I have a train to catch in about 60 mins so I really have to run. Someone else from IT can help you with other issues you might face, else I'll be back on Monday and will clean up/fix any issues you might have run into.

Comment 8

[Axel Hecht [:Pike]]

We could use some help on bm-l10n-dashboard to set up apache.

httpd is there as package, but apr is not, and from there it's not getting better. Apparently RHN is disabled, not sure if that's the culprit, or if some local package server isn't found.

Comment 9

[Zibi Braniecki [:zbraniecki][:gandalf]]

bm-l10n-db is set up and running. I'm claiming ownership over that VM and maintaining it.

Here are notes from installation: https://intranet.mozilla.org/L10n:Dashboard/Installation_notes#bm-l10n-db

Comment 10

[Axel Hecht [:Pike]]

Any update on the setup on bm-l10n-dashboard, re comment 8? It's not really usable as it is right now.

Comment 11

[Shyam Mani [:fox2mike]]

(In reply to comment #10)
> Any update on the setup on bm-l10n-dashboard, re comment 8? It's not really
> usable as it is right now.

Yeah, I'll fix them in a bit. Seems like we had some issues during kickstart.

Comment 12

[Axel Hecht [:Pike]]

The packages seem to work now.

The local times of both -db and -dashboard seem to be way off, though.

Comment 13

[Axel Hecht [:Pike]]

Can we get a python26-crypto package installed? One buildbot feature we use depends on the Crypto package, and yum only finds the python-crypo package, which I expect to be for 2.4 only.

Comment 14

[Shyam Mani [:fox2mike]]

Axel,

We'll have to build rpms for it in all probability.

Do you know if build uses a specific version of python-crypto? Or how the install python and related stuff?

Comment 15

[Axel Hecht [:Pike]]

tbh, I don't even know if build needs python-crypto. The piece that I need it for is in dashboard-specific code.

I was kinda hoping that the -crypto package would be around the same place where you got the other python26 packages from.

Comment 16

[Shyam Mani [:fox2mike]]

(In reply to comment #15)

> I was kinda hoping that the -crypto package would be around the same place
> where you got the other python26 packages from.

Heh, lot of pain to get those going, we had to build those....


(In reply to comment #12)
> The local times of both -db and -dashboard seem to be way off, though.

Should be fine now.

Comment 17

[Axel Hecht [:Pike]]

Installed python-crypto from the source for the dashboard user, which builds ok, but I get an error when using it:

	    import crypt
	exceptions.ImportError: /usr/lib/python2.6/lib-dynload/cryptmodule.so: undefined symbol: crypt
	
google hints at that being a python compile problem where it's missing an -lcrypt somewhere :-(

Comment 18

[Axel Hecht [:Pike]]

I tried to compile python2.6 myself on the redhat distro to work around the crypt issue, but that's a ton of work, too.

Shyam and I talked about using a distro for dashboard that comes with support for python2.6. We're having Ubuntu 9.04 on the l10n.mozilla.org server in AMS which doesn't cause that grief.

mrz, would it be OK to use that on a production system? We seem to have the choice between hacking ourselves into deps hell in RH and python 2.6, or use a distro that's not what we have on the other systems. Smells like rock and hard place.

Comment 19

[matthew zeier [:mrz]]

I'm less concerned about a non-RHEL distro on a VM.  On physical hardware, it's either RHEL or SUSE for hardware support.

openvz might be the way to go here too - host node runs RHEL and the VEs run whatever you want.

Comment 20

[Shyam Mani [:fox2mike]]

mrz,

I was thinking the same too, Virtuozzo with say Gentoo or Ubuntu should be far easier to manage than RHEL. Do want to give that a shot?

Comment 21

[matthew zeier [:mrz]]

You should be able to stage this on the Labs openvz servers.

Comment 22

[Shyam Mani [:fox2mike]]

Alright, will bug zandr about access to those.

Comment 23

[Axel Hecht [:Pike]]

Any update on this?

Comment 24

[Shyam Mani [:fox2mike]]

(In reply to comment #23)
> Any update on this?

Yup. I wiped bm-l10n-dashboard and it's now in the process of release upgrading itself to Ubuntu 9.04 after which I'll add your keys and the box is yours. Figured that would be easier to do that mucking around with the labs stuff..since you've already been delayed quite a bit (apologies!).

Will update when it's done.

Comment 25

[Shyam Mani [:fox2mike]]

Alright, bm-l10n-dashboard is up with Ubuntu 9.04.

Reopen if you have any issues. New bugs/IRC if you need any help with this or db.