617129 - beefier/real hardware for http://build.mozilla.org

Status: RESOLVED WORKSFORME

(Infrastructure & Operations :: RelOps: General, task)

Description

[Mike Taylor [:bear]]

We are going to be opening up to the public more services and right now our current set is being served by a build.m.o which I understand is a VM :(

We would need something that can handle loads from multiple dynamic web apps (pylons) as well as multiple cron jobs that will be querying data and aggregating results.

Open-ended I know, but I wanted to get the conversation started :)

Comment 1

[Shyam Mani [:fox2mike]]

Really depends on what you want to be doing/what levels of access it needs to the build network. If it's a simple site, we can stick it on any of the existing clusters, which we can scale.

Comment 2

[Mike Taylor [:bear]]

it doesn't need access, more so that internal build servers will be pushing to it I think - but it would be something that we would need to be able to tweak and configure.

I'm thinking like an nginx front end that has multiple pylon/django processes - is that what is available in the cluster now?

Sounds like we need to map out what services will be public facing and what their interactions are and then start filing detailed bugs - I'll morph this to more of a tracking bug after talking it over.  I should buy you some beer next week to haggle out the messy details :)

Comment 3

[Shyam Mani [:fox2mike]]

Whoa. Well, our standard stuff is Apache based. We do have nginx in a few places, for special tasks.

I'm wondering if we should just setup something for you guys to play with and let you manage it (vs IT managing it). Thoughts?

Comment 4

[Mike Taylor [:bear]]

nginx is just what I think of when I talk web servers - apache is fine.  this will be a mozilla public facing service so it really belongs in IT, last thing I want to do is start getting put on the IT paging config ;)

but yea, a basic setup for now that we can hash out how this thing will look would be perfect because we will need to start testing with clients who don't have buildvpn access so something that is close to what you guys would prefer is definitely a step in the right direction.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Let's start by getting a vhost with some static pages out there.  I'm thinking something similar to htpt://pulse.mozilla.org, perhaps http://releng.mozilla.org.  For the most part, it will be a documentation site, so it doesn't need to be bulletproof.  As we start rolling out dynamic services on the site (buildapi, for example), we can consider putting those on a different vhost, hosting them directly on the same server, or proxying them via the same server.

Comment 6

[Mike Taylor [:bear]]

Yea, I was avoiding throwing down the deploy details until after we figured out what all would be involved - right now we just need a public facing firewalled server :)

Comment 7

[Corey Shields [:cshields]]

Shyam, what about carving out a small bit of the seamicro for this once it is in production?

Comment 8

[matthew zeier [:mrz]]

Pulling in infrasec for any code review issues.

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

So 'build.mozilla.org' is already named 'dm-wwwbuild01', and seems publicly accessible, although protected by LDAP.  This machine is running a few web apps already.

In the short term, is there any problem with focusing a bit more attention on this system as the go-to location for Releng services and status (e.g., for BuildAPI, Self-Serve, Clobberer (there already), and any other publicly-accessible tools we put together)?

Long-term, if the load on this system starts getting out of hand, we can of course start migrating services elsewhere.

If this is OK, then this bug is just a placeholder, and doesn't block bug 614629 (in fact, it depends on it..)

Comment 10

[Zandr Milewski [:zandr]]

(In reply to comment #9)

> In the short term, is there any problem with focusing a bit more attention on
> this system as the go-to location for Releng services and status (e.g., for
> BuildAPI, Self-Serve, Clobberer (there already), and any other
> publicly-accessible tools we put together)?

That works for me, especially if dm-wwwbuild01 is monitored. Not hard to spin up a dedicated blade or two down the road.

Comment 11

[Shyam Mani [:fox2mike]]

Yeah, this is a VM though. So we could scale it up first before throwing hardware at it (when the time comes).

Comment 12

[Corey Shields [:cshields]]

kicking this over to releng, feel free to punt it back to us when it is actionable by server operations.

Comment 13

[Dustin J. Mitchell [:dustin] (he/him)]

Load on this system hasn't been a problem aside from the stackwalker CGI, which was quickly disabledThere's work afoot to break its services out into virtualhosts, which should allow us to migrate high-load stuff out.