370224 - Set up public HTTPS server for buildbot try service

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Robert Helmer [:rhelmer]]

We're working on a buildbot server that will allow trusted developers to upload patches and have them applied and build automatically. The patch is uploaded via a CGI, which notifies buildbot of the change.

We've got this set up now on test1-linux-buildbot.build.mozilla.org, but we want a production-level machine for this. It needs to have Buildbot and Apache at minimum.

This should have port 80 open to the public, but we should require LDAP authentication both to upload patches and to view the buildbot page. Buildbot runs it's own webserver on port 8010, so maybe Apache proxying can be used?

Comment 1

[J. Paul Reed [:preed]]

Rhelmer and I talked about this, and we can actually narrow down the requirements here:

What we need to do is deploy the CGI that Ben and Rhelmer have worked on that allows people to upload patches. We need some authentication data of some sort, but I don't know if we want to restrict to LDAP? Maybe we do, because we only want people with CVS access?

Anyway, we don't need buildbot on this machine, and we shouldn't need to punch holes in the build network firewall for this.

The CGI, from what I understand, needs to be able to drop files to a place where the webservers would be able to see it (so the remote side can pick them up).

I don't know if the cluster is the right place for this, or?

Also, we'll need to get this code checked in, so we can have you pull it from CVS or whatever. I'll rhelmer fill in those details and/or other requirements I may have missed. We don't need our own server for this, though, so if we can set this up in the existing "web projects infrastructure," that'd be great!

Comment 2

[Robert Helmer [:rhelmer]]

Making the summary more useful. Chatting with Aravind I think we've got the requirements nailed down now:

* IT supported VM
** Apache (SSL only), LDAP auth required for /
** some kind of file upload service (CGI/PHP/etc.)
** allows SSH logins from buildbot master

I've asked morgamic if the webtools team has some code we can reuse for the file upload and form submission.

Comment 3

[J. Paul Reed [:preed]]

(In reply to comment #2)

> ** allows SSH logins from buildbot master

Wouldn't it be easier to have the buildbot master get these patches via http(s)? If so, SSH logins from the buildbot master aren't a requirement.

Also, I asked rhelmer about this; I'm assuming the ssl requirement is because of the use of LDAP auth, so we wouldn't want passwords flying around in the clear?

For the first stab at this, if it's easier to not do ssl, we should do that and use htaccess or something similarly easy to setup.

Comment 4

[J. Paul Reed [:preed]]

(In reply to comment #3)

> For the first stab at this, if it's easier to not do ssl, we should do that and
> use htaccess or something similarly easy to setup.

(And to be clear, I meant "use htaccess and a shared password [or something equally hokey] for now.")

If it's just as easy to set up LDAP, though, then that's obviously better. :-)

Comment 5

[Michael Morgan [:morgamic]]

So far the only example of saving to disk is in addons v1:
http://lxr.mozilla.org/update1.0/source/developers/additem.php#58

That's a fairly common example.  If you're looking at PHP, their usage example is pretty solid:
http://php.osuosl.org/manual/en/features.file-upload.php

One thing to watch for would be the filename, to make sure path traversal is not possible.  This was handled in v1 via the check_filename() function:
http://lxr.mozilla.org/update1.0/source/core/inc_global.php#340

Aside from that I'd just check the filename against whatever reqs you'd have for it -- all alphanumeric, for example, could be something you'd want to check for.

The .xpi stuff is for AMO so it's not really necessary, but you get the idea.

Clouser might have other ideas, so he can add them if he does.  :)

Comment 6

[Michael Morgan [:morgamic]]

Also - note that just calling die() is not the way to handle file errors... the v1 code is old, but it's a working example.  <3

Comment 7

[Aravind Gottipati [:aravind]]

Okay, dm-buildbot01 is now available.  It is currently available only on https://buildbot.m.o.  We can open up http and have it redirect to https if you like.

Everything under https://buildbot.m.o is password protected, you will need an LDAP account to get in.

You guys (preed and rhelmer) can login to the box as well (with those login names), it uses your LDAP keys.

Install any cgi/php code under /var/www/html/buildbot/

To verify this, I have a CGI shell script and a php script at 
https://buildbot.mozilla.org/env.cgi
https://buildbot.mozilla.org/env.php

Let me know if you guys need anything else.

Comment 8

[J. Paul Reed [:preed]]

We just talked to Aravind, but since we have an entire web host for things, I'd prefer this host to be renamed to dm-wwwbuild01, with a CNAME pointing to build.mozilla.org, so people can put build.mozilla.org into their browser and get a useful webpage.

Also, Chris Cooper and Ben Hearsum will need access to this machine via LDAP.

Thanks for working on this Aravind; we might need your help to get the sub permissions/apache setup correct.

Comment 9

[Justin Fitzhugh]

Lets not do this as I assume this will become a production app, at which point we'll be admin'ing it and you guys won't have the box.  If you need a build.mozilla.org we can do that for sure, just don't think this is the right place.  Agree/disagree?

Comment 10

[J. Paul Reed [:preed]]

(In reply to comment #9)
> Lets not do this as I assume this will become a production app, at which point
> we'll be admin'ing it and you guys won't have the box.  If you need a
> build.mozilla.org we can do that for sure, just don't think this is the right
> place.  Agree/disagree?

Disagree.

The reason I suggested that we use this VM for other things is that it's a real waste of a VM to have a VM whose only purpose in life is to have patches uploaded to it, and serve them out via http.

I understand the reasons why this route was taken (isolation, mostly), but there are a number of things that we could do (publish logs, etc.) that would be helpful if we had access to a web server w/ supporting infrastructure that we could add things to and play around with, without a huge amount of lead time.

I may not be explaining this well in a bug, but this goes back to that age-old question about where the support lines start and stop, so maybe we should talk about it in person.

But the upshot is that it would be very useful to have a place to run miscellaneous CGIs and other web-related stuff that can't be in the build network, both in a development context and a production context, and it seems like this VM would fill that roll nicely; it really is a complete waste to have an entire 10 gig/1 gig of memory VM acting as a glorified scratch space.

Comment 11

[Justin Fitzhugh]

OK - makes sense.  I guess if we need to we can always move buildbot, but sounds like that won't be needed.

Comment 12

[Robert Helmer [:rhelmer]]

Logged in and started setting stuff up, but ran into a couple things I didn't think about before. Can you please make the following Apache config changes:

* enable MultiViews
* allow any machine in *.build.mozilla.org network to view /var/www/html/buildbot/patches/ without authentication
* enable Indexes for /var/www/html/buildbot/patches/

Comment 13

[J. Paul Reed [:preed]]

(In reply to comment #12)
> Logged in and started setting stuff up, but ran into a couple things I didn't
> think about before. Can you please make the following Apache config changes:

<greedy>Question for IT: would it be possible to get these configs checked in somewhere, and auto-pulled, so we can all make changes that are versioned/audited?

...

</greedy>

Comment 14

[Aravind Gottipati [:aravind]]

Rob:  I will make those changes and let you know when its ready.

Preed: not atm, we do plan on going to a versioned, config mgmt system.  But thats some time off.  AFAIK, the only config files that are versioned right now are the ones in the prod clusters.

Comment 15

[J. Paul Reed [:preed]]

(In reply to comment #14)

> Preed: not atm, we do plan on going to a versioned, config mgmt system.  But
> thats some time off.  AFAIK, the only config files that are versioned right now
> are the ones in the prod clusters.

Would it be possible to set something up in the short term? I don't mean a full config mgmt system like puppet or cfengine (which I hear you [guys] are looking into); I mean something much lighter weight for now.

It'd be really helpful, I think.

Comment 16

[Aravind Gottipati [:aravind]]

MultiViews enabled sitewide,
Indexes enabled for the patches directory,

hosts from *.build.mozilla.org can bypass the password auth if they request https://build.mozilla.org/patches/.

Please note that since the server name has changed, I moved the doc root to /var/www/html/build.  I moved the existing files over and deleted the old docroot.

Comment 17

[Robert Helmer [:rhelmer]]

This all looks great but one more thing :) Please change internal DNS from dm-buildbot01 to dm-wwwbuild01

Comment 18

[Aravind Gottipati [:aravind]]

fixed.