Closed Bug 571551 Opened 14 years ago Closed 13 years ago

Request for a Mozilla hosted VM for TinderboxPushlog

Categories

(mozilla.org Graveyard :: Server Operations, task)

Product:
mozilla.org Graveyard
Component:
Server Operations
Platform:
All
Other
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: ehsan.akhgari, Assigned: fox2mike)

References

Details

We're planning server side data caching for tbpl, and we need to get off Markus' server for that to happen.

Shaver promised me that we can get a VM setup within a week in order to host tbpl on.

I don't think we need anything beefy at this point.  Disk space will help us cache more stuff, so I'd say let's go for 50g of free disk space after the OS install, etc.  RAM is an important factor, because the processing that we're going to do requires handling large amounts of data, and it would suck if we get paged off to disk.  Could we get 4 gigs of RAM (and maybe even more?)

I'll also let Markus and Arpad weigh in on the hardware requirements.

What will matter most is fast (and I mean, *fast*) access to our current Tinderbox machines.  Would that be possible?
ehsan, johnath, mstange, dietrich should have LDAPized user accounts, and sudo sufficient to config web server and suchlike.

All hail TBPL!
So should Arpad, I'd say.  He's been doing a lot of great hacking on TBPL!
No, Arpad is not allowed.












I'm kidding, he's awesome.  Good catch.
Thanks for bringing this topic up again, nice to see some progress there :)

The server side load is negligible. I was running that script on a 128M VM which I just recently upgraded to a 512M VM. I upped the memory limit a bit by I still got OOM in php, and sometimes SQL exceptions. Parsing smaller chunks of tb&pl should help a lot with that.

I will continue the rest of the discussion on bug 553549, thanks.
Arpad, on your test server as far as I can see you only pull data for two branches.  I'm thinking about having a system which handles all the branches that we have (both now and in the future.)  Also, Markus told me that you don't parse the data off of a cronjob.  If we choose to do that, the load is going to be a lot different.
Ehsan - am I right in assuming that this VM doesn't need any privileged ("inside the firewall") access to other mozilla systems? I know IT tends to want infrasec review for new VM usages, and that decision will likely be central.
And even otherwise, we might decide to keep it within our network, finances play a role as well :) Anything that resides within the network and needs to be opened up to the public will go through infrasec before being made public. That's the policy side of things.
As far as I can tell, as long as that server can serve web pages on port 80, can connect to external websites on port 80, and can give us an ssh shell on any port, we should be fine.

Something which would really help us here is blazing fast network access to tinderbox.mozilla.org.  Is that something which would affect this decision?
Is this something that needs to be generally accessible by outside users or can it be VPN-accessible only?
It needs to be accessible to everyone, using a publicly accessible DNS name.

(I suggest tbpl.mozilla.org.)
@clyon - what sort of review do you need to do here?
Assignee: server-ops → shyam
The VM is up, dm-tbpl01.mozilla.org, accessible only from MPT for now. I still need to work on the LDAP auth bits, so expect this by noon PDT US.

Once your app is setup and clears infrasec review, we can open up an external IP and setup tbpl.mozilla.org
clyon: per our conversation earlier, this should go up in a no-priv environment; making sure that total pwnership of the machine via tbpl doesn't provide a better place to stage attacks on other sensitive infra is probably the goal of any infrasec review.
(In reply to comment #12)
> The VM is up, dm-tbpl01.mozilla.org, accessible only from MPT for now. I still
> need to work on the LDAP auth bits, so expect this by noon PDT US.
> 
> Once your app is setup and clears infrasec review, we can open up an external
> IP and setup tbpl.mozilla.org

I hope you don't mind my ignorance, but how can I connect to this machine before it's available at tbpl.mozilla.org?

(In reply to comment #13)
> clyon: per our conversation earlier, this should go up in a no-priv
> environment; making sure that total pwnership of the machine via tbpl doesn't
> provide a better place to stage attacks on other sensitive infra is probably
> the goal of any infrasec review.

To second this, tbpl does not require anything more than comment 8, so by all means lock it down mercilessly!
(In reply to comment #13)
> clyon: per our conversation earlier, this should go up in a no-priv
> environment; making sure that total pwnership of the machine via tbpl doesn't
> provide a better place to stage attacks on other sensitive infra is probably
> the goal of any infrasec review.

Is there some webcode that we can review? I take it this is the prototype that we want to put up there on this server?

http://tbpl.swatinem.de/
The code is at http://hg.swatinem.de/tbpl-server/ but is hopelessly outdated and needs a rewrite.
A few questions to help get the code review started:

0. A quick intro to what this app does.

1. I'll be reviewing the code downloaded here(http://hg.swatinem.de/tbpl-server/archive/tip.tar.gz) Please confirm this is correct.

2. Is there a stage server running that I can also test against? If so, please let me know what machine the web server is running on.

3. Where would you like the bugs filed in bugzilla?  I need to know the product, component and if anyone specific should be copied on the bugs.

4. Please describe if this app will be connecting to any internal or external services or if it is able to integrate with the OS.

5. I can begin working on this today. When is this needed to be completed by?
(In reply to comment #17)
> 0. A quick intro to what this app does.

It downloads data from pushlog and tinderbox, parses it, saves it inside a mysql db and provides a web interface to query that data.

> 1. I'll be reviewing the code downloaded
> here(http://hg.swatinem.de/tbpl-server/archive/tip.tar.gz) Please confirm this
> is correct.

Thats the current code, yes

> 2. Is there a stage server running that I can also test against? If so, please
> let me know what machine the web server is running on.

It runs on tbpl.swatinem.de which is my private server.

> 3. Where would you like the bugs filed in bugzilla?  I need to know the
> product, component and if anyone specific should be copied on the bugs.

Webtools -> Tinderboxpushlog, you would want to cc mstange, ehsan and me.

> 4. Please describe if this app will be connecting to any internal or external
> services or if it is able to integrate with the OS.

It needs access to pushlog and tinderbox. It currently accesses those externally. In the future, it may access the build db internally, however we have no plans/code for that yet.

> 5. I can begin working on this today. When is this needed to be completed by?

Take your time. I would rather want to clean up and refactor the code before it is deployed.
Thanks for the info. I just want to clarify one item. Is it ok for me to perform tests against your private server? tbpl.swatinem.de

If so, is it possible to have ssh access? I have a few tests that will verify that files cannot be written outside of the intended locations.
(In reply to comment #14)
> I hope you don't mind my ignorance, but how can I connect to this machine
> before it's available at tbpl.mozilla.org?

Not at all.

Like I mentioned, the LDAP bits need to be setup. 3.6.4 related socorro work ate up all of my time today, so expect this by the end of the week. 

In the meanwhile, I can just put in your key and allow you to login as root till I complete the LDAP setup, if that's something that will help you setup the app faster. Let me know if you'd like this to be done.

You'll need access to mpt-vpn and will ssh into dm-tbpl01.mozilla.org
(In reply to comment #20)
> (In reply to comment #14)
> > I hope you don't mind my ignorance, but how can I connect to this machine
> > before it's available at tbpl.mozilla.org?
> 
> Not at all.
> 
> Like I mentioned, the LDAP bits need to be setup. 3.6.4 related socorro work
> ate up all of my time today, so expect this by the end of the week. 
> 
> In the meanwhile, I can just put in your key and allow you to login as root
> till I complete the LDAP setup, if that's something that will help you setup
> the app faster. Let me know if you'd like this to be done.
> 
> You'll need access to mpt-vpn and will ssh into dm-tbpl01.mozilla.org

Well, I don't think I'll be able to do a lot of useful work on the server before the end of this week anyways, but if putting up my key on the server is easy, then it would be great!

Out of curiosity, what distro have you set up on the VM?  /me silently hopes that it's something Debian based!
Gavin and Reed have requested this bug be made public.  Is this bug correctly
categorized? Does it need to be an infrastructure-related bug?
(In reply to comment #22)
> Gavin and Reed have requested this bug be made public.  Is this bug correctly
> categorized? Does it need to be an infrastructure-related bug?

If we are not worried about discussing our infrastructure related stuff here, there's no reason why this should not be public.
(In reply to comment #21)
> Well, I don't think I'll be able to do a lot of useful work on the server
> before the end of this week anyways, but if putting up my key on the server is
> easy, then it would be great!

In which case, I'll do that in case I can't setup by the end of the week.
 
> Out of curiosity, what distro have you set up on the VM?  /me silently hopes
> that it's something Debian based!

RHEL 5.5

All our production machines (barring very few) runs on RHEL. If there is something specific you're looking for, you need to let us know ASAP...and we'll see if it's practically doable. Else it's RHEL.
(In reply to comment #22)
> Gavin and Reed have requested this bug be made public.  Is this bug correctly
> categorized? Does it need to be an infrastructure-related bug?

Don't see why not. I think it was filed initially to server-ops with the infra bit set. Removing.
Group: infra
(In reply to comment #24)
> > Out of curiosity, what distro have you set up on the VM?  /me silently hopes
> > that it's something Debian based!
> 
> RHEL 5.5
> 
> All our production machines (barring very few) runs on RHEL. If there is
> something specific you're looking for, you need to let us know ASAP...and we'll
> see if it's practically doable. Else it's RHEL.

Well, I guess I can love with the rpm suckiness, I don't want to cause this machine have a different setup to the rest of our infrastructure.  :-)
heh, s/love/live/ obviously!
LDAP based ssh access is now up and running. So for example, you'd login as :

eakhgari@dm-tbpl01.mozilla.org with your ssh keys and it should work fine.

Johnath, Ehsan, Dietrich, Marcus and Arpad have access + full sudo access to the VM.
Anything else you folks need here?
Should we keep this bug open to move this box to a publicly accessible location, or should we file a new one when we're ready?
Depends on what you need done. This bug is public. I'm fine with leaving it open, while you setup things, file for a security review (and have that block this bug) and then we can take this "live" and close it off.

I'm unsure about the plan of action, hence the question in comment #29. Let me know and we'll proceed accordingly :)
(In reply to comment #31)
> Depends on what you need done. This bug is public. I'm fine with leaving it
> open, while you setup things, file for a security review (and have that block
> this bug) and then we can take this "live" and close it off.

The security review has already been done (bug 573823) and all of the discovered issues have been resolved.

> I'm unsure about the plan of action, hence the question in comment #29. Let me
> know and we'll proceed accordingly :)

Well, the next step is for one of us to set up services such as apache, php and mysql on the server, deploy the code to it and test it to make sure that things work correctly.  I've been meaning to do that for a long time now, but I haven't got to it yet due to vacation and other work.  Once that step is complete, the server should be ready for public deployment.
(In reply to comment #32)
> The security review has already been done (bug 573823) and all of the
> discovered issues have been resolved.

Cool. Once this is setup, I'd like mcoates to run through this (cursory check) before we take it live.

> Well, the next step is for one of us to set up services such as apache, php and
> mysql on the server, deploy the code to it and test it to make sure that things
> work correctly.  I've been meaning to do that for a long time now, but I
> haven't got to it yet due to vacation and other work.  Once that step is
> complete, the server should be ready for public deployment.

Alright. Let me know when it gets there. I'll leave this open for tracking.
Whiteboard: Waiting on Setup
Is dm-tbpl01.mozilla.org still alive? I can't connect to it through MPT.
Very much so, what's the error you're getting? I just logged in fine.
When I open the URL in Firefox or ping it, OpenDNS answers instead. Nick Thomas couldn't connect to it either when I asked him on IRC just now:

nthomas	[#developers] mstange_: I can't get to that either using MPT
nthomas	[#developers] mstange_: I tried to connect with ssh and couldn't open a connection
Never mind, it works on my other machine. I guess I need to find out why I'm using OpenDNS.
Could be your ISP. Let me know if there are other issues.
Reopen when there's IT action.
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → INCOMPLETE
It would be good to get the vm and subdomain, even when we have no “real” server component yet.

One reason is for other core tbpl developers to be able to deploy much needed fixes without Markus’ involvement.

What issues are blocking this?
I think we just need to get it set up.

(In reply to comment #28)
> Johnath, Ehsan, Dietrich, Marcus and Arpad have access + full sudo access to
> the VM.

Do we need an extra password for sudo access? My normal password doesn't do it, or I'm testing with the wrong password.
Is this the same as bug 600404 ?
(In reply to comment #41)
> I think we just need to get it set up.

Exactly.

> Do we need an extra password for sudo access? My normal password doesn't do it,
> or I'm testing with the wrong password.

Seems like puppet overwrote my changes, fixing.
sudo access should be back now.
Blocks: 600607
I've set it up. Installed are now:
Apache, PHP 5.3.3, Python 2.6, MongoDB and the mongo drivers for PHP and Python.
DocumentRoot is /var/www/tinderboxpushlog. I've pulled tinderboxpushlog tip and added the correct tbplbot password file.

Deploying tbpl is now just a matter of

ssh <user>@dm-tbpl01.mozilla.org
cd /var/www/tinderboxpushlog/
sudo hg pull -u

Everything seems to work so far, so we're probably ready to give the machine a proper subdomain.
Reopening for the subdomain part: please set up tbpl.mozilla.org.
(Let me know if I should file a new bug instead.)
Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
http://tbpl.mozilla.org/ is live.
Status: REOPENED → RESOLVED
Closed: 13 years ago → 13 years ago
Resolution: --- → FIXED
Whiteboard: Waiting on Setup
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.