555449 - Get a clobberer server running for SeaMonkey

Status: RESOLVED FIXED

(SeaMonkey :: Release Engineering, defect)

Description

[Robert Kaiser]

SeaMonkey's build and release machinery has grown and logging into 15 slaves and manually clobbering things just gets to be too much work, so I think it's time to get a real clobberer to use.

We need to be able to either share some installation set up elsewhere or set up our own instance, whatever is more reasonable. We don't have more hardware to use, though.

Ben, can you point me to what I need there - or tell me who else can help me to get that?

Comment 1

[Justin Wood (:Callek)]

I wonder if adding our boxes to the MoMo clobberer is possible; from my _very brief_ look at all this stuff it might be.

If not I wonder what on MoCo/MoFo side can assist us here.

Comment 2

[Aki Sasaki (not active)]

I'm not that familiar, but the source appears to be here:

  http://hg.mozilla.org/build/tools/file/default/clobberer

using a sqlite backend.

We use the base_clobber_url and default_clobber_time settings and the MozillaClobberer step in buildbotcustom/steps/misc.py .

Does that help?

Comment 3

[Robert Kaiser]

Thanks, the URL to the source does help, but I don't have any dedicated machine to set this up, so I wonder where to do it and how to set it up.

I know the buildbot side, that's no problem, the server side is the problem, esp. right now how and where to set it up.

Comment 4

[Serge Gautherie (:sgautherie)]

(In reply to comment #1)
> I wonder if adding our boxes to the MoMo clobberer is possible

gozer, ping?

(In reply to comment #3)
> the server side is the problem, esp. right now how and where to set it up.

At first glance, it seems "any" php-enabled web server could host index.php.
Is that what you're missing?
Could one be installed on SeaMonkey "buildbot master" or the like?

Comment 5

[Robert Kaiser]

(In reply to comment #4)
> At first glance, it seems "any" php-enabled web server could host index.php.
> Is that what you're missing?

One that is hosted at a good place like near to the master it needs to talk to, yes. Not having some machines is a common theme in SeaMonkey and will not be solved fast. And apparently I need to solve some other communication problems before even starting to talk about possible solutions I have in my head. Sorry, expect nothing to happen there. And no, I won't settle for any machine not hosted by Mozilla, thanks.

> Could one be installed on SeaMonkey "buildbot master" or the like?

If we want even more slaves to time out because of the master being overloaded (and that's the cause we're currently running into), I guess this would be a good recipe. Unfortunately, I don't know if any of the prerequisites, e.g. a web server or PHP, is installed on the master, but I know that machine is running into pretty heavy load at times and causing it to drop connections to slaves, I currently don't dare to screw around with it much.
Let's see how things look once we upgrade everything we have to buildbot 0.8, which I'm planning to do as soon as Mozilla RelEng calls their soon-to-happen upgrade a success. And yes, it will likely waste at least one workday of mine.

Sorry, I'm about as unhappy as you are with the situation, but if I knew of a good way to solve this, believe me, I'd already have done some work on it. :(

Comment 6

[Philippe M. Chiasson (:gozer)]

(In reply to comment #4)
> (In reply to comment #1)
> > I wonder if adding our boxes to the MoMo clobberer is possible
> 
> gozer, ping?

Well, it's a possibility, but not exactly as simple as it sounds. Currently, the clobbereer code is somewhat simple and is buried into our build network at the moment. Opening it up to external builders would be possible, but it presents a few challenges. 

> (In reply to comment #3)
> > the server side is the problem, esp. right now how and where to set it up.
> 
> At first glance, it seems "any" php-enabled web server could host index.php.
> Is that what you're missing?
> Could one be installed on SeaMonkey "buildbot master" or the like?

Yeah, this would be a 10 minute job. That's what it took me to get it running on our side, just simple php with a sqlite backend, really simple.

Comment 7

[Robert Kaiser]

(In reply to comment #6)
> Yeah, this would be a 10 minute job. That's what it took me to get it running
> on our side, just simple php with a sqlite backend, really simple.

That sounds good. My largest concern right now is that I don't have any free machine around, the only one where it would possibly fit being the buildmaster, which is known to be somewhat overloaded at times right now up to losing connections to slaves (even those that run on the same Parallels host), though. From that, you can see how I'm reluctant to put anything on there.

Gozer, is your clobberer running on a dedicated VM or does it share the host with something else? How much server load do you estimate it adds (qualitatively - nothing, almost nothing, some, a lot)?

Comment 8

[Serge Gautherie (:sgautherie)]

(In reply to comment #7)
> Gozer, is your clobberer running on a dedicated VM or does it share the host
> with something else? How much server load do you estimate it adds
> (qualitatively - nothing, almost nothing, some, a lot)?

gozer, ping?

Comment 9

[Robert Kaiser]

Serge, please leave any bugs with requests for machines for SeaMonkey alone until we have the meeting in Vienna. I'll have a few talks next week that either make our requests for machines, including this one, succeed or ultimately fail.

Comment 10

[Philippe M. Chiasson (:gozer)]

(In reply to comment #7)
> (In reply to comment #6)
> > Yeah, this would be a 10 minute job. That's what it took me to get it running
> > on our side, just simple php with a sqlite backend, really simple.
> [...]
> Gozer, is your clobberer running on a dedicated VM or does it share the host
> with something else?

Since I already ran Apache on the buildbot master (for unrelated stuff), it was just convenient to drop the clobberer (it's just a little php script) there as well.

> How much server load do you estimate it adds
> (qualitatively - nothing, almost nothing, some, a lot)?

Almost nothing, it's pretty much 1 SELECT or 1 INSERT statement per build.

I've had issues with it and sqlite for high-availability, but I am moving it to MySQL.

I'd say just run one on your buildbot master, it'll take 10 minutes to have it running.

Comment 11

[Robert Kaiser]

Thanks, gozer, sounds encouraging, I feared it would cause too much load to put it on the master, but given your report, I might try that! :)

Comment 12

[Robert Kaiser]

The clobberer should be set up at the base address of the SeaMonkey buildmaster, I will only mark this fixed once I've seen that it actually works, though.

Comment 13

[Robert Kaiser]

I requested a clobber of comm-central-trunk builds to test if clobberer works, if it does, we can mark this fixed.

Comment 14

[Robert Kaiser]

Seeing the following output:

comm-central-trunk-macosx:Our last clobber date:  2010-11-06 16:21:24
comm-central-trunk-macosx:Server clobber date:    2010-11-07 07:23:16
comm-central-trunk-macosx:Server is forcing a clobber
comm-central-trunk-macosx:Clobbering...

and similar things.


Looks like it works. :)

Comment 15

[Justin Wood (:Callek)]

Ok, for reference:

Current install is at seabld:~/tools/clobberer  (where tools is the build tools repo)

DB is at seabld:~/tools/clobberer/db/clobberer.db

we are set up with /etc/http.d/conf/httpd.conf (points at the clobberer directory above)

I am also guessing (based on what I see) that:

chkconfig --add httpd
/etc/init.d/httpd start

was run.

I'll let kaiRo tell me if that sounds right.

Comment 16

[Robert Kaiser]

Thanks for digging that up, sounds just like what I did back then!