Open Bug 537323 Opened 15 years ago Updated 12 years ago

New machines for a third SeaMonkey tree/branch

Categories

(mozilla.org :: Community Giving, task)

Product:
mozilla.org
Component:
Community Giving
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

People

(Reporter: kairo, Assigned: sethb)

References

Details

I know, not all the infrastructure the SeaMonkey project has requested (originally bug 464325, now bug 526208 still open), but it's this time again.

In order to support an additional tree/branch for the upcoming SeaMonkey 2.1 series (which will be based on 1.9.2 or 1.9.3, start development ASAP for a release between April and July), we will need additional machines in our pools to run builds and tests from that code.

My estimate for what we need for that are 3 machines or VMs per platform (linux, win32, mac).

The good new of some kind is that bug 528817 seems to have fixed the last issues we had with Parallels, so that is a valid option again for the extension again. The not completely good news with that is that bug 493321 is WONTFIX, i.e. we cannot have more VMs than CPUs on a Parallels host. That means if we want to have those new 9 VMs on a Parallels Xserve, we need 9 or more CPUs in it (we also need to consider that we possibly might want to cover bug 526208 with VMs instead of minis after all, which could play into this).

Of course, we also can do things with minis and ESX VMs, whatever you guys feel is better suited.

Seth, I guess this also needs a word from community giving.
Assignee: server-ops → mrz
Assignee: mrz → sethb
Component: Server Operations → Community Giving
QA Contact: mrz → community-giving
Added MRZ.

MRZ, can we discuss (in this bug) the VM vs CPU set up KaiRo is mentioning above? 

With the existing infrastructure, Sea Monkey has 1.9.1 supported fully, and trunk mostly (fully with Mac Minis that are on order).  The team will need to support an upcoming branch of 1.9.2 or 1.9.3.  KaiRo, when will you know which branch?

Lastly, I believe you spelled this out last time, but can you articulate in this bug specifically what the three machines per platform will be used for?
(In reply to comment #1)
> Added MRZ.
> 
> MRZ, can we discuss (in this bug) the VM vs CPU set up KaiRo is mentioning
> above? 

I also just heard that the new IX machines might even be cheaper than VMs, so that might also be an option. Whatever IT thinks is best and allows us to run the machines we need is fine with me.

> With the existing infrastructure, Sea Monkey has 1.9.1 supported fully, and
> trunk mostly (fully with Mac Minis that are on order).  The team will need to
> support an upcoming branch of 1.9.2 or 1.9.3.  KaiRo, when will you know which
> branch?

Most likely when we know a roadmap for 1.9.3 - it's basically, if there is a 1.9.3 in Summer, then we are most likely on that. If it's coming in winter, we probably need to go with 1.9.2 instead. If it's in between, we'll need to look closer at what we want to do in that release.

> Lastly, I believe you spelled this out last time, but can you articulate in
> this bug specifically what the three machines per platform will be used for?

We nowadays have machine pools per platform with all our available machines taking jobs as they are available.
But as we are doing 1) dep builds (ideally optimized and debug), 2) nightly and L10n builds (including updates), and 3) various automated test suites on all platforms, the experience tells us that we need at least 3 machines for every platform to cover the load coming up because of all those tasks.
Not clear on what I need to buy.  How many and what OS?
(In reply to comment #3)
> Not clear on what I need to buy.  How many and what OS?

I thought I said it clearly enough in comment #0 that what we need is 3 build machines per platform on Linux, Mac (Leopard), and Windows.

The open question is what solutions are best in terms of price, maintenance, etc. to use for that - more of Parallels, ESX VMs and minis or the new IX machines I just heard of this week.
And mrz, I think it's you who knows most about that, and from what I saw, that's what Seth also wanted your opinion on.
RelEng would know more than me about how well each platform works.  I'll only comment that the 1u iX systems have yet to be proved.
Right, the 1u iX systems haven't been too thoroughly tested yet, but from what RelEng people tell me, they look quite promising right now and the preliminary testing that was done showed no problems at all, apparently (on the contrary, it showed they were *very* fast).

Also, VMs have unfortunately been proven to not handle themselves too well where a lot of disk, CPU *and* I/O is needed, and our build setups have just that.


So, I think what we want for this bug is actually 6 1u iX machines and 3 minis - would that be possible?
(In reply to comment #6)
> So, I think what we want for this bug is actually 6 1u iX machines and 3 minis
> - would that be possible?

Seth, any word on that?
I need to chat with MRZ and others on the request.  Thanks for pinging.  I'll get back to you soon on this.
(In reply to comment #8)
> I need to chat with MRZ and others on the request.  Thanks for pinging.  I'll
> get back to you soon on this.

Did you come around to that yet? This comment has been more than a month ago...
(In reply to comment #9)
> (In reply to comment #8)
> > I need to chat with MRZ and others on the request.  Thanks for pinging.  I'll
> > get back to you soon on this.
> 
> Did you come around to that yet? This comment has been more than a month ago...

No.  I've been traveling internationally (and domestically) since Feb 15.  I will ask him this week.
Seth, ping?

Another months has passed, can we get some tracking here?

Meanwhile, I hear from releng that all their iX machines have been working nicely in production for a while, BTW.
Ping?

This bug has been open for 5 months now, and though we have ignored 1.9.2 in our development to concentrate on a 1.9.3-based release, that new branch is about to be created now and we are still lacking the machine power for such an additional release branch. It would really be helpful to get this going so we can actually deliver on the release we are working on.

Would that be possible, please?
What I have a hard time understanding is the long-term hardware strategy and needs for SeaMonkey.  Reflecting on this bug, there seemed to be a sense of immediacy when the bug was first filed.  

Without excusing the tardiness of completing this bug, I am interested to find out what is immediate and what is the long-term hardware need for Sea Monkey.  

Can you put together a long-term (one year) hardware strategy for us?  That will give me a better sense of ongoing needs, what hardware we have for you, what we might be able to recycle, what we need in the future.  We can also try to invest in machines or a setup that allows you to manage an active release and then actively develop on a new release.  Make sense?
(In reply to comment #13)
> Reflecting on this bug, there seemed to be a sense of
> immediacy when the bug was first filed.

Due to punting on the 1.9.2 branch, this immediacy has shifted to now, when 1.9.3 is being branched.

> Can you put together a long-term (one year) hardware strategy for us? 

I should be able to do that, but what we need, what we want, and what we'd like are three different things. The real needs don't go much beyond the machines right in this bug, the wants and would-likes go beyond it for sure.
I'll try to bring up something in that regard, but I think I should do that outside of this bug.
(In reply to comment #14)
> I should be able to do that, but what we need, what we want, and what we'd like
> are three different things. The real needs don't go much beyond the machines
> right in this bug, the wants and would-likes go beyond it for sure.
> I'll try to bring up something in that regard, but I think I should do that
> outside of this bug.

That should be good.  I think I would benefit from seeing the plan so I have something to reference going forward and can also be something that helps us budget needs over the course of a year through many releases.  You can email me with the plan.
Blocks: 572395
KaiRo/Callek, I wonder what the status of this is wrt. the rapid release train. Should this bug be closed or transformed?
The rapid release train doesn't change much there, it comes out with more or less the same requirement, and bug 572395 is where we actually deal with getting the new machines in place - they actually have been delivered but not installed yet.
You need to log in before you can comment on or make changes to this bug.