550280 - Add a couple(?) more backend servers for hg/svn http side of things.

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Aravind Gottipati [:aravind]]

The current servers (dm-vcview01/02/03) are heavily utilized and need some help carrying the load from all the new build machines.  We should probably think of adding a couple more backend servers into the mix to spread the load a little.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

We've noticed in the last couple of days that hg.m.o is responding less promptly, and there were several nightly builds which failed two nights ago. Taken with bug 550256 could we bump this up from minor ? It's also possible that something is going wrong on the existing servers, perhaps from the recent changes there.

Comment 2

[Aravind Gottipati [:aravind]]

(In reply to comment #1)
> It's also possible that something is going wrong on the existing servers, perhaps from the recent
> changes there.

What changes are you referring to here?  AFAIK, there were no changes on the backend servers.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Perhaps I'm confusing the http-reading side with the ssh-pushing issues people had a couple of days ago.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

FWIW, bug 550256 was originally summarized "http://hg.mozilla.org is quite slow".

Comment 5

[Aravind Gottipati [:aravind]]

(In reply to comment #3)
> Perhaps I'm confusing the http-reading side with the ssh-pushing issues people
> had a couple of days ago.

Oh yeah, those are un-related.  The ssh stuff lives on different servers than the http side of things.  Will talk to mrz and see what we have on the hardware side to get this going.  Bumping it up to normal (ETA ~1 Week).

Comment 6

[Shyam Mani [:fox2mike]]

Aravind,

Do we want to try and convert this stuff to puppet too, while we're at this? Makes it far easier for us to manage?

Let me know if you need a hand with that part, I'll be glad to help.

Comment 7

[Aravind Gottipati [:aravind]]

I added one more server into the mix, if you notice anything off with hgweb, please comment here.

Comment 8

[Aravind Gottipati [:aravind]]

The server seems to be holding up well, and load pages on the others seem to be in check as well.