629482 - try clones failing all over the place again

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[[e-mail] (:bhearsum)]

Same symptoms as the start of https://bugzilla.mozilla.org/show_bug.cgi?id=629268. Looks like an Apache restart may have improved things last night?

Comment 1

[Aravind Gottipati [:aravind]]

The server has a load avg of 50.  Probably means there are a ton of try servers all cloning at the same time.  We have to find out a way to throttle the number of clones that run at the same time.  There is a hardware fix for this as well, but that will take some time.

In the meantime fixing 629268 might help.

Comment 2

[[e-mail] (:bhearsum)]

Attachment: lower lock to 20

Comment 3

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #1)
> The server has a load avg of 50.  Probably means there are a ton of try servers
> all cloning at the same time.  We have to find out a way to throttle the number
> of clones that run at the same time.  

We get spikes in load, on different branches, unpredictably, and hg.m.o needs to be able to handle this load.


> There is a hardware fix for this as well,
> but that will take some time.
Please clarify what is needed here, and I'll do my best to make sure you get it.




> In the meantime fixing 629268 might help.
Happy to reset try repo in bug#629268, and see if that helps. If yes, then we should do this as-routine-proceedure at start of every month to avoid rehitting these developer-visible-failures.

Meanwhile, would another apache restart help?

Comment 4

[[e-mail] (:bhearsum)]

Comment on attachment 507593 [details] [diff] [review]
lower lock to 20

Landed this ptach and reconfiged the try master

Comment 5

[Aravind Gottipati [:aravind]]

(In reply to comment #3)
> We get spikes in load, on different branches, unpredictably, and hg.m.o needs
> to be able to handle this load.

Handle spikes up to what load..  We can throw more hardware at the problem, and scale it to say double the current capacity.  And then you add more try slaves and then that isn't enough anymore.  

Maybe the fix is to clone it once per push and replicate that clone internally somehow (bittorrent..etc)?  I think this has to involve a software solution along with the hardware side of things..

> Please clarify what is needed here, and I'll do my best to make sure you get
> it.

We are discussing hardware options in IT and plan to rebuild/move the entire hg.m.o infrastructure to phx.  Those discussions are underway, not sure if there is anything for build and release to do there.

> Meanwhile, would another apache restart help?

Not really, Ben pushed out a change to limit the number of clones to 20.  That should help.

Comment 6

[Chris AtLee [:catlee]]

(In reply to comment #5)
> (In reply to comment #3)
> > We get spikes in load, on different branches, unpredictably, and hg.m.o needs
> > to be able to handle this load.
> 
> Handle spikes up to what load..  We can throw more hardware at the problem, and
> scale it to say double the current capacity.  And then you add more try slaves
> and then that isn't enough anymore.  
> 
> Maybe the fix is to clone it once per push and replicate that clone internally
> somehow (bittorrent..etc)?  I think this has to involve a software solution
> along with the hardware side of things..

This is already happening in bug 589885.

Comment 7

[[e-mail] (:bhearsum)]

I guess this bug is FIXED, since the immediate issue is dealt with.