452784 - 'hg clone' and 'hg pull' timing out/getting "connection interrupted" on many build machines

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[[e-mail] (:bhearsum)]

Our nightly m-c/tracemonkey/actionmonkey builds mostly failed out with this last night. we're seeing the problem on hourly builds as well. On m-c, seamonkey, and the 3.1a2 linux release build I'm doing.

Could this be related to bug 450648?

It's holding the Firefox tree closed, so I'm filing as a blocker.

Comment 1

[Jeremy Orem [:oremj]]

Paged Aravind.

Comment 2

[Aravind Gottipati [:aravind]]

bumped up the max servers, increased timeouts and enabled python optimizations in wsgi.

Comment 3

[Aravind Gottipati [:aravind]]

seems like that has resolved the problem - at least for now.  I will keep an eye on the server side of things.  Please re-open if this continues to be a problem and I will revert things to the old servers.

Comment 4

[Dave Townsend [:mossop]]

The linux mozilla-central nightly build machine just hit a connection problem again and I'm still occasionally getting timeouts accessing hg's web interface.

http://tinderbox.mozilla.org/showlog.cgi?log=Firefox/1220029804.1220029863.321.gz

Comment 5

[Aravind Gottipati [:aravind]]

Will be back in the office in about a half hour and will look at this then.

Comment 6

[Aravind Gottipati [:aravind]]

Added the old servers back into rotation, so stuff is faster now.  This will mean, re-opening bug 450648, since I now have to find a couple of beefy servers that can handle this load.