461873 - Hg pull: intermittent "abort: HTTP Error 500: Internal Server Error"

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Serge Gautherie (:sgautherie)]

Bug 460054 comment 11:
{
From  Serge Gautherie   2008-10-15 15:05:14 PDT

I noticed "HTTP 500" errors [...] on boxes only very very recently.
}

This still happens: see
http://tinderbox.mozilla.org/showlog.cgi?log=Thunderbird/1225130656.1225130237.10742.gz
Linux comm-central check on 2008/10/27 11:04:16

Comment 1

[Aravind Gottipati [:aravind]]

How often is this happening?  This is the first I am hearing of this since the last change I made on the hgweb boxes.

Comment 2

[Serge Gautherie (:sgautherie)]

Other occurrences:

http://tinderbox.mozilla.org/showlog.cgi?log=Thunderbird/1225136158.1225135738.24126.gz
Linux comm-central build on 2008/10/27 12:35:58
http://tinderbox.mozilla.org/showlog.cgi?log=Thunderbird/1225136158.1225135731.24104.gz
MacOSX 10.4 comm-central build on 2008/10/27 12:35:58

(In reply to comment #1)
> How often is this happening?  This is the first I am hearing of this since the
> last change I made on the hgweb boxes.

+/- same for me, but I had been paying less attention recently. We'll see...

Comment 3

[Phil Ringnalda (:philor)]

From a very quick scan:

20081026 03:00ish, Linux and Windows (Mac seems to be different bustage)
20081025 09:00ish, Linux
20081024 12:50ish, Linux and Windows
20081022 23:40ish, Linux and Mac and Windows, build and check
20081022 19:17ish, Linux and Mac
20081020 21:00ish, Linux and Windows
20081020 20:30ish, Mac twice over
20081020 13:00ish, Linux and Mac and Windows (missed the Mac tester, though, only a 5 out of 6)

Comment 4

[Phil Ringnalda (:philor)]

20081028 09:26, Linux, Mac, and Windows check (Thunderbird, as were comment 3, which I see I didn't mention)

Comment 5

[Phil Ringnalda (:philor)]

20081028 12:45:32 Tbird Linux check

Comment 6

[Phil Ringnalda (:philor)]

20081029 01:51:36 Tbird Linux build

Comment 7

[Aravind Gottipati [:aravind]]

I don't think this is a server load issue type bug.  justdave says we had similar issues with CVS (before we went to lock free cvs pserver implementation).  I am not sure about the internal workings of mercurial, so I am not sure if we could be running into similar issues.  I am going to instrument the server to collect some load metrics overnight.  

Please comment here if you guys notice anymore hg pulls timing out.

Also, ccing djc for input on mercurial locking.  We run the hgwebs off a read-only mount of the same NFS share as the regular hg servers.  I doubt it, but would someone pushing stuff cause locking issues on the pulls?

Comment 8

[Philippe M. Chiasson (:gozer)]

Quite possibly hitting this known (and fixed) mercurial bug:

http://www.selenic.com/mercurial/bts/issue1320

Read the discussion around it, it's quite insightfull. Basically, using NFS is not always safe when clients are pulling during a push.

Comment 9

[Serge Gautherie (:sgautherie)]

http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey/1225359292.1225360367.13978.gz
Win2k3 comm-central build on 2008/10/30 02:34:52

Comment 10

[Aravind Gottipati [:aravind]]

(In reply to comment #9)
> http://tinderbox.mozilla.org/showlog.cgi?log=SeaMonkey/1225359292.1225360367.13978.gz
> Win2k3 comm-central build on 2008/10/30 02:34:52

I looked at the system logs from around 2:30 and it seems like there are a lot of hits on the server around that time.  However, the server seems to be doing fine with the load.. I can't rule the server side out completely, but I *think* that that may not be the problem.  Will continue looking at it.  Please continue posting information about more of these timeouts as you find them.  The bug that gozer linked to seems interesting.

Comment 11

[Aravind Gottipati [:aravind]]

(In reply to comment #10)
> I can't rule the server side out completely, but I *think*
> that that may not be the problem.  Will continue looking at it.

I just looked at the load average results for that time period as well and those seem well under control.  I am more sure now that this is probably not a performance related server side issue.

Comment 12

[Aravind Gottipati [:aravind]]

I have not heard anything else on this since my last comment.  At this point, I am fairly sure this is not a server setup issue.  Please re-open if needed.