606809 - Bugzilla showing 500 Internal Server Errors

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Justin Dow [:jabba]]

I noticed it around 6:30am, any bugzilla page gave me a 500 ISE. I saw this in #bmo scrollback:

<Milos> [04:12:31] anyone has problems with bugzila/
<Milos> [04:12:31] ?
<Milos> [04:13:19] from time to time it displays 500 internal server error(like 10 times in 30 mins), and after every link I need to log-in again(yes, cookies are enabled)

Comment 1

[Shyam Mani [:fox2mike]]

Saw this in the logs :

[Sun Oct 24 06:38:38 2010] [error] Software caused connection abort at /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/Template.pm line 163.\n
[Sun Oct 24 06:53:17 2010] [error] Software caused connection abort at /usr/lib64/perl5/vendor_perl/5.8.8/x86_64-linux-thread-multi/Template.pm line 163.\n

Dave will probably poke at this soonish.

Comment 2

[Shyam Mani [:fox2mike]]

Also,

access_log

 - - [24/Oct/2010:07:12:51 -0700] "POST /jsonrpc.cgi HTTP/1.1" 500 51480 "https://bugzilla.mozilla.org/show_bug.cgi?id=606719" "Mozilla/5.0 (X11; Linux x86_64; rv:2.0b8pre) Gecko/20101020 Firefox-4.0/4.0b8pre" 0 bugzilla.mozilla.org

error_log

[Sun Oct 24 07:12:52 2010] [error] Software caused connection abort at /data/www/bugzilla.mozilla.org/Bugzilla/WebService/Server/JSONRPC.pm line 71.\n

Comment 3

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

This is happening pretty frequently for the few of us that are in #developers this morning.

Comment 4

[Dave Miller [:justdave]]

I pushed out a config change to Apache to force the X-Backend-Server header to be included in error pages as well, to help isolate the broken server.  However, since pushing that out, I haven't been able to reproduce this anymore.  I suspect the apache restarts fixed it?

Comment 5

[Dave Miller [:justdave]]

Some googling on the error message that was included in the log suggests one of the servers may have been running out of memory.  That's consistant with an apache restart fixing the problem (as that would have dumped all of the memory that was in use).  Guess I'll call this fixed for now, but we'll have to revisit to figure out why the memory got used up if that's what happened.