993458 - self-serve running extremely slow and often getting "Service Unavailable" messages

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Ryan VanderMeulen [:RyanVM]]

All trees currently closed.

Comment 1

[Jordan Lund (:jlund)]

bug 983125 landed last week and could be related. I'm just poking logs of our machines that run selfserve

Comment 2

[Jordan Lund (:jlund)]

selfserve agents seem to be running OK according to logs.

dustin pointed out the web app is buildapi and looks to be the source of the problem.

restarting now

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

Hm, lots of 

[Tue Apr 08 08:30:34 2014] [error] [client 10.22.81.208] Script timed out before returning headers: buildapi.wsgi, referer: https://secure.pub.build.mozilla.org/buildapi/self-serve/mozilla-inbound/rev/468b75c559b8

Restarting Apache on the webheads seems to have it responding faster, but of course that just means it will happen again :(

Comment 4

[Jordan Lund (:jlund)]

seems to have gotten better thanks to dustins probing.

for the record (and my own), it it's a HTTP problem like request timeouts, it is the buildapi web app issue. if requests are not getting completed (from here: https://secure.pub.build.mozilla.org/buildapi/self-serve/jobs), it's probably do to the self serve agents.

troubleshooting both can be found here as of right now: https://wiki.mozilla.org/ReleaseEngineering/How_To/Restart_BuildAPI

closing. if it happens again we can re open

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

I'm fairly certain this will happen again.  We'll see how quickly.  I'm going to move this to webops since I'll likely be pinging them for help debugging when this happens again.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

OK, we dialed mod_wsgi up to 6 processes, with one thread each, rather than the 2/2 it was configured for.  I'm guessing here, but presumably we've reached the point where we get more than four simultaneous buildapi connections, at which point requests queue up.

This had the side-effect of restarting Apache, so whether this medicine worked or not, it's back to normal now.  Still, I'm going to optimistically call this done.