797846 - please increase the ram allocated to bugstage1.dmz.scl3 to 12g

Status: RESOLVED INCOMPLETE

(Developer Services :: General, task)

Description

[:glob ✱]

earlier today pir reported that bugstage1.dmz.scl3 was unresponsive and required a hard reboot due to memory exhaustion.

it has 3g of ram, however bugzilla is configured for production, which has 12g.

as bugzilla increases its use of caching, i would expect similar problems to reoccur.


would it be possible to increase the ram allocated to this vm to match production?


thanks.

Comment 1

[Shyam Mani [:fox2mike]]

Peter,

Just curious, what happened to bugstage1.dmz.scl3? (before I decide on what to do to long term fix the issue).

Thanks!

Comment 2

[Peter Radcliffe [:pir]]

My guess would be that it ran out of swap.

It went completely unresponsive on the net to anything but ping. ssh connections timed out, etc. When I went to the console it responded but had the vmware key repeat bug so there was no way to log in and do further debugging.
I shut it down, fixed the keyrepeat in vmware, brought it back up again.

Comment 3

[Shyam Mani [:fox2mike]]

Thanks!

glob : I'm going to let this one sit for a bit and see if we can pull some more stats before we decide to up the RAM.

Comment 4

[:glob ✱]

(In reply to Shyam Mani [:fox2mike] from comment #3)
> glob : I'm going to let this one sit for a bit and see if we can pull some
> more stats before we decide to up the RAM.

that's fine.

Comment 5

[Peter Radcliffe [:pir]]

bugstage1 stopped taking ssh connections for a while just now but recovered enough on it's own to let me get in and poke about.

It was well into swap, httpd processes taking up most of the cpu. I restarted the httpd service and all the issues recovered and swap usage dropped considerably.

The box has 3Gb of memory, I think it needs increasing. Graphing memory usage of httpd over time might be interesting to see if there are any memory leaks.

Comment 6

[Shyam Mani [:fox2mike]]

So the only thing stopping me from doing this right away is Bugzilla's entire stage enviro is undergoing and overhaul and this box will probably be gone by Q1 2013. If it happens again we can bump it up temporarily, but since it's stage and it's going away, I'm not too keep to do it right away.

Going to mark this incomplete for now, please re-open if it breaks badly again.

Comment 7

[Shyam Mani [:fox2mike]]

Happened again today. Found the reason why. Will open an opsec bug and CC you all there.

Machine is x86, so max RAM it can have is 4GB.