566718 - [hudson] Restart hudson every 1-2 days

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Jeff Balogh (:jbalogh)]

Please set up a cron job to restart hudson every 1 or 2 days.  Hudson is a Java beast that will keep gobbling up memory forever.  It starts running pretty slowly eventually, and now it's not even responding.

This should happen late at night when only wenzel might be affected.

`/etc/init.d/hudson restart` should take care of it. Thanks!

Comment 1

[Shyam Mani [:fox2mike]]

Yuck.

Comment 2

[Shyam Mani [:fox2mike]]

Going to see if I can make this happen in a nicer way rather than just restart it all the time.

Comment 3

[Shyam Mani [:fox2mike]]

Okay, I've enabled -Xincgc which is the incremental garbage collector (vs doing it all in one shot after ages). This is off by default (yay java!) and so let's see if this helps.

Jeff, let me know if it makes a difference :) It should...hopefully.

Comment 4

[Fred Wenzel [:wenzel]]

(In reply to comment #0)
> This should happen late at night when only wenzel might be affected.

boo.

(In reply to comment #2)
> Going to see if I can make this happen in a nicer way rather than just restart
> it all the time.

yay!

Comment 5

[Shyam Mani [:fox2mike]]

Jeff, do you see any improvement?

Comment 6

[Jeff Balogh (:jbalogh)]

It's still running over 2 gigs of memory, but it's still responsive a week after comment 3.  I'll reopen the bug if it gets bogged down again.  Thanks Shyam.

Comment 7

[Shyam Mani [:fox2mike]]

For the sake of completeness, this is what I added to /etc/sysconfig/hudson :

HUDSON_JAVA_OPTIONS="-Djava.awt.headless=true -Xincgc -Xms1024m -Xmx2048m"