448967 - BES offline due to pm-vmware03 reboot on Aug 2

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Dave Miller [:justdave]]

pm-vmware03 hung and had to get rebooted on Aug 2.  The BES server was one of the VMs affected, and BES never recovered.

Comment 1

[matthew zeier [:mrz]]

BES starts up on boot up - not clear what else to check, passing to Sean.

Comment 2

[matthew zeier [:mrz]]

Can't tell if it's fixed - I restarted the BES service and sent Shaver and email.  BES says it forwarded it.  Shaver?

Comment 3

[Mike Shaver (:shaver emeritus)]

I'm getting BES mail now too.  Can we get nagios coverage on this, by any chance?

Comment 4

[Reed Loden [:reed]]

(In reply to comment #3)
> Can we get nagios coverage on this, by any chance?

We have Windows service checks on it now to see if the various BlackBerry services are running, but that never seems to fail when there's a problem. Any ideas/recommendations on how to better test this with nagios?

Comment 5

[Mike Shaver (:shaver emeritus)]

Track the logs and send heartbeat messages through?  I don't have access to any of the knobs or gauges that might help answer that question, but I'd be pretty surprised if RIM doesn't have a stock answer.

Comment 6

[Sean Alamares]

Passing over to Aravind for the monitor stuff.

Comment 7

[Aravind Gottipati [:aravind]]

I am not sure which log file should be monitored for activity for this.  It looks like the sync log file is the place to look.  I added a check for that file.