931308 - tableau1.metrics.scl3.mozilla.com:Windows Memory is CRITICAL

Status: RESOLVED WONTFIX

(mozilla.org Graveyard :: Server Operations, task)

Description

[Adrian J Fernandez [:Aj]]

Hello Ben,

Not sure if you are currently doing any specific task but this machine is almost using all of it's available memory;

physical memory: Total: 32G - Used: 31.5G (98%) - Free: 513M (2%)

Comment 1

[Adrian J Fernandez [:Aj]]

This recovered but leaving open to see if this will be a common trend.

Comment 2

[Rick Bryce [:rbryce]]

Seemed to me a one time event.  Closing for now.

Comment 3

[Peter Radcliffe [:pir]]

This has been reoccurring in the last couple of days and has gone up considerably in the last day. :bsullins it looks like this needs reducing, from whatever is running on the machine at 7am.


Subject: PROBLEM(WARNING): tableau1.metrics.scl3.mozilla.com/Windows Memory     
Info: WARNING: physical memory: Total: 32G - Used: 27.9G (87%) - Free: 4.11G (13
%)  warning              
Date: 11-18-2013 07:00:49

Subject: RECOVERY(OK): tableau1.metrics.scl3.mozilla.com/Windows Memory         
Info: OK: physical memory: 23.7G
Date: 11-18-2013 07:10:49

Subject: PROBLEM(CRITICAL): tableau1.metrics.scl3.mozilla.com/Windows Memory    
Info: CRITICAL: physical memory: Total: 32G - Used: 30.7G (96%) - Free: 1.25G (4
%)  critical             
Date: 11-19-2013 07:00:46

Subject: RECOVERY(OK): tableau1.metrics.scl3.mozilla.com/Windows Memory         
Info: OK: physical memory: 23.8G
Date: 11-19-2013 07:10:46

Comment 4

[bsullins]

Thanks for the heads up. It appears that one of our extract refreshes is taking up a considerable amount of time to finish which is where I'm assuming this is stemming from. I'm going to try converting it to a live connection versus in-memory and hopefully mitigate this issue. More to come...

Comment 5

[Peter Radcliffe [:pir]]

again today:

<nagios-scl3:#sysadmins> Wed 07:00:28 PST [5080] 
  tableau1.metrics.scl3.mozilla.com:Windows Memory is CRITICAL: CRITICAL: 
  physical memory: Total: 32G - Used: 29.9G (93%) - Free: 2.09G (7%)  critical 
  (http://m.allizom.org/Windows+Memory)

Comment 6

[bsullins]

Sorry, yesterday I got pulled into meetings all day and didn't have time to kill the job that is causing this. I've killed it now, and am going to research how to mitigate this. Will update once I have something in place I think is going to work.

Comment 8

[Peter Radcliffe [:pir]]

Alerted during Ashish's shift today but not as high as before:

00:41:15 < nagios-scl3> Fri 00:41:15 PST [5927] tableau1.metrics.scl3.mozilla.com:Windows Memory is WARNING: WARNING: physical memory: Total: 32G - Used: 26G (81%) - Free: 6G (19%)  warning (http://m.allizom.org/Windows+Memory)

Comment 9

[Rick Bryce [:rbryce]]

Wontfixing. We will just have to handle these memory spikes with Ben as they come.