Closed Bug 1081046 Opened 10 years ago Closed 10 years ago

Load on mozdef

Categories

(Security Assurance :: General, task)

Other
Other
task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: dgarvey, Unassigned)

References

()

Details

(Whiteboard: [id=nagios1.private.scl3.mozilla.com:439910])

+++ This bug was initially created as a clone of Bug #1081042 +++

Automated alert report from nagios1.private.scl3.mozilla.com:

Hostname: mozdef1.private.scl3.mozilla.com
Service:  Load
State:    WARNING
Output:   WARNING - load average: 24.54, 20.25, 20.09

Runbook:  http://m.allizom.org/Load
This and mozdef2.private.scl3.mozilla.com have been flapin all night. The load/cpu is due to rabbitmq.
Whiteboard: [id=nagios1.private.scl3.mozilla.com:439910] → [kanban:https://kanbanize.com/ctrl_board/4/1554] [id=nagios1.private.scl3.mozilla.com:439910]
Whiteboard: [kanban:https://kanbanize.com/ctrl_board/4/1554] [id=nagios1.private.scl3.mozilla.com:439910] → [kanban:https://kanbanize.com/ctrl_board/4/1555] [id=nagios1.private.scl3.mozilla.com:439910]
Here is an example of all the noise that occurred.

Oct 08 15:47:32 <nagios-scl3>	Wed 15:47:32 PDT [5685] mozdef2.private.scl3.mozilla.com:Load is WARNING: WARNING - load average: 22.05, 19.03, 12.01 (http://m.mozilla.org/Load)
Oct 08 15:48:21 <nagios-scl3>	Wed 15:48:21 PDT [5686] mozdef1.private.scl3.mozilla.com:Load is CRITICAL: CRITICAL - load average: 25.27, 22.37, 15.08 (http://m.mozilla.org/Load)
Oct 08 15:52:31 <nagios-scl3>	Wed 15:52:31 PDT [5687] mozdef2.private.scl3.mozilla.com:Load is OK: OK - load average: 8.06, 13.13, 11.45 (http://m.mozilla.org/Load)
Oct 08 15:53:21 <nagios-scl3>	Wed 15:53:21 PDT [5688] mozdef1.private.scl3.mozilla.com:Load is OK: OK - load average: 11.97, 18.90, 15.90 (http://m.mozilla.org/Load)
Oct 09 08:07:15 <nagios-scl3>	Thu 08:07:15 PDT [5394] mozdef2.private.scl3.mozilla.com:Load is WARNING: WARNING - load average: 22.94, 20.06, 17.06 (http://m.mozilla.org/Load)
Oct 09 08:17:15 <nagios-scl3>	Thu 08:17:15 PDT [5407] mozdef2.private.scl3.mozilla.com:Load is OK: OK - load average: 9.49, 13.28, 15.63 (http://m.mozilla.org/Load)
Oct 09 17:11:32 <nagios-scl3>	Thu 17:11:32 PDT [5710] mozdef1.private.scl3.mozilla.com:Load is CRITICAL: CRITICAL - load average: 26.23, 23.28, 19.82 (http://m.mozilla.org/Load)
Oct 09 17:15:54 <nagios-scl3>	dgarvey: acknowledged service mozdef1.private.scl3.mozilla.com:Load
Oct 09 17:15:56 <nagios-scl3>	Thu 17:15:56 PDT mozdef1.private.scl3.mozilla.com:Load is ACKNOWLEDGEMENT (CRITICAL): CRITICAL - load average: 26.23, 23.28, 19.82 (http://m.mozilla.org/Load) (nagiosadmin) (dgarvey)rabbitmq running wild
Oct 09 17:16:12 <nagios-scl3>	Thu 17:16:12 PDT [5713] mozdef2.private.scl3.mozilla.com:Load is WARNING: WARNING - load average: 22.80, 19.98, 17.31 (http://m.mozilla.org/Load)
Oct 09 17:26:31 <nagios-scl3>	Thu 17:26:31 PDT [5719] mozdef1.private.scl3.mozilla.com:Load is WARNING: WARNING - load average: 22.96, 25.79, 23.85 (http://m.mozilla.org/Load)
Oct 09 17:36:28 <nagios-scl3>	Thu 17:36:28 PDT [5738] mozdef1.private.scl3.mozilla.com:Load is CRITICAL: CRITICAL - load average: 25.06, 24.26, 23.72 (http://m.mozilla.org/Load)
Oct 09 17:46:28 <nagios-scl3>	Thu 17:46:27 PDT [5744] mozdef1.private.scl3.mozilla.com:Load is WARNING: WARNING - load average: 24.88, 24.56, 24.09 (http://m.mozilla.org/Load)
Oct 09 17:51:28 <nagios-scl3>	Thu 17:51:28 PDT [5745] mozdef1.private.scl3.mozilla.com:Load is CRITICAL: CRITICAL - load average: 25.07, 23.83, 23.85 (http://m.mozilla.org/Load)
Oct 09 17:56:18 <nagios-scl3>	Thu 17:56:18 PDT [5747] mozdef2.private.scl3.mozilla.com:Load is OK: OK - load average: 14.02, 15.57, 18.50 (http://m.mozilla.org/Load)
Oct 09 17:56:28 <nagios-scl3>	Thu 17:56:28 PDT [5748] mozdef1.private.scl3.mozilla.com:Load is OK: OK - load average: 16.97, 20.71, 22.63 (http://m.mozilla.org/Load)
Oct 09 18:37:25 <nagios-scl3>	Thu 18:37:25 PDT [5757] mozdef1.private.scl3.mozilla.com:Load is WARNING: WARNING - load average: 21.39, 20.78, 19.35 (http://m.mozilla.org/Load)
Oct 09 18:42:25 <nagios-scl3>	Thu 18:42:25 PDT [5765] mozdef1.private.scl3.mozilla.com:Load is OK: OK - load average: 19.33, 19.40, 19.15 (http://m.mozilla.org/Load)
Oct 09 20:53:30 <nagios-scl3>	Thu 20:53:30 PDT [5836] mozdef1.private.scl3.mozilla.com:Load is WARNING: WARNING - load average: 20.22, 22.53, 19.93 (http://m.mozilla.org/Load)
Oct 09 20:58:35 <nagios-scl3>	Thu 20:58:34 PDT [5839] mozdef1.private.scl3.mozilla.com:Load is OK: OK - load average: 15.49, 17.82, 18.53 (http://m.mozilla.org/Load)
Oct 09 22:49:26 <nagios-scl3>	Thu 22:49:26 PDT [5878] mozdef1.private.scl3.mozilla.com:Load is WARNING: WARNING - load average: 23.27, 21.12, 19.29 (http://m.mozilla.org/Load)
Oct 09 22:54:27 <nagios-scl3>	Thu 22:54:27 PDT [5880] mozdef1.private.scl3.mozilla.com:Load is OK: OK - load average: 14.26, 18.44, 18.77 (http://m.mozilla.org/Load)
Oct 09 23:03:26 <nagios-scl3>	Thu 23:03:26 PDT [5893] mozdef1.private.scl3.mozilla.com:Load is WARNING: WARNING - load average: 20.84, 19.80, 19.37 (http://m.mozilla.org/Load)
Oct 09 23:13:26 <nagios-scl3>	Thu 23:13:26 PDT [5901] mozdef1.private.scl3.mozilla.com:Load is OK: OK - load average: 18.20, 19.14, 19.33 (http://m.mozilla.org/Load)
Oct 09 23:52:33 <nagios-scl3>	Thu 23:52:33 PDT [5912] mozdef1.private.scl3.mozilla.com:Load is WARNING: WARNING - load average: 21.69, 19.52, 18.27 (http://m.mozilla.org/Load)
Oct 09 23:57:32 <nagios-scl3>	Thu 23:57:32 PDT [5914] mozdef1.private.scl3.mozilla.com:Load is OK: OK - load average: 14.80, 18.21, 18.10 (http://m.mozilla.org/Load)
Oct 10 01:14:18 <nagios-scl3>	Fri 01:14:18 PDT [5952] mozdef1.private.scl3.mozilla.com:Load is CRITICAL: CRITICAL - load average: 26.99, 22.30, 20.12 (http://m.mozilla.org/Load)
Oct 10 01:24:17 <nagios-scl3>	Fri 01:24:17 PDT [5964] mozdef1.private.scl3.mozilla.com:Load is WARNING: WARNING - load average: 23.32, 23.51, 21.81 (http://m.mozilla.org/Load)
Oct 10 01:34:18 <nagios-scl3>	Fri 01:34:17 PDT [5972] mozdef1.private.scl3.mozilla.com:Load is CRITICAL: CRITICAL - load average: 26.33, 24.32, 22.35 (http://m.mozilla.org/Load)
Oct 10 01:39:18 <nagios-scl3>	Fri 01:39:18 PDT [5975] mozdef1.private.scl3.mozilla.com:Load is WARNING: WARNING - load average: 23.07, 22.75, 22.21 (http://m.mozilla.org/Load)
Oct 10 01:49:19 <nagios-scl3>	Fri 01:49:18 PDT [5978] mozdef1.private.scl3.mozilla.com:Load is OK: OK - load average: 16.54, 20.53, 21.65 (http://m.mozilla.org/Load)
Oct 10 02:03:15 <nagios-scl3>	Fri 02:03:14 PDT [5979] mozdef1.private.scl3.mozilla.com:Load is WARNING: WARNING - load average: 24.54, 20.25, 20.09 (http://m.mozilla.org/Load)
Oct 10 02:08:14 <nagios-scl3>	Fri 02:08:14 PDT [5980] mozdef1.private.scl3.mozilla.com:Load is CRITICAL: CRITICAL - load average: 25.46, 17.73, 18.48 (http://m.mozilla.org/Load)
Oct 10 02:13:04 <nagios-scl3>	Fri 02:13:04 PDT [5981] mozdef2.private.scl3.mozilla.com:Load is WARNING: WARNING - load average: 21.45, 22.39, 18.01 (http://m.mozilla.org/Load)
Oct 10 02:18:05 <nagios-scl3>	Fri 02:18:04 PDT [5983] mozdef2.private.scl3.mozilla.com:Load is OK: OK - load average: 16.00, 17.60, 17.08 (http://m.mozilla.org/Load)
Whiteboard: [kanban:https://kanbanize.com/ctrl_board/4/1555] [id=nagios1.private.scl3.mozilla.com:439910] → [id=nagios1.private.scl3.mozilla.com:439910]
Whiteboard: [id=nagios1.private.scl3.mozilla.com:439910] → [kanban:https://kanbanize.com/ctrl_board/4/1556] [id=nagios1.private.scl3.mozilla.com:439910]
Whiteboard: [kanban:https://kanbanize.com/ctrl_board/4/1556] [id=nagios1.private.scl3.mozilla.com:439910] → [kanban:https://kanbanize.com/ctrl_board/4/1557] [id=nagios1.private.scl3.mozilla.com:439910]
Reassigning this bug (operations for this host is handled by security).
Assignee: server-ops-webops → nobody
Component: WebOps: Other → Security Assurance: Operations
Product: Infrastructure & Operations → mozilla.org
QA Contact: nmaul
Whiteboard: [kanban:https://kanbanize.com/ctrl_board/4/1557] [id=nagios1.private.scl3.mozilla.com:439910] → [id=nagios1.private.scl3.mozilla.com:439910]
Now it is elasticsearch on mozdefes4. 

Bug1081410

Critical:

[root@mozdefes4.private.scl3 ~]# top

top - 17:11:15 up 162 days,  4:43,  2 users,  load average: 31.68, 27.52, 18.13
Tasks: 186 total,   1 running, 185 sleeping,   0 stopped,   0 zombie
Cpu(s): 79.5%us,  6.6%sy,  0.0%ni,  4.2%id,  8.0%wa,  0.0%hi,  1.6%si,  0.0%st
Mem:  32880920k total, 32629668k used,   251252k free,    62900k buffers
Swap:  2097144k total,    11788k used,  2085356k free, 14529252k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                    
18621 elsrch    20   0  933g  22g 6.2g S 348.7 73.1 324:26.55 java                                      
 1440 root      20   0     0    0    0 S  1.7  0.0   1849:12 rpciod/3                                   
 1444 root      20   0     0    0    0 S  1.0  0.0   1824:23 nfsiod                                     
 1148 root      20   0 15036 1300  944 R  0.3  0.0   0:00.04 top                                        
 1438 root      20   0     0    0    0 S  0.3  0.0 344:26.35 rpciod/1                                   
 1439 root      20   0     0    0    0 S  0.3  0.0   1925:04 rpciod/2                                   
    1 root      20   0 19364 1408 1104 S  0.0  0.0   0:38.67 init                                       
    2 root      20   0     0    0    0 S  0.0  0.0   0:02.02 kthreadd                                   
    3 root      RT   0     0    0    0 S  0.0  0.0   0:13.72 migration/0                                
    4 root      20   0     0    0    0 S  0.0  0.0   4:30.55 ksoftirqd/0                                
    5 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/0                                
    6 root      RT   0     0    0    0 S  0.0  0.0   0:17.72 watchdog/0                                 
    7 root      RT   0     0    0    0 S  0.0  0.0   0:19.78 migration/1                                
    8 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/1                                
    9 root      20   0     0    0    0 S  0.0  0.0   3:47.35 ksoftirqd/1                                
   10 root      RT   0     0    0    0 S  0.0  0.0   0:11.06 watchdog/1                                 
   11 root      RT   0     0    0    0 S  0.0  0.0   0:56.10 migration/2                                
   12 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/2                                
   13 root      20   0     0    0    0 S  0.0  0.0   3:23.69 ksoftirqd/2                                
   14 root      RT   0     0    0    0 S  0.0  0.0   0:10.78 watchdog/2                                 
   15 root      RT   0     0    0    0 S  0.0  0.0   1:12.97 migration/3                                
   16 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/3                                
   17 root      20   0     0    0    0 S  0.0  0.0   4:27.64 ksoftirqd/3                                
   18 root      RT   0     0    0    0 S  0.0  0.0   0:11.82 watchdog/3                                 
   19 root      20   0     0    0    0 S  0.0  0.0   7:34.39 events/0                                   
   20 root      20   0     0    0    0 S  0.0  0.0   7:07.50 events/1                                   
   21 root      20   0     0    0    0 S  0.0  0.0   8:18.10 events/2                                   
   22 root      20   0     0    0    0 S  0.0  0.0  13:48.66 events/3                                   
   23 root      20   0     0    0    0 S  0.0  0.0   0:00.00 cgroup                                     
   24 root      20   0     0    0    0 S  0.0  0.0   0:00.00 khelper                                    
   25 root      20   0     0    0    0 S  0.0  0.0   0:00.00 netns                                      
   26 root      20   0     0    0    0 S  0.0  0.0   0:00.00 async/mgr                                  
   27 root      20   0     0    0    0 S  0.0  0.0   0:00.00 pm                                         
   28 root      20   0     0    0    0 S  0.0  0.0   0:33.30 sync_supers                                
   29 root      20   0     0    0    0 S  0.0  0.0   0:37.59 bdi-default                                
   30 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kintegrityd/0                              
   31 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kintegrityd/1                              
   32 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kintegrityd/2                              
   33 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kintegrityd/3                              
   34 root      20   0     0    0    0 S  0.0  0.0   1:01.08 kblockd/0                                  
   35 root      20   0     0    0    0 S  0.0  0.0   1:00.72 kblockd/1                                  
   36 root      20   0     0    0    0 S  0.0  0.0   0:41.84 kblockd/2                                  
   37 root      20   0     0    0    0 S  0.0  0.0   0:31.68 kblockd/3                                  
[root@mozdefes4.private.scl3 ~]# 

Warning:
[dgarvey@mozdefes4.private.scl3 ~]$ top

top - 17:24:09 up 162 days,  4:56,  2 users,  load average: 26.45, 27.22, 23.90
Tasks: 179 total,   2 running, 177 sleeping,   0 stopped,   0 zombie
Cpu(s): 87.5%us,  6.9%sy,  0.0%ni,  3.6%id,  0.5%wa,  0.0%hi,  1.6%si,  0.0%st
Mem:  32880920k total, 31724256k used,  1156664k free,    64280k buffers
Swap:  2097144k total,    11788k used,  2085356k free, 13618116k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                    
18621 elsrch    20   0  936g  22g 5.8g S 379.7 71.7 371:40.28 java                                      
 1440 root      20   0     0    0    0 R  2.3  0.0   1849:27 rpciod/3                                   
 1444 root      20   0     0    0    0 S  1.0  0.0   1824:30 nfsiod                                     
 1437 root      20   0     0    0    0 S  0.3  0.0 424:59.90 rpciod/0                                   
 1439 root      20   0     0    0    0 S  0.3  0.0   1925:06 rpciod/2                                   
    1 root      20   0 19364 1408 1104 S  0.0  0.0   0:38.67 init                                       
    2 root      20   0     0    0    0 S  0.0  0.0   0:02.02 kthreadd                                   
    3 root      RT   0     0    0    0 S  0.0  0.0   0:13.72 migration/0                                
    4 root      20   0     0    0    0 S  0.0  0.0   4:30.55 ksoftirqd/0                                
    5 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/0                                
    6 root      RT   0     0    0    0 S  0.0  0.0   0:17.72 watchdog/0                                 
    7 root      RT   0     0    0    0 S  0.0  0.0   0:19.78 migration/1                                
    8 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/1                                
    9 root      20   0     0    0    0 S  0.0  0.0   3:47.35 ksoftirqd/1                                
   10 root      RT   0     0    0    0 S  0.0  0.0   0:11.06 watchdog/1                                 
   11 root      RT   0     0    0    0 S  0.0  0.0   0:56.10 migration/2                                
   12 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/2                                
   13 root      20   0     0    0    0 S  0.0  0.0   3:23.69 ksoftirqd/2                                
   14 root      RT   0     0    0    0 S  0.0  0.0   0:10.78 watchdog/2                                 
   15 root      RT   0     0    0    0 S  0.0  0.0   1:12.97 migration/3                                
   16 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/3                                
   17 root      20   0     0    0    0 S  0.0  0.0   4:27.69 ksoftirqd/3                                
   18 root      RT   0     0    0    0 S  0.0  0.0   0:11.82 watchdog/3                                 
   19 root      20   0     0    0    0 S  0.0  0.0   7:34.42 events/0                                   
   20 root      20   0     0    0    0 S  0.0  0.0   7:07.52 events/1                                   
   21 root      20   0     0    0    0 S  0.0  0.0   8:18.13 events/2                                   
   22 root      20   0     0    0    0 S  0.0  0.0  13:48.70 events/3                                   
   23 root      20   0     0    0    0 S  0.0  0.0   0:00.00 cgroup                                     
   24 root      20   0     0    0    0 S  0.0  0.0   0:00.00 khelper                                    
   25 root      20   0     0    0    0 S  0.0  0.0   0:00.00 netns                                      
   26 root      20   0     0    0    0 S  0.0  0.0   0:00.00 async/mgr                                  
   27 root      20   0     0    0    0 S  0.0  0.0   0:00.00 pm                                         
   28 root      20   0     0    0    0 S  0.0  0.0   0:33.31 sync_supers                                
   29 root      20   0     0    0    0 S  0.0  0.0   0:37.59 bdi-default                                
   30 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kintegrityd/0                              
   31 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kintegrityd/1                              
   32 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kintegrityd/2                              
   33 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kintegrityd/3                              
   34 root      20   0     0    0    0 S  0.0  0.0   1:01.08 kblockd/0                                  
   35 root      20   0     0    0    0 S  0.0  0.0   1:00.73 kblockd/1                                  
   36 root      20   0     0    0    0 S  0.0  0.0   0:41.84 kblockd/2                                  
   37 root      20   0     0    0    0 S  0.0  0.0   0:31.69 kblockd/3                                  
   38 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kacpid                                     
[dgarvey@mozdefes4.private.scl3 ~]$ 

Critical again:

[dgarvey@mozdefes4.private.scl3 ~]$ top

top - 17:28:52 up 162 days,  5:01,  2 users,  load average: 28.31, 27.97, 25.08
Tasks: 179 total,   1 running, 178 sleeping,   0 stopped,   0 zombie
Cpu(s): 75.4%us,  6.5%sy,  0.0%ni,  6.7%id,  9.7%wa,  0.0%hi,  1.7%si,  0.0%st
Mem:  32880920k total, 32373704k used,   507216k free,    64384k buffers
Swap:  2097144k total,    11788k used,  2085356k free, 14266268k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                    
18621 elsrch    20   0  937g  22g 5.9g S 328.8 72.1 389:13.96 java                                      
 1440 root      20   0     0    0    0 S  2.7  0.0   1849:32 rpciod/3                                   
 1444 root      20   0     0    0    0 S  1.0  0.0   1824:33 nfsiod                                     
 1437 root      20   0     0    0    0 S  0.3  0.0 425:00.93 rpciod/0                                   
 1438 root      20   0     0    0    0 S  0.3  0.0 344:29.77 rpciod/1                                   
 1439 root      20   0     0    0    0 S  0.3  0.0   1925:07 rpciod/2                                   
 4079 dgarvey   20   0 15036 1296  944 R  0.3  0.0   0:00.04 top                                        
32488 root      20   0     0    0    0 S  0.3  0.0   0:00.43 flush-0:19                                 
    1 root      20   0 19364 1408 1104 S  0.0  0.0   0:38.67 init                                       
    2 root      20   0     0    0    0 S  0.0  0.0   0:02.02 kthreadd                                   
    3 root      RT   0     0    0    0 S  0.0  0.0   0:13.72 migration/0                                
    4 root      20   0     0    0    0 S  0.0  0.0   4:30.55 ksoftirqd/0                                
    5 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/0                                
    6 root      RT   0     0    0    0 S  0.0  0.0   0:17.72 watchdog/0                                 
    7 root      RT   0     0    0    0 S  0.0  0.0   0:19.78 migration/1                                
    8 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/1                                
    9 root      20   0     0    0    0 S  0.0  0.0   3:47.35 ksoftirqd/1                                
   10 root      RT   0     0    0    0 S  0.0  0.0   0:11.06 watchdog/1                                 
   11 root      RT   0     0    0    0 S  0.0  0.0   0:56.10 migration/2                                
   12 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/2                                
   13 root      20   0     0    0    0 S  0.0  0.0   3:23.69 ksoftirqd/2                                
   14 root      RT   0     0    0    0 S  0.0  0.0   0:10.78 watchdog/2                                 
   15 root      RT   0     0    0    0 S  0.0  0.0   1:12.98 migration/3                                
   16 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/3                                
   17 root      20   0     0    0    0 S  0.0  0.0   4:27.69 ksoftirqd/3                                
   18 root      RT   0     0    0    0 S  0.0  0.0   0:11.82 watchdog/3                                 
   19 root      20   0     0    0    0 S  0.0  0.0   7:34.44 events/0                                   
   20 root      20   0     0    0    0 S  0.0  0.0   7:07.53 events/1                                   
   21 root      20   0     0    0    0 S  0.0  0.0   8:18.14 events/2                                   
   22 root      20   0     0    0    0 S  0.0  0.0  13:48.72 events/3                                   
   23 root      20   0     0    0    0 S  0.0  0.0   0:00.00 cgroup                                     
   24 root      20   0     0    0    0 S  0.0  0.0   0:00.00 khelper                                    
   25 root      20   0     0    0    0 S  0.0  0.0   0:00.00 netns                                      
   26 root      20   0     0    0    0 S  0.0  0.0   0:00.00 async/mgr                                  
   27 root      20   0     0    0    0 S  0.0  0.0   0:00.00 pm                                         
   28 root      20   0     0    0    0 S  0.0  0.0   0:33.31 sync_supers                                
   29 root      20   0     0    0    0 S  0.0  0.0   0:37.59 bdi-default                                
   30 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kintegrityd/0                              
   31 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kintegrityd/1                              
   32 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kintegrityd/2                              
   33 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kintegrityd/3                              
   34 root      20   0     0    0    0 S  0.0  0.0   1:01.08 kblockd/0                                  
   35 root      20   0     0    0    0 S  0.0  0.0   1:00.73 kblockd/1                                  
[dgarvey@mozdefes4.private.scl3 ~]$
Summary: Load on mozdef1.private.scl3.mozilla.com is WARNING: WARNING - load average: 24.54, 20.25, 20.09 → Load on mozdef
Yes, it was catching up from crashing and trying to work through a 13million event backlog.

It is caught up now and should be fine.
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Component: Operations Security (OpSec): General → General
Product: mozilla.org → Enterprise Information Security
You need to log in before you can comment on or make changes to this bug.