Socorro Admin - cron_submitter-crash-reports.allizom.org.log on sp-admin01.phx1.mozilla.com is CRITICAL: FILE_AGE CRITICAL: /var/log/socorro/cron_submitter-crash-reports.allizom.org.log is 1208 seconds old and 571534250 bytes

RESOLVED FIXED

Status

mozilla.org Graveyard
Server Operations: MOC
RESOLVED FIXED
4 years ago
3 years ago

People

(Reporter: MOC Nagios API, Unassigned)

Tracking

Details

(Whiteboard: [id=nagios1.private.phx1.mozilla.com:336190], URL)

(Reporter)

Description

4 years ago
Automated alert report from nagios1.private.phx1.mozilla.com:

Hostname: sp-admin01.phx1.mozilla.com
Service:  Socorro Admin - cron_submitter-crash-reports.allizom.org.log
State:    CRITICAL
Output:   FILE_AGE CRITICAL: /var/log/socorro/cron_submitter-crash-reports.allizom.org.log is 1208 seconds old and 571534250 bytes

Runbook:  http://m.allizom.org/Socorro+Admin+-+cron_submitter-crash-reports.allizom.org.log

Comment 1

4 years ago
[dgarvey@sp-admin01.phx1 ~]$ ps -efww | grep socorro
socorro  15197 15194  0 00:00 ?        00:00:00 /bin/bash /data/socorro/application/scripts/crons/crontabber.sh
socorro  15201 15197  0 00:00 ?        00:00:04 /data/socorro/socorro-virtualenv/bin/python /data/socorro/application/socorro/cron/crontabber_app.py --admin.conf=/etc/socorro/crontabber.ini
socorro  15372 15369  0 00:05 ?        00:00:00 /bin/sh -c /data/socorro/application/scripts/crons/cron_libraries.sh >> /var/log/socorro/cron_libraries.log 2>&1
socorro  15373 15372  0 00:05 ?        00:00:00 /bin/bash /data/socorro/application/scripts/crons/cron_libraries.sh
socorro  15393 15373 31 00:05 ?        00:05:14 /data/socorro/socorro-virtualenv/bin/python /data/socorro/application/socorro/external/hbase/hbase_client.py -h 10.8.100.212 export_jsonz_tarball_for_ooids /tmp/tmp.aRVwG2kixZ /tmp/tmp.aRVwG2kixZ/Firefox_29.0.1.tar
dgarvey  17996 17613  0 00:21 pts/0    00:00:00 grep socorro
[dgarvey@sp-admin01.phx1 ~]$ tail /var/log/socorro/cron_submitter-crash-reports.allizom.org.log
2014-06-14 23:54:23,405 DEBUG  - Thread-1 - submission response: CrashID=bp-91bf06fa-e134-4c06-ad35-324a72140615
2014-06-14 23:54:23,477 DEBUG  - Thread-3 - submitted e9554757-d2f3-4e81-bd3e-7b1c32140615 (original crash_id)
2014-06-14 23:54:23,477 DEBUG  - Thread-3 - submission response: CrashID=bp-e9554757-d2f3-4e81-bd3e-7b1c32140615
2014-06-14 23:54:23,511 DEBUG  - Thread-2 - submitted 77fc3036-175e-4a87-9305-552542140615 (original crash_id)
2014-06-14 23:54:23,511 DEBUG  - Thread-2 - submission response: CrashID=bp-77fc3036-175e-4a87-9305-552542140615
2014-06-14 23:54:32,144 DEBUG  - QueuingThread - the loop didn't actually loop
2014-06-14 23:54:32,144 DEBUG  - QueuingThread - we're quitting queuingThread
2014-06-14 23:54:32,144 DEBUG  - QueuingThread - waiting for standard worker threads to stop
2014-06-14 23:54:32,145 DEBUG  - QueuingThread - all worker threads stopped
2014-06-14 23:54:32,162 INFO  - MainThread - done.
[dgarvey@sp-admin01.phx1 ~]$

Comment 2

4 years ago
HBASE? I thought that was only used for dev?
(Reporter)

Comment 3

4 years ago
Automated alert acknowledgement: (w0ts0n)1025603
Status: NEW → ASSIGNED
(Reporter)

Comment 4

4 years ago
Automated alert recovery:

Hostname: sp-admin01.phx1.mozilla.com
Service:  Socorro Admin - cron_submitter-crash-reports.allizom.org.log
State:    OK
Output:   FILE_AGE OK: /var/log/socorro/cron_submitter-crash-reports.allizom.org.log is 3 seconds old and 0 bytes
Status: ASSIGNED → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
(Assignee)

Updated

3 years ago
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.