Closed Bug 1025591 Opened 10 years ago Closed 10 years ago

Socorro Admin - cron_submitter-crash-reports.allizom.org.log on sp-admin01.phx1.mozilla.com is CRITICAL: FILE_AGE CRITICAL: /var/log/socorro/cron_submitter-crash-reports.allizom.org.log is 1208 seconds old and 571534250 bytes

Categories

(mozilla.org Graveyard :: Server Operations: MOC, task)

Product:
mozilla.org Graveyard
Component:
Server Operations: MOC
Platform:
Other
Other
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: nagiosapi, Unassigned)

References

(
URL
)

Details

(Whiteboard: [id=nagios1.private.phx1.mozilla.com:336190]) 

Automated alert report from nagios1.private.phx1.mozilla.com:

Hostname: sp-admin01.phx1.mozilla.com
Service:  Socorro Admin - cron_submitter-crash-reports.allizom.org.log
State:    CRITICAL
Output:   FILE_AGE CRITICAL: /var/log/socorro/cron_submitter-crash-reports.allizom.org.log is 1208 seconds old and 571534250 bytes

Runbook:  http://m.allizom.org/Socorro+Admin+-+cron_submitter-crash-reports.allizom.org.log
[dgarvey@sp-admin01.phx1 ~]$ ps -efww | grep socorro
socorro  15197 15194  0 00:00 ?        00:00:00 /bin/bash /data/socorro/application/scripts/crons/crontabber.sh
socorro  15201 15197  0 00:00 ?        00:00:04 /data/socorro/socorro-virtualenv/bin/python /data/socorro/application/socorro/cron/crontabber_app.py --admin.conf=/etc/socorro/crontabber.ini
socorro  15372 15369  0 00:05 ?        00:00:00 /bin/sh -c /data/socorro/application/scripts/crons/cron_libraries.sh >> /var/log/socorro/cron_libraries.log 2>&1
socorro  15373 15372  0 00:05 ?        00:00:00 /bin/bash /data/socorro/application/scripts/crons/cron_libraries.sh
socorro  15393 15373 31 00:05 ?        00:05:14 /data/socorro/socorro-virtualenv/bin/python /data/socorro/application/socorro/external/hbase/hbase_client.py -h 10.8.100.212 export_jsonz_tarball_for_ooids /tmp/tmp.aRVwG2kixZ /tmp/tmp.aRVwG2kixZ/Firefox_29.0.1.tar
dgarvey  17996 17613  0 00:21 pts/0    00:00:00 grep socorro
[dgarvey@sp-admin01.phx1 ~]$ tail /var/log/socorro/cron_submitter-crash-reports.allizom.org.log
2014-06-14 23:54:23,405 DEBUG  - Thread-1 - submission response: CrashID=bp-91bf06fa-e134-4c06-ad35-324a72140615
2014-06-14 23:54:23,477 DEBUG  - Thread-3 - submitted e9554757-d2f3-4e81-bd3e-7b1c32140615 (original crash_id)
2014-06-14 23:54:23,477 DEBUG  - Thread-3 - submission response: CrashID=bp-e9554757-d2f3-4e81-bd3e-7b1c32140615
2014-06-14 23:54:23,511 DEBUG  - Thread-2 - submitted 77fc3036-175e-4a87-9305-552542140615 (original crash_id)
2014-06-14 23:54:23,511 DEBUG  - Thread-2 - submission response: CrashID=bp-77fc3036-175e-4a87-9305-552542140615
2014-06-14 23:54:32,144 DEBUG  - QueuingThread - the loop didn't actually loop
2014-06-14 23:54:32,144 DEBUG  - QueuingThread - we're quitting queuingThread
2014-06-14 23:54:32,144 DEBUG  - QueuingThread - waiting for standard worker threads to stop
2014-06-14 23:54:32,145 DEBUG  - QueuingThread - all worker threads stopped
2014-06-14 23:54:32,162 INFO  - MainThread - done.
[dgarvey@sp-admin01.phx1 ~]$
HBASE? I thought that was only used for dev?
Automated alert acknowledgement: (w0ts0n)1025603
Status: NEW → ASSIGNED
Automated alert recovery:

Hostname: sp-admin01.phx1.mozilla.com
Service:  Socorro Admin - cron_submitter-crash-reports.allizom.org.log
State:    OK
Output:   FILE_AGE OK: /var/log/socorro/cron_submitter-crash-reports.allizom.org.log is 3 seconds old and 0 bytes
Status: ASSIGNED → RESOLVED
Closed: 10 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.