Talos object processing stalled on datazilla

RESOLVED INVALID

Status

Infrastructure & Operations
WebOps: Other
RESOLVED INVALID
5 years ago
4 years ago

People

(Reporter: jeads, Unassigned)

Tracking

Details

(Reporter)

Description

5 years ago
The last object processed on datazilla appears to be from Tue Jul 16 04:11:50 2013. It appears object processing has stalled for talos data. This likely corresponds to a recent set of issues in talos that have caused enormous numbers of replicates to be submitted for certain test/branch combinations.

We need some more information to diagnose accurately. 

1.) Could someone confirm that there is a process running on the admin node that corresponds to this command?

manage.py process_objects --cron_batch small --loadlimit 25

If it's not found in the process list, please try running the following command directly from the command line with the debug switch and capture the output.

manage.py process_objects --cron_batch small --loadlimit 1 --debug

2.) If there is a process running, please run an 'strace -p pid' and an 'lsof -p pid' on that process and post the results
(Reporter)

Comment 1

5 years ago
Ok not sure what happened but it looks like object processing has resumed. I have a theory about why this occurred but I need to investigate further. Closing this bug for now...
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → INVALID
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.