Closed Bug 1009218 Opened 12 years ago Closed 12 years ago

Datazilla Object Processing Problem

Categories

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: jeads, Assigned: cturra)

Details

The last object processed in datazilla in b2g_objectstore_1 appears to be from 05/09/14 9:38:49am, lots of objects have been written to the objectstore since then but they are not being processed. This seems to be the case with all of the datazilla projects including talos. Json objects are still being written to the objectstore but are not getting loaded into the corresponding *_perftest_1 schemas. The most likely explanation for this is that the process_objects manage.py command is not being run on cron. Could the crons have been disabled from work on Bug 993665? or are there any process_objects lock files around? Maybe the first thing to check would be to confirm that the process_objects commands are running in the process list. If not, try running the command on the admin node and see if there are any error messages generated. manage.py process_objects --project b2g --loadlimit 25
looks like we had the talos process_objects backing up again. [root@datazillaadm.private.scl3 ~]# ps aux | grep [t]alos | wc -l 282 i have manually killed these processes and am manually running the process_object command to catch us back up. it looks like there are conditions where the process_objects for talos does not complete in the 2 minutes aloted for this cron. when that happens, another kicks off and doesn't appear to exit when it finds the lock file. as a result, it appears that these processes all get stuck in some sort of lock condition. :jeads - can you confirm code exits to check that the lock file is present and if so, exists correctly?
Assignee: server-ops-webops → cturra
Flags: needinfo?(jeads)
OS: Mac OS X → All
Hardware: x86 → All
looks like the object processing is catching up after the manual intervention.
I'm still working on reproducing this locally. I agree there must be some set of conditions that result in the lock file not being observed. I will have more to report soon.
Flags: needinfo?(jeads)
Ok, I was able to reproduce part of this scenario. Still not entirely sure what causes it but I think I have a solid work around. I set a timeout of 10 seconds, if a lock is not acquired in that time period the command exits. You can see the code here: https://github.com/mozilla/datazilla/blob/master/datazilla/controller/admin/management/commands/base.py#L133 https://github.com/mozilla/datazilla/blob/master/datazilla/controller/admin/management/commands/base.py#L150 I tested this extensively and the command reproducibly exits after the time interval when another process has a lock file. I'm going to continue to dig until I can find why this is occurring. This "except" is not being reached in production and I don't have an explanation for that yet. https://github.com/mozilla/datazilla/blob/master/datazilla/controller/admin/management/commands/base.py#L146 Regardless, I like the safety net of an explicit timeout here, it should prevent this scenario from reoccurring permanently. Please push out the modified repository.
Flags: needinfo?(cturra)
i just pushed this change to stage. would you like me to do the same to prod?
Flags: needinfo?(cturra) → needinfo?(jeads)
Yes please! Stage looks good, this change is only relevant to the process_objects manage command which is only run on the admin node.
Flags: needinfo?(jeads)
this code has been pushed to prod also. let me know when you want to mark this bug as r/fixed or if there are any outstanding items you want to run through.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.