Closed
Bug 1009218
Opened 12 years ago
Closed 12 years ago
Datazilla Object Processing Problem
Categories
(Infrastructure & Operations Graveyard :: WebOps: Other, task)
Infrastructure & Operations Graveyard
WebOps: Other
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: jeads, Assigned: cturra)
Details
The last object processed in datazilla in b2g_objectstore_1 appears to be from 05/09/14 9:38:49am, lots of objects have been written to the objectstore since then but they are not being processed. This seems to be the case with all of the datazilla projects including talos. Json objects are still being written to the objectstore but are not getting loaded into the corresponding *_perftest_1 schemas. The most likely explanation for this is that the process_objects manage.py command is not being run on cron.
Could the crons have been disabled from work on Bug 993665? or are there any process_objects lock files around?
Maybe the first thing to check would be to confirm that the process_objects commands are running in the process list. If not, try running the command on the admin node and see if there are any error messages generated.
manage.py process_objects --project b2g --loadlimit 25
| Assignee | ||
Comment 1•12 years ago
|
||
looks like we had the talos process_objects backing up again.
[root@datazillaadm.private.scl3 ~]# ps aux | grep [t]alos | wc -l
282
i have manually killed these processes and am manually running the process_object command to catch us back up.
it looks like there are conditions where the process_objects for talos does not complete in the 2 minutes aloted for this cron. when that happens, another kicks off and doesn't appear to exit when it finds the lock file. as a result, it appears that these processes all get stuck in some sort of lock condition.
:jeads - can you confirm code exits to check that the lock file is present and if so, exists correctly?
Assignee: server-ops-webops → cturra
Flags: needinfo?(jeads)
OS: Mac OS X → All
Hardware: x86 → All
| Assignee | ||
Comment 2•12 years ago
|
||
looks like the object processing is catching up after the manual intervention.
| Reporter | ||
Comment 3•12 years ago
|
||
I'm still working on reproducing this locally. I agree there must be some set of conditions that result in the lock file not being observed. I will have more to report soon.
Flags: needinfo?(jeads)
| Reporter | ||
Comment 4•12 years ago
|
||
Ok, I was able to reproduce part of this scenario. Still not entirely sure what causes it but I think I have a solid work around. I set a timeout of 10 seconds, if a lock is not acquired in that time period the command exits. You can see the code here:
https://github.com/mozilla/datazilla/blob/master/datazilla/controller/admin/management/commands/base.py#L133
https://github.com/mozilla/datazilla/blob/master/datazilla/controller/admin/management/commands/base.py#L150
I tested this extensively and the command reproducibly exits after the time interval when another process has a lock file.
I'm going to continue to dig until I can find why this is occurring. This "except" is not being reached in production and I don't have an explanation for that yet.
https://github.com/mozilla/datazilla/blob/master/datazilla/controller/admin/management/commands/base.py#L146
Regardless, I like the safety net of an explicit timeout here, it should prevent this scenario from reoccurring permanently. Please push out the modified repository.
Flags: needinfo?(cturra)
| Assignee | ||
Comment 5•12 years ago
|
||
i just pushed this change to stage. would you like me to do the same to prod?
Flags: needinfo?(cturra) → needinfo?(jeads)
| Reporter | ||
Comment 6•12 years ago
|
||
Yes please! Stage looks good, this change is only relevant to the process_objects manage command which is only run on the admin node.
Flags: needinfo?(jeads)
| Assignee | ||
Comment 7•12 years ago
|
||
this code has been pushed to prod also. let me know when you want to mark this bug as r/fixed or if there are any outstanding items you want to run through.
| Reporter | ||
Updated•12 years ago
|
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Updated•7 years ago
|
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in
before you can comment on or make changes to this bug.
Description
•