Closed Bug 815378 Opened 12 years ago Closed 12 years ago

Celery countdown reschedules tasks many hours into the future

Categories

(Marketplace Graveyard :: General, defect, P2)

Product:
Marketplace Graveyard
Component:
General
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED
Milestone:
2013-05-23

People

(Reporter: robhudson, Assigned: wraithan)

References

Details

If you grep the logs for "eta:" you'll see tasks that are getting scheduled, many with a countdown of 3 seconds, that get scheduled 8 hours (plus or minus) in the future instead of just seconds. I suspect a timezone bug somewhere. Or a missed timezone setting.
that sounds pretty important. I assume ntpdate or something is running on all the servers but CCing oremj and jason in case anything comes to mind.
'date' on all the web and celery servers returns the same thing.
I tested a few celery settings locally and couldn't get this to work as I'd expect. I used the Google and found places where Ask has said that celery 2.x had timezone issues and to use 3.x. But that'd be a bigger upgrade.
Depends on: 825938
Allen - any ideas on this?
I'm getting this locally too. I was confused why payment retries weren't occuring: 16:34:59 celery:INFO Got task from broker: mkt.purchase.webpay_tasks.purchase_notify[f30e9843-2498-4efd-883f-8a6c5efcf7c1] eta:[2013-01-22 00:35:05.789159-08:00] The eta is 8 hours in the future, which is suspiciously similar to my difference from GMT.
FWIW if you explicitly set an eta, instead of a countdown it seems to work fine. We could use this until a celery upgrade happens.
Whiteboard: [waiting on celery 3]
comment 6 sounds like a good idea. let's do that instead of waiting on celery 3
Whiteboard: [waiting on celery 3]
payment retries sound pretty important so making this a p2
Assignee: nobody → wraithan
Priority: -- → P2
We've been using the workaround in comment 6. This should be fixed in the celery 3 releases but I'd be tempted to just use the workaround for now and close this.
Couldn't find anything that is retrying this way.
Status: NEW → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Target Milestone: --- → 2013-05-23
You need to log in before you can comment on or make changes to this bug.