1073657 - Retry tasks that fail due to timeouts and retry tasks in exponential delays

Status: RESOLVED FIXED

(Websites :: Basket, defect)

Description

[Osmose [:osmose, :mkelly]]

Quoting pmac:

While we're at it, let's make sure that a timeout error (likely socket.timeout) results in a task being retried.

NICE TO HAVE: We currently just retry every X minutes for X times. Would be better if we used an exponential time scale for retries. That is:

1st retry in 1min.
2nd retry in 2min.
3rd retry in 4min.
4th retry in 8min.
5th retry in 16min.
6th retry in 32min.
etc.

Basically every subsequent retry happens in 2x the previous interval. I believe there are celery examples of how to accomplish this. But using this we could retry in this way for a max of 8 times and get a really long retry time w/o bogging down the workers with failed tasks.

Comment 1

[Osmose [:osmose, :mkelly]]

After some IRC discussion we're going to catch any URLErrors coming from tasks.

Comment 2

[[github robot]]

Commit pushed to master at https://github.com/mozilla/basket

https://github.com/mozilla/basket/commit/efb1f8f36d6dd0abea9dc2733aa28673e22f5d17
Fix bug 1073657: Retry tasks with increasing delays 8 times.