Retry tasks that fail due to timeouts and retry tasks in exponential delays

RESOLVED FIXED

Status

Websites
Basket
RESOLVED FIXED
4 years ago
4 years ago

People

(Reporter: mkelly, Assigned: mkelly)

Tracking

Details

(Assignee)

Description

4 years ago
Quoting pmac:

While we're at it, let's make sure that a timeout error (likely socket.timeout) results in a task being retried.

NICE TO HAVE: We currently just retry every X minutes for X times. Would be better if we used an exponential time scale for retries. That is:

1st retry in 1min.
2nd retry in 2min.
3rd retry in 4min.
4th retry in 8min.
5th retry in 16min.
6th retry in 32min.
etc.

Basically every subsequent retry happens in 2x the previous interval. I believe there are celery examples of how to accomplish this. But using this we could retry in this way for a max of 8 times and get a really long retry time w/o bogging down the workers with failed tasks.
(Assignee)

Comment 1

4 years ago
After some IRC discussion we're going to catch any URLErrors coming from tasks.
(Assignee)

Updated

4 years ago
Assignee: nobody → mkelly

Updated

4 years ago
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.