1047002 - Move all of the subscription machinery to the worker

Status: RESOLVED FIXED

(Websites :: Basket, defect)

Description

[Paul [:pmac] McLanahan]

Stop sending the token in the response to "subscribe" unless the "sync=Y" is sent along with a valid API key. Then when "sync" is not requested, do all of the subscription work in the task. This will greatly speed up subscriptions and allow us to retry more subscriptions if ET returns an error or times out.

While we're at it, let's make sure that a timeout error (likely socket.timeout) results in a task being retried.

NICE TO HAVE: We currently just retry every X minutes for X times. Would be better if we used an exponential time scale for retries. That is:

1st retry in 1min.
2nd retry in 2min.
3rd retry in 4min.
4th retry in 8min.
5th retry in 16min.
6th retry in 32min.
etc.

Basically every subsequent retry happens in 2x the previous interval. I believe there are celery examples of how to accomplish this. But using this we could retry in this way for a max of 8 times and get a really long retry time w/o bogging down the workers with failed tasks.

Comment 1

[Osmose [:osmose, :mkelly]]

The retry logic is covered by bug 1073657 now.

Comment 2

[[github robot]]

Commits pushed to master at https://github.com/mozilla/basket

https://github.com/mozilla/basket/commit/ec2d2ebadd603a515cf1ac1a6854a90a96e99877
Fix bug 1047002: Do not generate token in subscribe view.

Includes rewritten tests for the view and utility function.

https://github.com/mozilla/basket/commit/4ed06808b728bb72e70036f8be65b8361114dfb2
Merge pull request #107 from Osmose/async-subscribe

Fix bug 1047002: Async subscribe