Closed Bug 875369 Opened 13 years ago Closed 12 years ago

[Basket] Error logging and resend errored calls function

Categories

(www.mozilla.org :: Newsletters, defect)

Product:
www.mozilla.org
Component:
Newsletters
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: jdavis, Assigned: dpoirier)

Details

(Whiteboard: [kb=1118973] )

The way basket currently works when a call to Exact Target fails: * The retry is every 5 min for 6 times, or 30 min total (give or take) * Sentry (error logging tool) logs errors, but only keeps details of one example error, and then tells you how many times that error was experienced. It's great for seeing issue trends, not so much for resending errored jobs Improvements: Error logging tool should be upgraded to log errors that do not complete after the 30 min trial so that if/when something happens, we can manually resend the calls.
I'll take a look at this. We might also make the retry logic smarter. For example, instead of retrying every five minutes for 30 minutes, we might retry the first time after one minute, double the interval each time it fails, to a maximum interval of say four hours, and not give up for three days. That would cope with both short and longer outages, without beating on the ET server too hard as retrying tasks pile up.
Assignee: nobody → dpoirier
Status: NEW → ASSIGNED
OS: Mac OS X → All
Hardware: x86 → All
Improved logging: https://github.com/mozilla/basket/pull/74 I've played with changing the retry logic to binary exponential backoff, but want to get other folks' opinions on whether that's worth the added complexity. (Celery doesn't support it directly, so it takes some manual fiddling.)
Commits pushed to master at https://github.com/mozilla/basket https://github.com/mozilla/basket/commit/1089f07fb9ff9518ce8c4d87d24877676b4abec6 Bug 875369 - Better logging of basket task failures When a basket task fails, log the full arguments, in case we need to try to run it again after the problem is fixed. Also try to log fewer messages, and make the ones we log more similar to give Sentry a better change to aggregate the related ones. https://github.com/mozilla/basket/commit/32b7874a3f185a80f99427c6ac768affcb77f7d8 Merge pull request #74 from dpoirier/bug-875369-basket-task-failure-logging Bug 875369 - Better logging of basket task failures
The push above improved the information that is logged in Sentry. However, Sentry doesn't save all the logged messages, it samples. So we still need a persistent, reliable store of failed tasks. Celery supports storing the results of tasks, including full arguments and failure information. We're going to look into configuring that on our servers, or locating where that information is going.
Commits pushed to master at https://github.com/mozilla/basket https://github.com/mozilla/basket/commit/ca55849a3d01309a0094afda4d748bc4b7fbb0f1 Bug 875369 - Log failed Basket tasks When a basket task is unable to complete (after exhausting its allowed retries), log the task name, parameters, and whatever exception made it fail into a database table where we can look at it later. https://github.com/mozilla/basket/commit/3b8949fc3e9578d55ed83313a9b6f7fad6ce217b Merge pull request #79 from dpoirier/bug-875369-log-failed-tasks Bug 875369 - Log failed Basket tasks
What we ended up doing was creating a new "Failed Tasks" table and logging all the task info to it when a task exhausted its retries. For now, we can browse the failed tasks in the admin, but have not yet implemented a way to re-run those tasks. How about if we wrote a management command that would requeue all the tasks that failed between two timestamps? After successfully requeueing a task, it would delete the failed task record so we wouldn't queue it again. If the requeued task failed, it'd be added to the failed task table again in the same way.
This makes sense to me - what do you think pmac? My only gut check is that if a call repeatedly fails, we probably shouldn't keep sending it after a certain timeframe - and we should look into the particular reason why it's been failing repeatedly to fix it. Reasoning: It wouldn't be ideal if someone's subscription call failed repeatedly for days, and then finally succeeded and sent the welcome email 3+days to a week or even month later. If a call does fail repeatedly over the course of a day, we should fix it. Either if it's a timeout issue on Exact Target's end, or something in the code.
Management commands are run manually, so once the initial round of retries was exhausted, we'd only try the task again after looking at it in the admin and deciding it was appropriate to retry it.
And on reflection, if we're going to be looking at the failed tasks in the admin, we could write an admin action to requeue them from the admin action menu, after selecting the ones we thought were worth retrying.
Sounds like a plan - I like the idea of viewing in the admin and being able to manually resend if we need to. This could also help with troubleshooting if we manually send a call and it fails again. Great idea.
Whiteboard: [kb=1118973]
We'll want to test this out, but the new code is only triggered manually from the admin, so should be safe to push to prod anytime.
Sweet. Should we give it a go before it goes to prod? If so, where can we find it in the admin on dev?
Hey Dan, I get a 403 Forbidden when I go to the link above (even after signing in).
Jess, I think I have now given your user the necessary permissions. Please let me know if it works now or not.
Neat! It looks great! (I really do need to get these localized welcome emails up and running...or we'll be full-o-errors in here - like we currently are in sentry. I can just see them now.)
Dan, can you give my user permissions on prod's basket?
Jess, I don't think I have an account on basket prod, but pmac should be able to do it for you.
Flags: needinfo?(pmac)
Done.
Flags: needinfo?(pmac)
Hmm... I do... Basket.mozilla.org...unless I didn't add hu to the prod basket I thought I had access to.
I see it! And it's beeeautiful. I haven't tried to resend anything yet, but it looks pretty. I'll set up one of the welcome emails that doesn't exist currently, trigger the error, and resend the call to make sure it's fully functional. Then I'll mark this as verified. For now, great work. It's nice to see what's happening!
Status: ASSIGNED → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.