improve reprocessing so it's less likely reprocessing doesn't degrade standard processing
Categories
(Socorro :: Antenna, enhancement, P2)
Tracking
(Not tracked)
People
(Reporter: willkg, Assigned: willkg)
Details
Attachments
(1 file)
Currently, the processor pulls from three different queues to assemble a list of crash ids to process:
It pulls at most pull_max_messages from standard queue, then reprocessing queue, then priority queue and yields them one at a time. Let's ignore the priority queue for now since that rarely has any crash ids in it. Given that, if the standard and reprocessing queues have a lot of crash ids, the processor will pull from them equally and process accordingly.
The problem this creates is that if something dumps a gazillion crash ids in the reprocessing queue, the processor will process the standard and reprocessing queues equally which will result in the standard queue backing up, crash report processing falling behind crash reports ingested, and us getting alerted (this is a service degradation situation).
In the past, we "alleviated" this situation by adding rate-limiting to the reprocessing API and a sleep to the reprocess crashstats-tools command. The thinking is that if you were going to reprocess a gazillion crash ids, you would use a long sleep so that you could trickle the crash ids in rather than dumping them in all at once. This isn't great and for automated systems, this causes their runtime to exceed their max runtime and get terminated.
Instead of doing that, what we should do a few things:
- change the processor to prefer standard queue over the reprocessing queue; that way people can toss a bazillion items into the reprocessing queue and the processor will get to them eventually--whatevs
- change our alerting to only alert when the standard queue is backed up
- change the reprocess crashstats-tools command to have a smaller sleep argument
This bug covers that work.
|
Assignee | |
Updated•1 year ago
|
|
|
Assignee | |
Comment 1•1 year ago
|
||
|
|
Assignee | |
Comment 2•1 year ago
|
||
|
|
Assignee | |
Comment 3•1 year ago
|
||
After the stage deploy, I'll set up a reprocessing job on stage and then see how it does.
|
|
Assignee | |
Comment 4•1 year ago
|
||
The change deployed to stage. I reprocessed 1,000 crash reports and can see the before and after. The processor is still processing items from the reprocessing queue, but at a much lower rate than the standard queue. The standard queue isn't backing up at all.
Since stage only has one processor node, it's going to take a long while to get through the reprocessing queue, but that's ok.
|
||
Comment 5•1 year ago
|
||
This was deployed to prod in https://github.com/mozilla-services/socorro/releases/tag/v2025.03.27.
I'l submit some crash reports for reprocessing in prod now to see how the change behaves under production load.
|
|
Assignee | |
Comment 6•1 year ago
|
||
I tested this in stage and it was fine there. I was thinking that since we regularly get bursts of reprocessing from the symbol scrapers, we could wait and see how this fares when the next one of those happens and we can adjust the ratio accordingly.
|
|
||
Comment 7•1 year ago
|
||
:willkg I had already submitted 1,000 crash reports for reprocessing when I saw your message. The processor got through those in about 18 minutes, and the standard queue wasn't affected at all. I think we are good here and can close this bug.
|
|
Assignee | |
Comment 8•1 year ago
|
||
Awesome! The days of getting alerted because of reprocessing bursts are over! Thank you, Sven!
|
|
||
Comment 9•1 year ago
|
||
I actually think the current alert uses the maximum message age across all queues, so it will still alert if messages on the reprocessing queue are too old. I'll go ahead and exclude the reprocessing queue from the alert, so it will only include standard and priority queue.
|
|
||
Comment 10•1 year ago
|
||
I was wrong – we are only alerting on the standard queue, so all good.
Description
•