Closed Bug 1353878 Opened 9 years ago Closed 9 years ago

socorro -stage processing 10x crashes after Antenna transition

Categories

(Socorro :: Antenna, task)

Product:
Socorro
Component:
Antenna
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: willkg, Assigned: miles)

Details

Late yesterday, we switched Socorro on -stage to use Antenna instead of the Socorro collector. That went well. However, Antenna and Socorro collector handle incoming crashes that specify the crash id differently. The Socorro collector would apply the throttle rules to incoming crashes even if they specified their crash id. Since Antenna persists the throttle result in the crash id so that Pigeon knows what to do with the crash id, it can't change the crash id. Therefore it lets all crashes that specify the crash id through (i.e. it ACCEPTS them) and doesn't run them through throttling. The consequence of that is that the processor on Socorro -stage is processing around 2x more crashes than it was. That'd be fine except the processors in Socorro -stage can't handle that load as the infrastructure is currently configured. So the crashes backed up in the RabbitMQ socorro.normal queue and then we started getting emails and pages and all that. This bug covers figuring out what to do to fix this.
I have several possible options: 1. We add more processors to the "default" Socorro -stage cluster and move on with our lives (but at some added expense). 2. We could increase the node size of the Socorro -stage processors so that they can churn through more crashes faster. 3. I just fixed Pigeon so that it can throttle crashes. With that change, we could specify the queue for -stage as "10:socorro.normal" and it'll only send 10% of the accepted crashes to be processed. Lonnen, Miles: All three options are easy to do. Which one appeals to you?
Flags: needinfo?(miles)
Flags: needinfo?(chris.lonnen)
I've already scaled up stage. I think that leaving incoming un-double-throttled for this test week is good. After that, we might as well double-throttle.
Flags: needinfo?(miles)
Ok. Given that, I vote we undo the 2x by throttling at 50%. So when we're ready, we can change the PIGEON_QUEUE env var for Pigeon to: 50:socorro.normal That'll continue to send to the socorro.normal queue, but only send half. Miles: I'm going to assign this to you since that's an infra change.
Assignee: nobody → miles
Checking the crash-stats crashes-per-day report shows a 10x increase in crash processing between 4/4/2017 and 4/5/2017--not a 2x increase. I think Datadog showed a 2x increase because we were maxing out the processors' ability to process crashes. Once we added more processors to the cluster, that number went up. Miles is planning to change the PIGEON_QUEUE env var for Pigeon on -stage to: 10:socorro.normal which means it'll send 10% of the accepted crashes to the processor and that'll be close enough to what we had before with the Socorro collector.
Summary: socorro -stage processing 2x crashes after Antenna transition → socorro -stage processing 10x crashes after Antenna transition
Nixing the ni on Lonnen. He replied in email that he thinks sounds good: > I'm sure this is already dealt with, but in case it helps I'm +1 to this plan.
Flags: needinfo?(chris.lonnen)
Pretty sure Miles set it up this way and we're all set here. Marking as FIXED.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Switching Antenna bugs to Antenna component.
Component: General → Antenna
You need to log in before you can comment on or make changes to this bug.