881903 - Production support for Socorro + RabbitMQ

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Socorro, task, P1)

Description

[Selena Deckelmann :selenamarie :selena]

Socorro is transitioning to use RabbitMQ for production.  Socorro's peak supportable load appears to be around 500 crashes/second in the current infrastructure. 

This translates to a theoretical maximum of 500 inserts and 500 removals per second. 

Our more typical load is about 60 inserts/removals per second. 

We have talked a bit about using an existing queue rather than standing up our own special services. 

This ticket is to discuss and come to a decision on what the right infrastructure would be and to make a plan for deploying this in early Q3.

Comment 1

[Jake Maul [:jakem]]

For that kind of load, I'm very tempted to put this on our shared rabbitmq cluster. That's already set up for redundancy, and 500+500/sec is not enough that I'm even a little bit concerned about it. We've tested it much higher than that, and we can always add one or more nodes dedicated to Socorro if need be. This gets us off the ground much faster.

If this turns out to be a problem, it should be relatively easy to switch to a dedicated cluster later on.

Comment 2

[Erik Rose [:erik][:erikrose]]

Sounds sane to me.

Comment 3

[Selena Deckelmann :selenamarie :selena]

(In reply to Jake Maul [:jakem] from comment #1)
> For that kind of load, I'm very tempted to put this on our shared rabbitmq
> cluster. That's already set up for redundancy, and 500+500/sec is not enough
> that I'm even a little bit concerned about it. We've tested it much higher
> than that, and we can always add one or more nodes dedicated to Socorro if
> need be. This gets us off the ground much faster.
> 
> If this turns out to be a problem, it should be relatively easy to switch to
> a dedicated cluster later on.

+1  

Thanks, Jake! I'm about to finish up my integration testing, at which point we'll be ready when you are. :)

Comment 4

[Brandon Burton [:solarce]]

I'll file a bug to get flows for this in place.

:selenamarie, to confirm, processors and collectors need flows right? Admin host too?

Comment 5

[Brandon Burton [:solarce]]

(In reply to Brandon Burton [:solarce] from comment #4)
> I'll file a bug to get flows for this in place.
> 
> :selenamarie, to confirm, processors and collectors need flows right? Admin
> host too?

Answered my own questions by looking at https://bugzilla.mozilla.org/show_bug.cgi?id=878999

Comment 6

[Selena Deckelmann :selenamarie :selena]

Planning to turn this on in production 9/24/13.

Comment 7

[Brandon Burton [:solarce]]

Will outline work to be done in https://etherpad.mozilla.org/881903-socorro-rabbitmq-prod-turnon

Comment 8

[Brandon Burton [:solarce]]

------------------------------------------------------------------------
r75664 | [e-mail] | 2013-09-24 16:30:19 -0700 (Tue, 24 Sep 2013) | 1 line

final socorro prod crashmover and processor configs for rabbitmq, bug 917501
------------------------------------------------------------------------

/prod]
-> % svn ci -m "disabling socorro-monitor in prod now that we're using rabbitmq, bug 917501"
Sending        prod/data-bin/update-socorro.sh
Transmitting file data .
Committed revision 75666.

-> % svn ci -m "disabling socorro-monitor in prod now that we're using rabbitmq, bug 917501"
Sending        trunk/manifests/nodes/socorro.pp
Transmitting file data .

-> % svn ci -m "disabling socorro-monitor in prod now that we're using rabbitmq, bug 917501"
Sending        trunk/manifests/nodes/socorro.pp
Transmitting file data .
Committed revision 75668.
Committed revision 75667.

Comment 9

[Brandon Burton [:solarce]]

-> % svn ci -m "removing socorro monitor check, bug 881903" modules/nagios/
Sending        modules/nagios/manifests/mozilla/services.pp
Transmitting file data .
Committed revision 75744.

Comment 10

[Brandon Burton [:solarce]]

With the removal of all the monitor stuff, we can RF this

Seel also http://www.twobraids.com/2013/09/the-socorro-monitor-rest-in-peace.html