856417 - create crash storage system for RabbitMQ

Status: RESOLVED FIXED

(Socorro :: Backend, task)

Description

[K Lars Lohn [:lars] [:klohn]]

It's time to take the next step into the modern world and start eliminating the "ghetto queues" in Socorro.  This first queuing system implementation will be RabbitMQ.

We need two different implementations of the CrashStorage API for RabbitMQ and one new meta-queue type:

1) the first one just saves and serves `crash_ids`.  It will implement only the `save_raw_crash`, `save_processed_crash` and `new_crashes` methods, leaving all the others to their defaults.  
1.1) `save_raw_crash` - take the `crash_id` from the raw crash and push it onto the associated RabbitMQ queue
1,2) `save_processed_crash` - take the `crash_id` from the processed crash and push it onto the associated RabbitMQ queue
1.3) `new_crashes` - fetch a single `crash_id` from the associated RabbitMQ queue.  It is import that this be a non-blocking call.  If there is no `crash_id` available, this method should yield `None`

2) This implementation works with the entire crash rather than the just the `crash_id`.
2.1) `save_raw_crash` - pushes both the raw crash and the binary dumps onto the associated RabbitMQ queue.  To bind the crash components together as one object, they should be placed in a mapping with two keys: 'raw_crashes' and 'dumps'. 
2.2) `new_crashes` - this method should just yield the next 'raw_crash/dumps' combo frome the assocaited queue.  

3) The new meta-queue type should be implemented within .../external/crashstorage_base.py and be called 'PriorityQueue'.  As configman config settings, it should require two additonal CrashStorage instances called (I'm not married to these names): 'normal' and 'priority'.  Use the implementation of `FallbackCrashStorage` as a template.  The _only_ method that this class implements is `new_crashes`
3.1) `new_crashes` - yield items from the 'priority' crashstorage (using its `new_crashes` method), until it yields 'None'.  Then try to yield a single job from the 'normal' crashstorage.  Lather, rinse, repeat.  


This whole job should be relatively simple.  It appears that everything needed to implement this is within the RabbitMQ tutorial #2 at http://www.rabbitmq.com/tutorials/tutorial-two-python.html   When declaring a queue, the queue name should be an Option under control of Configman.  The queues themselves should be 'durable' and use 'fair dispatch'.  The implementations should use 'message acknowledgment'.

Comment 1

[Selena Deckelmann :selenamarie :selena]

See bug 883414