There are a number of ways to transport logs from their sources to the Indexer. By default the monolithic JAR acts as the Shipper, but frankly, running a bloated Java app on *every server we own* sounds like a terrible idea. Happily, there are numerous options , including a lightweight Shipper  by Jordan Sissel himself. Another avenue (mentioned by :casey) would be to abandon Shippers altogether and use rsyslog  as a transport directly to the Indexer. This would necessitate the implementation of an out-of-scope load balancer and/or buffering mechanism; however, it remains a compelling option as it removes the requirement for additional software to be installed on every server.  http://cookbook.logstash.net/recipes/log-shippers/  https://github.com/jordansissel/lumberjack  http://cookbook.logstash.net/recipes/rsyslog-agent/
Numerous discussions on the mailing list, IRC, and on Vidyo have generated the following salient points : 1. The use of a Broker is generally considered to be a good idea. 2. Of the Shippers that support a Broker, Beaver is likely the most mature. 3. Lumberjack presents an interesting bouquet of features, but has been designed specifically to not use the Shipper; I'm not entirely convinced this is a good idea, but I'd be happy to give it a try.  https://github.com/josegonzalez/beaver
No longer blocks: 829359
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
Based on a further conversation with jsissel himself : * As of Logstash 1.1.9, Redis is faster (better publish / consume rates) than RabbitMQ, but *slower* than ZeroMQ or Lumberjack. * As of Logstash 1.1.10, Redis is *significantly* improved; however, it is unknown when this will become a stable release. It will be up to us to test and determine whether the reduced speed is worth the benefits of having a broker, or whether those benefits aren't as important as we think and/or we'd rather have the speed of a non-brokered config.
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.