review Logstash Shipper options



Infrastructure & Operations
WebOps: Other
6 years ago
5 years ago


(Reporter: phrawzty, Assigned: phrawzty)





6 years ago
There are a number of ways to transport logs from their sources to the Indexer.  By default the monolithic JAR acts as the Shipper, but frankly, running a bloated Java app on *every server we own* sounds like a terrible idea.  Happily, there are numerous options [1], including a lightweight Shipper [2] by Jordan Sissel himself.

Another avenue (mentioned by :casey) would be to abandon Shippers altogether and use rsyslog [3] as a transport directly to the Indexer.  This would necessitate the implementation of an out-of-scope load balancer and/or buffering mechanism; however, it remains a compelling option as it removes the requirement for additional software to be installed on every server.



5 years ago
Blocks: 851116

Comment 1

5 years ago
Numerous discussions on the mailing list, IRC, and on Vidyo have generated the following salient points :

1. The use of a Broker is generally considered to be a good idea.
2. Of the Shippers that support a Broker, Beaver[1] is likely the most mature.
3. Lumberjack presents an interesting bouquet of features, but has been designed specifically to not use the Shipper; I'm not entirely convinced this is a good idea, but I'd be happy to give it a try.

No longer blocks: 829359
Last Resolved: 5 years ago
Resolution: --- → FIXED

Comment 2

5 years ago
Based on a further conversation with jsissel himself :

* As of Logstash 1.1.9, Redis is faster (better publish / consume rates) than RabbitMQ, but *slower* than ZeroMQ or Lumberjack.
* As of Logstash 1.1.10, Redis is *significantly* improved; however, it is unknown when this will become a stable release.

It will be up to us to test and determine whether the reduced speed is worth the benefits of having a broker, or whether those benefits aren't as important as we think and/or we'd rather have the speed of a non-brokered config.
Component: Server Operations: Web Operations → WebOps: Other
Product: → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.