review Logstash Shipper options

RESOLVED FIXED

Status

Infrastructure & Operations
WebOps: Other
P3
normal
RESOLVED FIXED
6 years ago
5 years ago

People

(Reporter: phrawzty, Assigned: phrawzty)

Tracking

Details

(Assignee)

Description

6 years ago
There are a number of ways to transport logs from their sources to the Indexer.  By default the monolithic JAR acts as the Shipper, but frankly, running a bloated Java app on *every server we own* sounds like a terrible idea.  Happily, there are numerous options [1], including a lightweight Shipper [2] by Jordan Sissel himself.

Another avenue (mentioned by :casey) would be to abandon Shippers altogether and use rsyslog [3] as a transport directly to the Indexer.  This would necessitate the implementation of an out-of-scope load balancer and/or buffering mechanism; however, it remains a compelling option as it removes the requirement for additional software to be installed on every server.

[1] http://cookbook.logstash.net/recipes/log-shippers/
[2] https://github.com/jordansissel/lumberjack
[3] http://cookbook.logstash.net/recipes/rsyslog-agent/
(Assignee)

Updated

5 years ago
Blocks: 851116
(Assignee)

Comment 1

5 years ago
Numerous discussions on the mailing list, IRC, and on Vidyo have generated the following salient points :

1. The use of a Broker is generally considered to be a good idea.
2. Of the Shippers that support a Broker, Beaver[1] is likely the most mature.
3. Lumberjack presents an interesting bouquet of features, but has been designed specifically to not use the Shipper; I'm not entirely convinced this is a good idea, but I'd be happy to give it a try.

[1] https://github.com/josegonzalez/beaver
No longer blocks: 829359
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
(Assignee)

Comment 2

5 years ago
Based on a further conversation with jsissel himself :

* As of Logstash 1.1.9, Redis is faster (better publish / consume rates) than RabbitMQ, but *slower* than ZeroMQ or Lumberjack.
* As of Logstash 1.1.10, Redis is *significantly* improved; however, it is unknown when this will become a stable release.

It will be up to us to test and determine whether the reduced speed is worth the benefits of having a broker, or whether those benefits aren't as important as we think and/or we'd rather have the speed of a non-brokered config.
Component: Server Operations: Web Operations → WebOps: Other
Product: mozilla.org → Infrastructure & Operations
You need to log in before you can comment on or make changes to this bug.