907610 - [bunker] Bunker is not transporting logs to the ES indexer nodes

Status: RESOLVED FIXED

(Infrastructure & Operations :: IT-Managed Tools, task, P3)

Description

[Eric Zounes [:zounese]]

After a day of debugging this issue, I think I may be close to figuring this out.

Here are the symptoms:
  * Logs are shipped to logstash and structured. They are not send to ES indexer nodes.
  * If logstash is restarted, logs will be sent for a brief period of time. (Roughly 5 seconds)
  * I see the following discovery error in the indexer logs:
              
[2013-08-21 07:12:14,493][WARN ][discovery.zen.ping.multicast] [node55_scl3] failed to receive confirmation on sent ping response to [[Fetter, Philip][lRrmOKpmQOipV0EQ_mO2BA][inet[/10.22.78.113:9300]]{client=true, data=false}]

Current conclusion:
  * For some reason the logstash ES clients start out with the correct indexer information, but the discovery protocol fails for some reason. I don't know what changed in ES between 2013/08/20 and 2013/08/21, but logs stopped being indexed at around the same time i pushed an update to the indexers. I've reverted all the changes such that all configs are in their original state before this issue.

Comment 1

[Daniel Maher [:phrawzty]]

Ugh.  I really don't know what the deal is.  Perhaps now would be a good time to just go ahead and re-start the interface nodes (or, indeed, the entire cluster).

Comment 2

[Daniel Maher [:phrawzty]]

I understand that this has been fixed ?  Could you please provide a quick summary of the analysis and solution ?  Thank you.

Comment 3

[Eric Zounes [:zounese]]

I'm fairly certain that this issue was actually caused by removing fields from the default mapping. Here is the sequence of events that might have led to it breaking:

1. The @type field was removed instead of @fields.type
2. The previous mapping dictates that @type should exist. Fields can't be removed from a mapping.
3. Logs stopped being indexed. A bug in Logstash may have caused the logs to be indexed for a brief period of time due to the "drop @type" filter not being applied immediately.

Note:

Document fields can't be dropped once they're added. However, fields can be added at any time (given dynamic mapping)
Thus, if a change is made which changes the structure of a document by removing a field, it will break the mapping.

Solution:

This issue was resolved by upgrading the ES cluster. I experienced an issue with the same symptoms on 09/03. I resolved it by removing the index. Any time we could make a change that involves modifying an existing type mapping we should wait until we have a fresh index to reduce the amount of log data we lose.