1179125 - Update pipeline heka pool size

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P2)

Description

[Wesley Dawson [:whd]]

On the CEP/DWL nodes currently a decode injects two messages (regular telemetry and executive summary). This is changing in the future (forthcoming bug), but we should bring plugin_chansize down as per the following scrollback:

[10:08:10] <trink> it looks like the input exhausted the message pool
[10:11:36] <trink> unclear why... technically the cep should have hit the same issue
[10:12:41] <trink> the dwl wedged immediately after restart yesterday
[...]
[10:21:27] <trink> it looks like the decoder was slow the input consumed all the packs (queuing more messages to decode) and now the decoder is blocked because it needs more packs to continue
[10:24:06] <trink> https://github.com/mozilla-services/heka/issues/867
[10:28:04] <trink> whd: in this case we would have to bring plugin_chansize down (like around 10)
[10:29:05] <mreid> trink: so just luck that we didn't jam up the CEP too?
[10:29:33] <trink> it is 100% timing
[10:29:47] <mreid> and it wouldn't necessarily matter if it was a multidecoder or single?
[10:30:29] <trink> what matters now is we are generating 2 messages for each 1 in
[10:30:47] <trink> that is why it now hangs
[10:31:02] <mreid> ok
[10:31:15] <mreid> so doing it as a filter instead of a decoder would be better
[10:33:44] <trink> from a won't hang perspective... instead it will jam up the router but only slowing it down (not hanging it)
[10:34:14] <mreid> k.
[10:35:53] <trink> in this config lowering the channel to 10 should have at most 88 packs in play at once

Comment 1

[Wesley Dawson [:whd]]

https://github.com/mozilla-services/puppet-config/pull/1421