1122967 - Larger payloads (32MB) for telemetry

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Metrics: Pipeline, defect)

Comment 1

[Mark Reid [:mreid]]

The build script in the repo over in bug 1122754 sets the payload size to 8MB (increased from 64KB) -- we can change it there as needed.

See:
https://github.com/mreid-moz/data-pipeline/blob/master/bin/build_pipeline_heka.sh#L20

Comment 2

[Mike Trinkala [:trink]]

fyi: in Heka's default configuration this could consume over 3.2GiB of memory.  If we set it any larger we should reduce the message pool size.

Comment 3

[Mark Reid [:mreid]]

What are the implications of reducing the pool size?

Comment 4

[Mike Trinkala [:trink]]

It all depends on the configuration and data profile but the worst cases:
1) reduced throughput
2) reduced number of messages that the decoder can output in a 1 to N case (hangs if exceeded)