In many cases when modifying a plugin in the data-pipeline repo a corresponding PR needs to be opened in puppet-config. These PR's must be deployed/rolled back together (across systems) but there is no easy way to coordinate this and it quickly becomes error prone. Investigate ways to improve this. e.g. one suggestion was to deploy the config and business logic with/on the system running the service and that information would be propagated to Heka. This is very host oriented which introduces its own issues such as upgraded/downgrades, deprecation, aggregation, security issues if we allow new input and output to be turned up etc.
If we can add to the rpm generated by the data-pipeline repo build script some config files (not in /etc/heka.d/ but somewhere where they can be copied to /etc/heka.d) we can use puppet to copy the configs over into /etc/heka.d based on node type when the new RPM is installed, which might be what you're looking for. That way, the only change that needs to happen in puppet-config is the RPM version bump. We would lose the ability to template based on host profile at runtime (e.g. maxprocs settings, looping over kafka partitions) unless we make the configs ERB templates, but getting those into puppet would be a more difficult business.
Component: Metrics: Pipeline → Pipeline Ingestion
Product: Cloud Services → Data Platform and Tools
Our process has improved lately, and probably doesn't need additional effort.
Status: NEW → RESOLVED
Last Resolved: 9 months ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.