If you think a bug might affect users in the 57 release, please set the correct tracking and status flags for Release Management.

Evaluate using docker for pipeline ingestion

NEW
Unassigned

Status

Data Platform and Tools
Pipeline Ingestion
P3
normal
5 months ago
5 months ago

People

(Reporter: whd, Unassigned)

Tracking

Details

(Whiteboard: [SvcOps])

(Reporter)

Description

5 months ago
Operations has standardized on using docker / dockerflow for most applications. We should evaluate whether it makes sense to use docker in our ingestion infrastructure. With end-to-end testing work :mreid and others are doing around docker, moving to it in production would mean our tests run in an environment more similar to production.

The main components of the ingestion infrastructure are kafka, zookeeper, the nginx edge, the DWL, and the CEP. It may make sense to move some or all of these to docker, the evaluation of which is the purpose of this bug.

There are also performance considerations to using docker. In at least one application run by operations, a 15%-20% performance decrease was observed when switching to docker-based deploys.

Updated

5 months ago
Component: Metrics: Pipeline → Pipeline Ingestion
Priority: -- → P3
Product: Cloud Services → Data Platform and Tools
You need to log in before you can comment on or make changes to this bug.