1357263 - Evaluate using docker for pipeline ingestion

Status: RESOLVED FIXED

(Data Platform and Tools :: General, enhancement, P3)

Description

[Wesley Dawson [:whd]]

Operations has standardized on using docker / dockerflow for most applications. We should evaluate whether it makes sense to use docker in our ingestion infrastructure. With end-to-end testing work :mreid and others are doing around docker, moving to it in production would mean our tests run in an environment more similar to production.

The main components of the ingestion infrastructure are kafka, zookeeper, the nginx edge, the DWL, and the CEP. It may make sense to move some or all of these to docker, the evaluation of which is the purpose of this bug.

There are also performance considerations to using docker. In at least one application run by operations, a 15%-20% performance decrease was observed when switching to docker-based deploys.

Comment 1

[Wesley Dawson [:whd]]

This has been evaluated. The new infrastructure will use it, but we're leaving the current infra as-is.