1453491 - Only run one process per docker container

Status: RESOLVED FIXED

(Localization Infrastructure and Tools :: Automation, enhancement)

Description

[Brian Pitts]

Currently in https://github.com/mozilla-services/elmo-automation we have two containers

1) a10n container: runs both the twisted daemon and the kombu daemon
2) bb container: runs both the l10n-master and the l10n-slave

This has several problems

1) Logs of different processes are interweaved, making them hard to follow
2) If one of the processes crashes (which is happening now for the kombu daemon) docker doesn't restart it
3) We can't use docker's tools for tracking resource utilization to understand the utilization of a single component 

We should have four containers that each run one component, i.e.

1) twisted daemon
2) kombu daemon
3) l10n-master
3) l10n-slave

Comment 1

[Brian Pitts]

Moving image discussion here from 1453502

From Axel: "https://github.com/mozilla-services/elmo-automation/commit/b9461252368e55463b6b856bdb3274664c92983b shows how to split the hg poller and the a10n worker, and does so in dc.

Not sure if that's good enough:

It uses the same image for both, and just uses different commands to run them. The commands can be seen in the docker-compose.yml.

Tested in my local d-c setup with rabbitmq.

Brian, is that good enough for you guys to set this up, or do we actually need to split the images?"

I think that reusing the image but running a different command is okay. Our deployment code supports passing in an arbitrary command to run. Looks like each image requires multiple commands to get started. Could we get two scripts created to take care of them for us? E.G. splitting entrypoint.sh into entrypoint_hg_poller.sh and entrypoint_a10n.sh?

The same approach should work for l10n master and slave.

Sound good to you Miles and Axel?

Comment 2

[Axel Hecht [:Pike]]

Please use my work account.

Comment 3

[Miles Crabill [:miles]]

Using separate commands without splitting images sounds fine to me.

I'm ok either way on the entrypoint side of things (just a bash command or a script) as long as there's a canonical way to run the different apps.

Comment 4

[Axel Hecht [:Pike]]

I'm not eagerly trying to split master and slave yet, I'd prefer to solve that problem once we see it's one.

I've created a follow-up removing the entrypoint.sh (which is misnamed anyway), and moved the hookup stuff to the appropriate places. The commands are now simple commands to bash.

I'm not really all that fond to add wrapper scripts to wrapper scripts, which would be the case for at least scripts/a10n.

Commands are now:

bash -c "/app/a10n/scripts/a10n --env=/app/env hg" (I'll see if I can remove the bash tomorrow)
bash -c 'PYTHONPATH=. /app/env/bin/python /app/env/bin/twistd -n --pidfile "" --umask 0077 get-pushes'

I find that pretty straightforward :-)

Comment 5

[Brian Pitts]

I think we're happy with the state of this now. We have three containers: poller, worker, and buildbot.