1853962 - disk manager process dies if db isn't available

Status: RESOLVED FIXED

(Tecken :: General, defect, P2)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

When the Tecken Docker container starts up, it runs tini which runs Honcho. Honcho runs:

/app/bin/run_web_disk_manager.sh

to start the disk manager. That runs:

python manage.py remove_orphaned_files --daemon

If the db isn't available or some other check fails, that returns a non-zero exit code which causes it to drop out of the loop and the disk manager process exits. Honcho sees that and then sends a sigterm to the web process. Once all processes exit, Honcho exits and the Docker container stops. Then the instance starts it up again. Badness ensues. That's what happened in bug #1853745.

We should:

1. add --skip-checks to the python manage.py remove_orphaned_files --daemon line
2. wrap the line in a sentry wrapper so any failures get sent to sentry
3. re-consider the loop since the script is set to die on error so the loop isn't helping

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: [mozilla-services/tecken] bug-1853962: improve disk manager resilience (#2840)

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

willkg merged PR [mozilla-services/tecken]: bug-1853962: improve disk manager resilience (#2840) in eab3500.

This improves the resiliency of the disk manager and also fixes sentry-wrap to work better with Django commands. This should improve Tecken's stability especially in relation to ephemeral issues like being unable to connect to the db.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

I pushed this to prod just now in bug #1867844. Marking as FIXED.