Closed Bug 1635495 Opened 6 years ago Closed 5 years ago

Refresh production configuration

Categories

(Location :: General, enhancement, P2)

Product:
Location
Component:
General
Type:
enhancement

Tracking

(Not tracked)

Status:
RESOLVED MOVED

People

(Reporter: jwhitlock, Assigned: jwhitlock)

References

(
URL
)

Details

Ensure Ichnaea's production configuration matches current standards. The configuration is spread over several files:

  • docker/run_web.sh
  • ichnaea/webapp/app.py
  • ichnaea/webapp/config.py
  • ichnaea/webapp/gunicorn_settings.py

A lot of Ichnaea's configuration choices were made in 2014, without much supporting documentation for decisions. These may have been the correct decisions, but I'd like to explore some areas:

  • Why was a customized gevent worker used, and is it still needed?
  • Why is the application initialized in a post_worker_event handler?
  • Are the manual shutdown steps in shutdown_worker needed?
  • Can gunicorn be used in development as well?
  • Should we continue to disable HTTP 1.1 / Keep-Alive requests in the app?

As a parallel effort, ensure the docs are up to date (bug 1628047), as well as adding code comments where needed.

No concrete answers, but some hunches from tracing through this code and git history:

  • The customized gevent worker adds a per-request timeout, which should kick in before the standard gunicorn method for detecting stuck workers
  • One feature of the global _APP, as well and the initialization and shutdown tied to gunicorn worker lifetime, is to ensure that the connections are closed and released for the database, cache, etc. on an unexpected worker shutdown.
  • It looks like the nginx layer could cache and collect HTTP 1.1 requests, leaving gunicorn to treat them like simpler 1.0 requests. This appears to be the suggested configuration in Deploying Gunicorn.

This all leads me to suspect that there were issues in 2014 with request terminations. This could be from clients with long or broken HTTP 1.1 requests, or from occasionally failing backend connections. I'll continue to look at code and issues to see if there is any further data, and add them as I find them.

At the same time, 2020 is different from 2014 - different clients, difference user base, different OS, different Python, different server environments, different code, etc. I want to slowly remove these customizations, and see if the problems return.

Assignee: nobody → jwhitlock
Status: NEW → ASSIGNED

Moved to https://github.com/mozilla/ichnaea/issues/1386

URL: https://github.com/mozilla/ichnaea/is...
Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Resolution: --- → MOVED
You need to log in before you can comment on or make changes to this bug.