1387487 - Reduce the gunicorn HTTP timeout (max request duration) from 29 to 20 seconds

Status: RESOLVED FIXED

(Tree Management :: Treeherder: API, enhancement, P1)

Description

[Ed Morley [:emorley]]

Heroku chops off HTTP requests at the routing layer that take more than 30 seconds. It's advantageous to also set a timeout within the app itself, since:
(a) otherwise the gunicorn worker will be still processing the request (even though it now has no where to return the result) causing worker starvation
(b) gunicorn ending the request shows up in New Relic (as a SystemExit exception) unlike Heroku ending the request (which only shows as an H13 error in Papertrail).

Currently we do set a max request duration of 29 seconds (by passing `--timeout 29` to gunicorn), however:
* the 29 seconds is counted from the app's point of view, whereas Heroku's 30 seconds includes the routing time too - which means Heroku is still often reaching its timeout first
* 29 seconds is still a long time for an HTTP request - something like 15 or 20 seconds would reduce the potential for us to suffer from lack of gunicorn workers

Fixing this would help with bug 1386331 in that there is greater New Relic visibility plus the webheads would be more likely to still be able to serve the static UI rather than just returning 503s for everything.

Comment 1

[GitHub Autolander Bot]

Attachment: [treeherder] mozilla:shorter-gunicorn-timeout > mozilla:master

Comment 2

[William Lachance (:wlach)]

Comment on attachment 8893863 [details] [review]
[treeherder] mozilla:shorter-gunicorn-timeout > mozilla:master

wfm

Comment 3

[Treeherder GitHub Bugbot]

Commit pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/33fa97ec842b8892252ac1908e020833e9525248
Bug 1387487 - Reduce gunicorn maximum request time to 20 seconds

To ensure that:
* the Heroku router's 30 second timeout doesn't beat gunicorn to it,
  given that routing time is included in Heroku's timing
* the app is more likely to remain responsive, when receiving many
  badly filtered API calls