1387536 - Figure out why seta-analyze-failures is hitting the hard not soft celery time limit

Status: RESOLVED INCOMPLETE

(Tree Management Graveyard :: Treeherder: SETA, enhancement, P3)

Description

[Ed Morley [:emorley]]

Celery has two time limits: hard and soft. The latter is preferred, since it still results in something showing up in New Relic.

As such, for all our tasks we've aimed to have the soft time limit be shorter than the hard time limit to ensure the task still gets to report to New Relic.

However I've just noticed on papertrail:

"""
Aug 04 01:15:30 treeherder-prod app/worker_default.2: [2017-08-04 00:15:30,097: ERROR/MainProcess] Task seta-analyze-failures[b002a923-2102-4e9e-ac2e-4e5b09463b5b] raised unexpected: TimeLimitExceeded(930,) 
Aug 04 01:15:30 treeherder-prod app/worker_default.2: Traceback (most recent call last): 
Aug 04 01:15:30 treeherder-prod app/worker_default.2:   File "/app/.heroku/python/lib/python2.7/site-packages/billiard/pool.py", line 645, in on_hard_timeout 
Aug 04 01:15:30 treeherder-prod app/worker_default.2:     raise TimeLimitExceeded(job._timeout) 
Aug 04 01:15:30 treeherder-prod app/worker_default.2: TimeLimitExceeded: TimeLimitExceeded(930,) 
Aug 04 01:15:30 treeherder-prod app/worker_default.2: [2017-08-04 00:15:30,098: ERROR/MainProcess] Hard time limit (930s) exceeded for seta-analyze-failures[b002a923-2102-4e9e-ac2e-4e5b09463b5b] 
"""

It looks like the soft limit isn't working for some reason.