1113235 - Add retry logic to all the non-periodic tasks

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect, P2)

Description

[Mauro Doglio [:mdoglio]]

We need to be sure that in case something goes wrong with a non-periodic task, this will be retried a number of times.

Comment 1

[Ed Morley [:emorley]]

It looks like we don't retry some of the pulse publishing tasks, which is problematic given errors we're seeing eg:
https://rpm.newrelic.com/accounts/677903/applications/5585473/traced_errors/5c731e-72bef07a-458c-11e5-97ea-b82a72d22a14/similar_errors?original_error_id=3601293082
https://rpm.newrelic.com/accounts/677903/applications/4180461/traced_errors/52946a-bed0de34-446d-11e5-97ea-b82a72d22a14/similar_errors?original_error_id=52946a-bed0de34-446d-11e5-97ea-b82a72d22a14

Comment 2

[Ed Morley [:emorley]]

Marking blocking bug 1176484, since on Heroku all dynos are restarted every 24 hours, so interrupting a task must not causes data loss.

Comment 3

[Ed Morley [:emorley]]

Most of our tasks retry, for the ones that don't the current situation is no worse than the current stage/prod infra.

Comment 4

[Ed Morley [:emorley]]

This isn't specific to Heroku.

Tasks that are not periodic and don't yet retry, are:

@task(name='publish-job-action')
@task(name='publish-resultset-action')
@task(name='publish-resultset-runnable-job-action')

@task(name='generate-alerts')

Comment 5

[Ed Morley [:emorley]]

Since comment 4, that list has now shrunk to:

@task(name='publish-job-action')

@task(name='generate-alerts')

Comment 6

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/3854

Comment 7

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder

https://github.com/mozilla/treeherder/commit/e0a9fe9bbf5028b51c842d883f033b913475a7a5
Bug 1113235 - Move custom ETL exceptions to their own module

This helps avoid circular import errors in the next commit (via the
`treeherder/workers/task.py` import of `MissingPushException`.

https://github.com/mozilla/treeherder/commit/9678efc63aff807d2faac28e5a91a8d547f51f0a
Bug 1113235 - Make all non-periodic tasks automatically retry

Most non-periodic (ie not schedule via celery beat) were already
automatically retrying, apart from these two.