1127940 - submit_bug_comment "ObjectNotFoundException((), {'bug_id': 1096324, 'job_id': 6103034})" due to read host DB race

Status: RESOLVED FIXED

(Tree Management :: Treeherder, defect, P2)

Description

[Ed Morley [:emorley]]

Started after the prod deploy of the read DB host fix.

This is likely just another case where we should use the master DB instead (and the read host fix just exposed the existing race condition):

https://rpm.newrelic.com/accounts/677903/applications/4180461/traced_errors/3000207132?original_error_id=3000830047

Traceback (most recent call last):
File "/usr/lib64/python2.6/site-packages/newrelic-2.40.0.34/newrelic/hooks/application_celery.py", line 66, in wrapper return wrapped(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/celery/app/trace.py", line 437, in __protected_call__ return self.run(*args, **kwargs)
File "/data/www/treeherder.mozilla.org/treeherder-service/treeherder/etl/tasks/tbpl_tasks.py", line 44, in submit_bug_comment submit_bug_comment.retry(exc=e, countdown=(1 + submit_bug_comment.request.retries) * 60)
File "/usr/lib/python2.6/site-packages/celery/app/task.py", line 680, in retry raise ret
Retry: Retry in 60s: ObjectNotFoundException((), {'bug_id': 1096324, 'job_id': 6103034})

I believe it comes from:
https://github.com/mozilla/treeherder-service/blob/cae85ae8c4cc5cb9fdecd86ffbdcf2367782b4d6/treeherder/etl/tbpl.py#L114

            bug_job_map = jm.get_bug_job_map_detail(self.job_id, self.bug_id)

Comment 1

[Ed Morley [:emorley]]

Attachment: Simplify BugzillaBugRequest username handling to avoid host DB race

Comment 2

[Treeherder GitHub Bugbot]

Commits pushed to master at https://github.com/mozilla/treeherder-service

https://github.com/mozilla/treeherder-service/commit/443706689202db51b4023ee43257993394f867b9
Bug 1127940 - Pass 'who' directly to BugzillaBugRequest to avoid DB race

We did not previously pass the username of the person who made a failure
classification to BugzillaBugRequest. As a result, BugzillaBugRequest
has to fetch the bug mapping again to find it out. After the fix for our
master-read DB setup, this fetch of the bug mapping was being performed
against the read host, which occasionally did not yet have the requested
mapping (due to replication delay) causing an ObjectNotFoundException.

The 'get_bug_job_map_detail' query could have been switched to use the
master host to avoid the race, however the more efficient fix is just to
pass the username to BugzillaBugRequest directly, like we already do for
OrangeFactorBugRequest, avoiding the query entirely.

https://github.com/mozilla/treeherder-service/commit/e61738b5a9db338b10e296cd6474786a7c93d2e2
Bug 1127940 - Remove the now unused get_bug_job_map_detail()

BugzillaBugRequest was the only consumer, so we can now remove it.