709263 - figure out how to make database operations in balrog more fault tolerant

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[[e-mail] (:bhearsum)]

From https://bugzilla.mozilla.org/show_bug.cgi?id=693593:
Rail raised a good point in that any of the db access methods could raise exceptions. We should figure out how to best deal with those, which doesn't have to necessarily be done here. A few ideas:
- Put the burden on the web side to retry all db operations individually
- Embed retry logic into the core (maybe in AUSTable operations, or maybe in Releases, Rules, etc.)
- Write some magic into a base class for the Views. Eg:
class BaseView:
  def get(self):
    return retry(self._get)

class AView:
  def _get(self):
    db.releases.dosomething()


---

Currently, Balrog does put all of the burden on consumers of the database later to do whatever retrying they do or don't want. If that's what we want to stick with, we should audit the existing code to make sure it's retrying everything it should. If we want to take one of the other routes noted above, we'll need to make deeper changes.

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

found in triage.

Comment 2

[[e-mail] (:bhearsum)]

(In reply to Ben Hearsum [:bhearsum] from comment #0)
> From https://bugzilla.mozilla.org/show_bug.cgi?id=693593:
> Rail raised a good point in that any of the db access methods could raise
> exceptions. We should figure out how to best deal with those, which doesn't
> have to necessarily be done here. A few ideas:
> - Put the burden on the web side to retry all db operations individually

We recently decided that this is the right decision.