figure out how to make database operations in balrog more fault tolerant

RESOLVED FIXED

Status

Release Engineering
General Automation
P3
normal
RESOLVED FIXED
6 years ago
4 years ago

People

(Reporter: bhearsum, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

6 years ago
From https://bugzilla.mozilla.org/show_bug.cgi?id=693593:
Rail raised a good point in that any of the db access methods could raise exceptions. We should figure out how to best deal with those, which doesn't have to necessarily be done here. A few ideas:
- Put the burden on the web side to retry all db operations individually
- Embed retry logic into the core (maybe in AUSTable operations, or maybe in Releases, Rules, etc.)
- Write some magic into a base class for the Views. Eg:
class BaseView:
  def get(self):
    return retry(self._get)

class AView:
  def _get(self):
    db.releases.dosomething()


---

Currently, Balrog does put all of the burden on consumers of the database later to do whatever retrying they do or don't want. If that's what we want to stick with, we should audit the existing code to make sure it's retrying everything it should. If we want to take one of the other routes noted above, we'll need to make deeper changes.
found in triage.
Component: Release Engineering → Release Engineering: Automation
QA Contact: release → catlee
(Assignee)

Updated

5 years ago
Product: mozilla.org → Release Engineering
(Reporter)

Comment 2

4 years ago
(In reply to Ben Hearsum [:bhearsum] from comment #0)
> From https://bugzilla.mozilla.org/show_bug.cgi?id=693593:
> Rail raised a good point in that any of the db access methods could raise
> exceptions. We should figure out how to best deal with those, which doesn't
> have to necessarily be done here. A few ideas:
> - Put the burden on the web side to retry all db operations individually

We recently decided that this is the right decision.
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.