1289178 - add update or insert to __heartbeat__ check for admin

Status: RESOLVED FIXED

(Release Engineering Graveyard :: Applications: Balrog (backend), defect)

Description

[[e-mail] (:bhearsum)]

During a push to prod today, we hit an issue with the admin box where we got 502 gateway errors when trying to make changes. It turned out that the db credentials were read-only due to some human error. Benson suggested that we add an update or insert to the __heartbeat__ endpoint to make sure this type of error can be caught by monitoring systems.

Seems like it shouldn't be tough to do this. We'd either want some special rule or release that is reserved for this purpose, or perhaps a separate table that isn't part of normal operations.

Comment 1

[Johan Lorenzo [:jlorenzo]]

Attachment: PR

Like discussed over IRC, no permission is checked for the dockerflow table. We actually either insert one (and only one) entry if the table is empty or update the entry.

Tests were added for the Dockerflow DAO class. Existing for the REST endpoints were updated.

The special case for read-only clients is done by giving a handler to create_docker_endpoints. I documented its API to explain the context a little bit more.

Comment 2

[[e-mail] (:bhearsum)]

Comment on attachment 8786323 [details] [review]
PR

r=me with the minor issue noted in the PR fixed.

Comment 3

[[github robot]]

Commit pushed to master at https://github.com/mozilla/balrog

https://github.com/mozilla/balrog/commit/6665c581392046c9cbcd452592bb282be5d5b3e7
Bug 1289178 - add update or insert to __heartbeat__ check for admin (#112)

* Make __heartbeat__ either update or insert the first entry of the Dockerflow table
* However, web keeps counting rule due to read-only access

Comment 4

[[e-mail] (:bhearsum)]

Johan, Benson suggested a small improvement to make it easier for us to verify that things are working:
(In reply to Benson Wong [:mostlygeek] from comment #2)
> The stage dockerflow table has the value: "19". 
> I think it would be ok to have __heartbeat__ simply return the value it
> wrote to the database. We really only monitor on the HTTP response status
> code. Anything other than 200 results in an alert. 

Do you want to do that?

Comment 5

[Johan Lorenzo [:jlorenzo]]

Done in https://github.com/mozilla/balrog/pull/116. I'm not sure if it's worth to open a new bug for that.

Comment 6

[[github robot]]

Commit pushed to master at https://github.com/mozilla/balrog

https://github.com/mozilla/balrog/commit/a09fd8ebd1f9df402966682a634eb3c729a3c72a
bug 1289178: Make __heartbeat__ return the value fetched in database (#116). r=bhearsum