822317 - treestatus_mozilla_org should handle failover gracefully

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Sheeri Cabral [:sheeri]]

This morning, we switched over the database backend behind treestatus_mozilla_org (basically, simulated a failover so we could do maintenance on the database). 

treestatus_mozilla_org was giving errors like:
	 [Mon Dec 17 09:02:49 2012] [error] StatementError: Can't reconnect until invalid transaction is rolled back (original cause: InvalidRequestError: Can't reconnect until invalid transaction is rolled back) 'SELECT trees.tree AS trees_tree, trees.status AS trees_status, trees.reason AS trees_reason \\nFROM trees \\nWHERE trees.tree = %s' [{u'%(139912741355152 param)s': u'mozilla-inbound'}]

restarting the treestatus app seems to fix the problem....but if there is a failover in general, treestatus handle it gracefully.

Comment 1

[Ed Morley [:emorley]]

I don't see anything in here that looks sensitive from my POV; any objections to me unhiding?

Comment 2

[Sheeri Cabral [:sheeri]]

I wasn't sure if it's a security problem to let the world know that treestatus doesn't handle failover gracefully. Feel free to unhide if that's not a problem.

Comment 3

[Ed Morley [:emorley]]

Ah I see. Thank you :-)

Comment 4

[Chris AtLee [:catlee]]

I think I fixed it:
https://github.com/catlee/treestatus/commit/6d3cb3e4a8333589a635f617ccca4dc1743ba13a#L0L601

It's live on https://treestatus-dev.allizom.org/. We should try kicking the DB and see if the app recovers properly.

Comment 5

[Sheeri Cabral [:sheeri]]

Agreed! Let's coordinate for sometime this week.

Comment 6

[Chris AtLee [:catlee]]

sheeri and I tested this today, and the app recovers properly when it loses its connection to the db.

However, I wasn't able to verify this by looking at the application logs - I was expecting to see some messages about retrying the request due to the DB failure, but I don't see any app-specific messages in the logs.

Comment 7

[Sheeri Cabral [:sheeri]]

We did a failover today and things worked, so the only thing left is to find the app logs....I'm un-cc'ing myself from this bug. Woo hoo for graceful failover!

Comment 8

[Ed Morley [:emorley]]

Not sure about the logs, but that's an orthogonal problem now - morphing back to the original issue and marking fixed.

Thank you for confirming :-)