550535 - Sisyphus - fail to reconnect to database

Status: RESOLVED FIXED

(Testing Graveyard :: Sisyphus, defect)

Description

[Bob Clary [:bc] (inactive)]

Attachment: patch

Currently, database related function calls attempt to filter connection related exceptions by looking for conn_request in the exception text. This can result in failures to recover from database connection errors when different exceptions are thrown. Also, sisyphus.couchdb.connectToDatabase() does not terminate until a connection is reestablished. This can cause problems if the couchdb call enters a permanent failure mode independent of the actual database availability.

This patch eliminates the exception check for conn_request and the limits the number of attempts that connectToDatabase() will attempt to reconnect before failing.

Comment 1

[Bob Clary [:bc] (inactive)]

Attachment: patch v2

I missed some 'conn_request' conditionals in the worker.

Comment 2

[cmtalbert]

The only issue I see here (and I can't really see it in the diff view) is that removing some of these "raise" statements might have some unintended consequences, so just be sure this is well tested.  I think it is a good approach though, overall to do the connection retries this way instead of guessing at "which error should I reconnect on"

Comment 3

[Bob Clary [:bc] (inactive)]

I'll double check tonight. Definitely the CouchDBExceptions such as update or delete conflicts should be handled properly and not temporarily ignored. The right way to do this is to audit couchquery and httplib2 for all connection related exceptions and only retry for them. Or I could check for /couchquery/ or /httplib2/ in the exceptionValue. That might be the preferred method. See http://qm-crash-linux01:5984/crashtest/_design/default/_view/logs?startkey=%222010-03-08T08%22 for this list of exceptions since I pushed these recent changes to the server. Let me redo this with the path checks and see how that behaves.

Comment 4

[Bob Clary [:bc] (inactive)]

Attachment: patch v3

Ok, I think this is more robust.

Comment 5

[cmtalbert]

Comment on attachment 431367 [details] [diff] [review]
patch v3

I think so too.

Comment 6

[Bob Clary [:bc] (inactive)]

http://hg.mozilla.org/qa/sisyphus/rev/f3960cfcee3a

Comment 7

[Bob Clary [:bc] (inactive)]

Attachment: patch to recreate the httplib2 Http object on error.

When a connection related error occurs, recreate the httplib2 Http object.

Comment 8

[Bob Clary [:bc] (inactive)]

http://hg.mozilla.org/automation/sisyphus/rev/f69f54fd7c43

Comment 9

[Bob Clary [:bc] (inactive)]

This doesn't quite work when the scripts restart and the last logMessage can not recover. The fix is to recreate the couchquery Database object when the database connection fails.

Comment 10

[Bob Clary [:bc] (inactive)]

Attachment: patch ultimo

I tested this by running several workers and killing the couchdb server then restarting it after a delay. They do reconnect fine.

Comment 11

[Bob Clary [:bc] (inactive)]

http://hg.mozilla.org/automation/sisyphus/rev/cfb5f44ca9b1