637523 - syncserver should never connect to a db marked as down in the adm1 db

Status: VERIFIED WONTFIX

(Cloud Services Graveyard :: Server: Sync, defect, P2)

Description

[Pete Fritchman [:petef]]

If a node is marked as downed in the available_nodes table, we should not try to connect to it's corresponding database instance.

It appears we are making a connection ons server start to establish the connection pooling -- I was able to work around this by removing the impacted nodes from storage.hosts, but ops would prefer one place to tweak whether or not a node is up or down.

Also: we define the "default" storage.sqluri to point @ the first DB, but what if this DB is actually down/having trouble? What should we point this default sqluri to? Can it just be nothing?

Comment 1

[Pete Fritchman [:petef]]

FYI, the same "default" question applies to auth.sqluri.

Comment 2

[Toby Elliott [:telliott]]

We'll need to figure out a way to write availability into memcache, then make sure pooling doesn't pick that one up.

Comment 3

[Ryan Kelly [:rfkelly]]

(In reply to Pete Fritchman [:petef] from comment #0)
> It appears we are making a connection ons server start to establish the
> connection pooling

AFAICT the SQLAlchemy pool class does not connect until you actually try to use the database, and our code should not be doing anything to the database at startup time.  We should check whether this connect-on-startup behaviour is still happening with the latest SQLAlchemy.

(In reply to Toby Elliott [:telliott] from comment #2)
> We'll need to figure out a way to write availability into memcache, then
> make sure pooling doesn't pick that one up.

The syncstorage server has code that checks in memcache for a key "down:<HOST>", and if it exists then a 503-Retry-After is immediately returned without processing the request.  Is this memcache key currently populated by the node downing/upping process?

Comment 4

[:Atoll]

(In reply to Ryan Kelly [:rfkelly] from comment #3)
> The syncstorage server has code that checks in memcache for a key
> "down:<HOST>", and if it exists then a 503-Retry-After is immediately
> returned without processing the request.  Is this memcache key currently
> populated by the node downing/upping process?

No. I think that was part of a long-delayed admin tools update from Toby a year or two ago.

Comment 5

[:Atoll]

(In reply to Pete Fritchman [:petef] from comment #0)
> Also: we define the "default" storage.sqluri to point @ the first DB, but
> what if this DB is actually down/having trouble? What should we point this
> default sqluri to? Can it just be nothing?

The default sqluri is not valid in a sharding setup and should be ignored or disallowed when we're using shards.

Comment 6

[Ryan Kelly [:rfkelly]]

(In reply to Richard Soderberg [:atoll] from comment #5)
> (In reply to Pete Fritchman [:petef] from comment #0)
> > Also: we define the "default" storage.sqluri to point @ the first DB, but
> > what if this DB is actually down/having trouble? What should we point this
> > default sqluri to? Can it just be nothing?
> 
> The default sqluri is not valid in a sharding setup and should be ignored or
> disallowed when we're using shards.

Indeed.  One (ugly) way around it for now would be to set it to some parsable-but-invalid URI.  I filed Bug 785315 with this suggestion.

Comment 7

[Ryan Kelly [:rfkelly]]

This is not relevant to sync1.5 and we've no plans to change anything for sync1.1, so I think we can safely close this out.

Comment 8

[James Bonacci [:jbonacci]]

OK. good.