623600 - 503 on all backend failures

Status: RESOLVED FIXED

(Cloud Services Graveyard :: Server: Sync, defect)

Description

[Tarek Ziadé (:tarek)]

Make sure :

* the auth server
** returns 503 on ldap failures
** get back on track when the ldap server is back

* the storage server
** returns 503 on ldap, sql and memcached failures
** get back on track when the ldap or sql server is back

Comment 1

[Tarek Ziadé (:tarek)]

for sql/memcached
* in core: http://hg.mozilla.org/services/server-core/rev/84104a934dd4
* in storage: http://hg.mozilla.org/services/server-storage/rev/321ee6560a64

Comment 2

[Tarek Ziadé (:tarek)]

Attachment: retry-after+503 for LDAP

This patch makes sure that:
- add_s and modified_s are also re-tried when a SERVER_DOWN error occured. python-ldap's ReconnectLDAP class did not cover those for some reason. The test is not easy to reproduce in the unittest: it consists or relaunching the ldap server and make sure existing connectors are still running correctly.

- SERVER_DOWN and OTHER are now caught on every failure like TIMEOUT

- a new config was added: global.retry_after. Defaults to 1800.

Comment 3

[Tarek Ziadé (:tarek)]

Attachment: Doc for teh retry-after option

Goes with patch 503766

Comment 4

[Toby Elliott [:telliott]]

Comment on attachment 503766 [details] [diff] [review]
retry-after+503 for LDAP

Looks good. I worry a little that 10 retries is too many - you can clog up a lot of pipes if you're waiting 10s for auth - but that's a separate issue.

Comment 5

[Tarek Ziadé (:tarek)]

(In reply to comment #4)
> Comment on attachment 503766 [details] [diff] [review]
> retry-after+503 for LDAP
> 
> Looks good. I worry a little that 10 retries is too many - you can clog up a
> lot of pipes if you're waiting 10s for auth - but that's a separate issue.

Right, those are aggressive defaults, I'll set the retries to 3 attempts, and the delay between two retries to 100 ms.

Comment 6

[Tarek Ziadé (:tarek)]

Code commited in http://hg.mozilla.org/services/server-core/rev/e164ac00dd81

Comment 7

[Tarek Ziadé (:tarek)]

Doc commited. Preview: http://sync.ziade.org/doc/configuration.html#global