1196597 - Verify Sync 1.1 Client EOL behavior for AP, NODE, REG, and JPAKE traffic.

Status: VERIFIED FIXED

(Cloud Services Graveyard :: Server: Sync, defect)

Description

[Bob Micheletto [:bobm]]

Verify the client behavior when talking the following to AP, NODE, REG, and JPAKE sync endpoints when the response is:

* 503 + backoff
* EOLination
* Nothing

The goal is to determine if additional complexity is required for the EOLinator service in Bug 1194030.  It might be required, for instance, to stand up an NODEinator that assigns the EOLinator as a node to all requesters.

Comment 1

[Edwin Wong [:edwong]]

NI: Karl can you help test here.  Thanks

Comment 2

[Karl Thiessen [:kthiessen, he/him]]

Yes - the request in the etherpad for :bobm to book some time on my calendar was so that we could test it together.

Bob, I'm out this Friday and next Friday, but otherwise my calendar is current.  Please let me know when is convenient for you.

Comment 3

[Ryan Kelly [:rfkelly]]

> * EOLination

I don't think there's much value in getting anything other than the storage nodes to return an EOL-related response.  The logic that checks for EOL messages lives in the handler for "weave:engine:sync:error" in the client, which IIUC is only triggered for errors from the storage node, not other services in the cluster.

IMO we should try "503 + backoff" and "nothing" first, and only if neither of those produces acceptable behaviour should we try to dome something more clever.

Comment 4

[Karl Thiessen [:kthiessen, he/him]]

Attachment: sync.log

Sync log showing variety of 513, 200, and 404 responses.

Comment 5

[Karl Thiessen [:kthiessen, he/him]]

Attachment: Screen shot showing black and yellow bars

Is this the desired behaviour?

Comment 6

[Karl Thiessen [:kthiessen, he/him]]

Changing services.sync.jpake.serverURL to the EOLinator or blank (or to something that doesn't resolve) doesn't change the JPAKE/pairing flow at all, except that it will never succeed.  No indication is given that the jpake.serverURL does not exist.

Comment 7

[Ryan Kelly [:rfkelly]]

> No indication is given that the jpake.serverURL does not exist.

This is a shame, but I don't think there's much we can do about it at this stage.

Comment 8

[Karl Thiessen [:kthiessen, he/him]]

Bob and I worked out a time to get this testing done -- clearing needinfo.

Comment 9

[Bob Micheletto [:bobm]]

Based on the outcome of the tests, closing this bug as fixed.  Karl please set to verified, or re-open if required.

Comment 10

[Karl Thiessen [:kthiessen, he/him]]

I will mark this bug VERIFIED after we throw a bit of load at the mass-produced EOLinator.

Comment 11

[Karl Thiessen [:kthiessen, he/him]]

EOLinators tested in both stage and prod, including expected production load.  We're good to go, folks.