784890 - Idea for cooperative session-level locking in sync2.0

Status: RESOLVED WONTFIX

(Cloud Services :: General, enhancement)

Description

[Ryan Kelly [:rfkelly]]

I've been thinking about the idea of session-level locking for Sync2.0, and whether we can do it within the scope of the current protocol.  Just want to throw this out there as a suggestion.  If nothing else, it will be interesting explore the shortcomings of this simplistic approach.

Idea: Use the presence of a special WBO record as a simple exclusive lock.  If you're able to create the WBO then you get the lock.  Delete it when you're done.  Set an expiry time on it so that the lock doesn't get held indefinitely.

This is basically how people advise to do locking with memcache.

To take the lock:   PUT /2.0/userid/storage/meta/lock
                    X-If-Unmodified-Since: 0
                    {"payload": "", "ttl": 3600}

If you get a "201 Created" response, congratulations you got the lock.  If you get a "412 Precondition Failed" response then someone else currently holds the lock, try again soon.

Problems with this approach:

* There's no way to wait for the lock, and no indication of how long it might be held for.  Clients would just have to poll in a loop until they acquire it.

* If you're doing lots of writes, the lock might expire before you're finished.  This could be mitigated somewhat in the client by checking how much time has elapsed and refreshing the lock periodically, but it's still rather inelegant.

Thoughts?

Comment 1

[Gregory Szorc [:gps]]

Another problem with this model is it isn't enforced. "Stupid" clients could blow through an already-acquired lock, invalidating assumptions in "intelligent" clients and cause pain.

A problem with enforced locks is, well, it is enforced. What if a misbehaving or compromised client didn't release a lock, etc?

Locks are complicated. I think they warrant a dedicated API rather than shoehorning it into the existing storage model. If we can do what we want with the existing storage model, I'm all for it. But, I don't think we'll be able to without some protocol enhancements. At that point, you have API support :)

Comment 2

[Ryan Kelly [:rfkelly]]

(In reply to Gregory Szorc [:gps] from comment #1)
> Locks are complicated.

Let's go shopping...

> I think they warrant a dedicated API rather than
> shoehorning it into the existing storage model. If we can do what we want
> with the existing storage model, I'm all for it. But, I don't think we'll be
> able to without some protocol enhancements.

Agree, I don't see a way to do *enforced* locking with the current protocol.

Comment 3

[Richard Newman [:rnewman]]

Crazy thought: can you have any write ensure that the record exists with some expiry? That is, a non-compliant client can come write without worrying, and a compliant client will see a lock having been taken.

Comment 4

[Ryan Kelly [:rfkelly]]

(In reply to Richard Newman [:rnewman] from comment #3)
> Crazy thought: can you have any write ensure that the record exists with
> some expiry? That is, a non-compliant client can come write without
> worrying, and a compliant client will see a lock having been taken.

If we're going to go that far, I think we should just implement a full locking API.

Comment 5

[Richard Newman [:rnewman]]

(In reply to Ryan Kelly [:rfkelly] from comment #4)

> If we're going to go that far, I think we should just implement a full
> locking API.

That depends on how much you care about "within the scope of the current protocol". :)

Comment 6

[Ryan Kelly [:rfkelly]]

(In reply to Gregory Szorc [:gps] from comment #1)
> "Stupid" clients could blow through an already-acquired lock, invalidating
> assumptions in "intelligent" clients and cause pain.

I wonder if using cooperative locking in conjunction with X-If-Unmodified-Since would get around the issue of stupid clients?  Use /meta/lock as a signal to smart clients that you're making changes and they should come back later to avoid contention.  Use X-If-Unmodified-Since on each request to guard yourself against stupid clients who ignored your lock and wrote stuff anyway.

This may be a concrete improvement over the status quo, where you currently have to use X-If-Unmodified-Since to guard yourself against all clients who might be making concurrent changes.

You wouldn't be able to use such locking to simplify the logic in the client code, but it may allow you to reduce the rate of contention and conflicts at runtime.

Anyway, this bug is really just "Can we do useful locking semantics with the current API?" and I'm quite happy to accept "no" as an answer.  The higher-level bug of "Do we want to try to add session-level locking to Sync2.0?" hasn't been floated, and I'm wary of scope-creep in that regard.  IIRC we postponed it from original sync2.0 scope because it was too big of a change.

Comment 7

[Ryan Kelly [:rfkelly]]

> I'm quite happy to accept "no" as an answer.

Ya, let's go with that.  Three years later.  Time flies!