865894 - Allow deletion of multiple documents

Status: RESOLVED FIXED

(Mozilla Metrics :: Metrics Data Ping, defect)

Description

[Gregory Szorc [:gps]]

Currently, clients use a simple algorithm for keeping track of uploaded document IDs:

1) Generate new ID
2) Upload document
3) Record ID as last ID when response is received

There are some minor race conditions between #2 and #3. e.g. the server receives the upload but the client never sees the response.

I'd like to change the behavior of the client so it records document IDs just before upload occurs. At that point, it assumes the document is stored remotely. Only after a successful request to delete that document ID does it mark it as non-remotely stored.

This means that the client will be storing multiple document IDs and will possibly want to delete multiple documents. Currently, we are forced to issue multiple HTTP requests.

I propose that Bagheera's upload and delete APIs grow the ability to delete multiple documents so the client only needs to perform a single HTTP request.

Comment 1

[Daniel Einspanjer [:dre] [:deinspanjer]]

I don't think we should change the DELETE method unless there is a good way to implement multi-delete that doesn't violate general REST behavior.

It should be trivial to make sure that Bagheera can support processing multiple X-Obsolete-Document headers in a single POST request though.

Comment 2

[Gregory Szorc [:gps]]

I agree it would be a violation of HTTP/REST to overload DELETE to remove multiple resources at different URIs. So, I guess that means we will need to implement a new HTTP POST API for multi-delete.

Comment 3

[Daniel Einspanjer [:dre] [:deinspanjer]]

What is the specific use case that requires supporting multiple deletes without a corresponding new document?

Could we get by with issuing multiple DELETE statements if we have some form of cleanup to do without a corresponding submit?  I don't think this should happen often enough to be likely to cause performance issues.

Comment 4

[Gregory Szorc [:gps]]

Unified multi-delete is easier for clients to implement, is more considerate of client's bandwidth, is more likely to succeed (since there would be only a single HTTP request, and saves server resources (assuming it's cheaper to process a single request containing N actions then N requests each performing a single action).

I mostly care about multi-delete on upload. If that's the only API you can provide, I'll take it. But, I'd much prefer to have multi-delete on both operation types.

Comment 5

[Mark Reid [:mreid]]

I have code ready to support multiple X-Obsolete-Doc headers (or a comma-separated list of doc ids in a single header) here:

https://github.com/mozilla-metrics/bagheera/pull/17

I expect it to end up in Bagheera 0.11, which will probably be released sometime next week.

For the moment there are no plans to support multiple document deletion when sending an HTTP DELETE request.

Comment 6

[Daniel Einspanjer [:dre] [:deinspanjer]]

Harsha, can you update us with the status of whether multi-delete is in production yet or not? If not, can we do a push of it next week?

I also want to post a quick recap of some info I shared via IRC on this topic.

The way the currently implemented multi-delete works is to pass a comma separated list of IDs in the X-Obsolete-Document header.

The DELETE method still takes only a single ID in the URI, but I believe that we can support the POST of an empty document with a multi-delete header in it to work around that limitation.  We need a separate bug to track that testing.

Comment 7

[Harsha [:harsha]]

Daniel,
       we didn't deployed the multi-delete yet in production. Will schedule it to deploy on 19th june.

Comment 8

[Gregory Szorc [:gps]]

Was this deployed?

Comment 9

[Harsha [:harsha]]

deploying it now.

Comment 10

[Harsha [:harsha]]

bagheera-0.11 with X-Obsolete-Document header for deleting multiple docs is live on production.
-Harsha