1465314 - Determine how often clients wipe server data in practice

Status: RESOLVED INCOMPLETE

(Cloud Services Graveyard :: Server: Sync, enhancement)

Description

[Ryan Kelly [:rfkelly]]

Bug 1465313 identifies some ways in which Firefox Sync clients may cause server data to be wiped, but IIUC we don't really know how likely those events are to happen in practice.  Let's find out.

:markh, do you know if we happen to have any existing client-side telemetry for these behaviours?

:bobm, one way we could dig into this on the server-side, is to look for DELETE requests in the server logs.  Are the syncstorage nginx logs somewhere where devs can examine them?  I recall you setting up a pipeline at some point.

Comment 1

[Ryan Kelly [:rfkelly]]

> one way we could dig into this on the server-side, is to look for DELETE requests in the server logs.

It's worth noting that the tokenserver will also send DELETE requests to the storage nodes, when cleaning up after a node migration:

  https://github.com/mozilla-services/tokenserver/blob/master/tokenserver/scripts/purge_old_records.py#L78

We'll have to see if we can reliably distinguish between client-initiated and tokenserver-initiated deletes.

Comment 2

[Bob Micheletto [:bobm]]

(In reply to Ryan Kelly [:rfkelly] from comment #0)

> :bobm, one way we could dig into this on the server-side, is to look for DELETE requests in the server logs.

Looks like client delete requests happen in a session with a series of requests that have an Fx UA string, and are in basically the following format:

DELETE /1.5/SYNCUID/storage HTTP/1.1
DELETE /1.5/SYNCUID/storage/clients HTTP/1.1
DELETE /1.5/SYNCUID/storage/prefs HTTP/1.1
DELETE /1.5/SYNCUID/storage/passwords HTTP/1.1
DELETE /1.5/SYNCUID/storage/tabs HTTP/1.1
DELETE /1.5/SYNCUID/storage/bookmarks HTTP/1.1
DELETE /1.5/SYNCUID/storage/addons HTTP/1.1
DELETE /1.5/SYNCUID/storage/forms HTTP/1.1
DELETE /1.5/SYNCUID/storage/history HTTP/1.1
DELETE /1.5/SYNCUID/storage/prefs HTTP/1.1
DELETE /1.5/SYNCUID/storage/passwords HTTP/1.1
DELETE /1.5/SYNCUID/storage/tabs HTTP/1.1
DELETE /1.5/SYNCUID/storage/addons HTTP/1.1

Tokenserver delete requests have the py-requests UA string, come from one of a set of known IPs, and look like this:

DELETE /1.5/SYNCUID

> Are the syncstorage Nginx logs somewhere where devs can examine them?

Sort of, it's probably easiest if I package them up for examination or run the queries and report.  Happy to do either.

> I recall you setting up a pipeline at some point.

There's a sample based Kibana dashboard and logging 1.0 pipeline.  We're moving to Logging 2.0 as part of the automation work.

Comment 3

[Bob Micheletto [:bobm]]

:bbangert since you have an interest in this, would a sample of 30 days be sufficient to answer this question?

Comment 4

[Mark Hammond [:markh] [:mhammond]]

(In reply to Ryan Kelly [:rfkelly] from comment #0)
> :markh, do you know if we happen to have any existing client-side telemetry
> for these behaviours?

Unfortunately we do not, although it's not tricky to add them if this would be useful on an ongoing basis.

Comment 5

[Ben Bangert [:benbangert]]

We could probably close this, as it seemed when talking in SF that it would be hard/impossible to determine intentional deletes vs. any sort of possible client error.

Comment 6

[Ryan Kelly [:rfkelly]]

> We could probably close this

WFM