1343564 - HMAC mismatch when syncing history

Status: RESOLVED FIXED

(Firefox :: Sync, defect)

Description

[Lina Butler [:lina]]

Attachment: 8980629.txt

Splitting this out from bug 1343414 for Desktop. Could this be caused by keys changing mid-sync, or a truncated record?

Comment 1

[Richard Newman [:rnewman]]

{"stylishsync":1487968095.8,"prefs":1488219720.73,"tabs":1488104111.51,"clients":1488315601.15,"crypto":1488042288.82,"forms":1488312288.65,"meta":1488104121,"bookmarks":1488232979.83,"addons":1488107903.5,"greasemonkey":1488315601.66,"history":1488316442.72}

stylishsync: 1487968095.8
crypto:      1488042288.82

The keys are newer than the stylishsync data. That means we tried automated HMAC recovery, reuploading our key bundle, and this presumably triggered a reupload from other clients.

It's a bit confusing to see later in the log:

1488316583129	Sync.Collection	DEBUG	mesg: GET success 200 https://sync-385-us-west-2.sync.services.mozilla.com/1.5/61345190/storage/history?newer=1488316097.69&full=1&limit=5000

Keys changing mid-sync *could* happen, and would cause this symptom, but it's a tight race. You'd need:

  Client A: uploads keys and data.
  Client B: downloads keys and data.
  Client B: uploads new data.
  Client A: begins sync.
  Client A: downloads info, meta, checks keys.
  Client B: changes keys, wipes storage, uploads new history.
  Client A: downloads history encrypted with new keys that it doesn't have.

Another explanation would be a stale server timestamp, which we hypothesize in Bug 1342679 Comment 10 might be a server bug: in this case the race window broadens massively, as Client A won't see that the keys have changed.

Comment 2

[Richard Newman [:rnewman]]

However, it's worth noting that the keys collection shouldn't change -- when you change your FxA password we reupload the same keys. When you reset your FxA you don't go to the same server. So an HMAC mismatch should be impossible in Sync 1.5.

That raises the specter of encoding and crypto problems.

Consider Bug 1331984, in which iOS saw a malformed record uploaded by a desktop device: the body was invalid base64. Mishandling of data might well cause an HMAC computation to be incorrect.

Comment 3

[:Grisha Kruglov]

Unless there's reason to believe in presence of another root cause (other than what we saw couple of months ago), considering fixed.