1413949 - extension-storage sync engine should implement item size limits on the client

Status: RESOLVED DUPLICATE of bug 1311516

(WebExtensions :: Storage, defect, P3)

Description

[lolipopplus]

Attachment: error-sync-1509634436854.txt

User Agent: Opera/9.80 (Windows NT 6.0) Presto/2.12.388 Version/12.14
Build ID: 20171030163911

Steps to reproduce:

I sign up a new account & try to sync. I have totally 31 add-on.


Actual results:

Firefox sync got error, in about:sync-log I found error message


Expected results:

Firefox sync should not limit maximum bytes per object because users won't know what's happening

Comment 1

[lolipopplus]

Or, otherwise, tell users what's blocking the sync

Comment 2

[Edouard Oger [:eoger]]

I'm hitting the same problem actually on my home computer (but not on my 2nd computer which is on the same sync account).

Comment 3

[Edouard Oger [:eoger]]

> 1509634436830	Sync.Engine.Extension-Storage	ERROR	Syncing storage-sync-crypto: request failed: Error: HTTP 507; Error: HTTP 507 Insufficient Storage: Resource access is forbidden for this user (Maximum bytes per object exceeded (8733 > 8192 Bytes.) (resource://services-common/kinto-http-client.js:2354:21) JS Stack trace: [e-mail]:2377:21 < [e-mail]:2352:14

Comment 4

[lolipopplus]

(In reply to Edouard Oger [:eoger] from comment #2)
> I'm hitting the same problem actually on my home computer (but not on my 2nd
> computer which is on the same sync account).

I think 8733 bytes may be enough before, but most probably not for now and in near future.

Comment 5

[Ethan Glasser-Camp (:glasserc)]

Hi, sorry for taking so long to respond to this issue. The 8192 byte-per-object limit comes from Chrome, whose documentation we took to be the "spec" for this feature: https://developer.chrome.com/extensions/storage. However, we realized today that because we encrypt and base64 each record, the 8192 limit for records gets surpassed by encrypted records. For this reason, we're going to increase the per-object limit on the server side to 16kb. (See bug 1415634.) I know that (for instance) uBlock Origin has some logic to try to split keys that go over that limit, so hopefully this will fix syncing for extensions that respect the limit.

There's another, related issue, which is that all of these storage limits are only ever enforced on the server side, with no information ever provided on the server side. This was originally done for simplicity's sake, and because most quotas can be violated on the server side by several clients, each within the quota limit. However, the "maximum bytes per object" limit should probably be enforced on the client. This would at least hopefully make it easier for extension authors to know that their extension has a bug in it.

Although this bug is titled "Firefox should not limit maximum bytes per object", I don't believe that's likely to happen. The quotas are in place to prevent abuse of the storage.sync service. The alternative, "or indicate blocking", doesn't really make sense either -- all quota violations are bugs in add-ons, so telling the user isn't helpful, and by the time we are syncing storage.sync, it's too late to do anything about it anyhow. Firefox generally doesn't expose sync failures and I don't know if it makes sense for the storage.sync engine to try to reverse that just for storage.sync errors.

Comment 6

[Mark Hammond (PTO; back Oct-3) [:markh] [:mhammond]]

(In reply to Ethan Glasser-Camp (:glasserc) from comment #5)
> There's another, related issue, which is that all of these storage limits
> are only ever enforced on the server side, with no information ever provided
> on the server side. This was originally done for simplicity's sake, and
> because most quotas can be violated on the server side by several clients,
> each within the quota limit. However, the "maximum bytes per object" limit
> should probably be enforced on the client. This would at least hopefully
> make it easier for extension authors to know that their extension has a bug
> in it.

As Ethan mentions, we aren't going to remove these limits. We have bug 1415634 for changing how the server counts the size but best I can tell no bug to implement the above - so I'm going to morph this bug into that.

> Firefox generally doesn't expose sync failures and I don't know if it makes 
> sense for the storage.sync engine to try to reverse that just for storage.sync
> errors.

IMO it doesn't, but we can make noise in both sync logs and the browser console, which is still likely to help addon authors.

Comment 8

[Asif Youssuff]

Is there any way to find out which WebExtensions is causing the issue so that we can open a bug with the add-on author?

I have the about sync add-on installed, and it complains to me all the time about sync issues due to this bug. https://addons.mozilla.org/en-US/firefox/addon/about-sync/

I'd prefer not to turn it off, as I am also enabling the rust based bookmark sync and would like to report issues around those if I see them.

Comment 10

[Luca Greco [:rpl] [:luca] [:lgreco]]

Merging this into Bug 1311516 (which is also about implementing client-side quotas)