1422673 - Uptick in http 507 errors since ~November 20

Status: NEW

(WebExtensions :: Storage, defect, P3)

Description

[Mark Hammond [:markh] [:mhammond]]

There are a significant number of extension-storage syncs failing since around November 20 with a http 507 error. This is the most common error by far for that engine.

https://gist.github.com/mhammond/649a6f6c5606cff0e0d9d8a4f0b9e9a1

Comment 1

[Ethan Glasser-Camp (:glasserc)]

507 errors are the "insufficient storage" = "quota violation" error message. It's hard to know exactly what to do about these because it's not clear whether the 507 is a legitimate report of an extension using too much space, or a user having too many extensions, or a bug in the server (as it has been before). Apparently I have the telemetry already in place to look at this [1] but I don't see any alarming numbers in either the SCALARS_STORAGE.SYNC.API.USAGE.STORAGE_CONSUMED (typically less than 4k per collection and only being used by four collections (keyring + three extensions)) nor ITEMS_STORED (almost always less than 10).

I think the obvious fix here is to address bug 1311516 (client-side quota enforcement). Another thing might be to ask :wezhou to run some SQL queries in production to see what the quota records look like on the server side and whether they are typically as small as the client reports. But even if there are quite a few records with really high quotas, it's still possible that those are users who don't self-report via telemetry, so still not necessarily indicative of a bug on the server.

[1] https://telemetry.mozilla.org/new-pipeline/dist.html#!cumulative=1&end_date=2018-02-08&keys=DEFAULT%252FSTORAGE-SYNC-CRYPTO!DEFAULT%252FHTTPS-EVERYWHERE%2540EFF.ORG!DEFAULT%252F%257B73A6FE31-595D-460B-A920-FCC0F8843232%257D!DEFAULT%252F%257B7A7A4A92-A2A0-41D1-9FD7-1E92480D612D%257D&max_channel_version=nightly%252F60&measure=SCALARS_STORAGE.SYNC.API.USAGE.STORAGE_CONSUMED&min_channel_version=null&processType=*&product=Firefox&sanitize=1&sort_keys=submissions&start_date=2018-01-22&table=0&trim=1&use_submission_date=0