1592136 - Disable LSNG (LocalStorage NextGen) in Firefox 70 release for dot release.

Status: RESOLVED FIXED

(Core :: Storage: localStorage & sessionStorage, task, P1)

Description

[Andrew Sutherland [:asuth] (he/him)]

A number of bugs have been filed since Firefox 70 release that report site breakage that's consistent with LocalStorage throwing errors. The plan is to turn off LocalStorage on release in the first available dot release.

We currently believe what we're seeing to be a mixture of 2 things:

1. Most Common: QuotaManager storage initialization failures cascading into LocalStorage breakage.
2. Least Common: Edge cases related to unexpected LSNG corruption that are not properly recovered from. We have at least one report where it seems that the database has 2 zeroed pages that results in a corruption error at query time but not at database open time.

• The corruption superficially looks hardware-related, but we have not yet ruled out the possibility of improvements in browser shutdown leaving a database in an inconsistent state.

Next steps will be to add improved telemetry that can characterize the health of the LocalStorage API and what failures are happening.

I'm currently marking known manifestations of the problem as blocked by this bug.

Comment 1

[Andrew Sutherland [:asuth] (he/him)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=30b5c18dff15a24688f72c00353f7fef4496c364

Comment 2

[Andrew Sutherland [:asuth] (he/him)]

Attachment: Bug 1592136 - Disable LSNG in 70 release. r=janv

Comment 3

[Jan Varga [:janv]]

There are some xpcshell-test failures, I'm looking at it right now.

Comment 4

[Roman]

Regarding zeroed out pages: if the "pages" are 4096 bytes long at 4096-aligned offsets, and on NTFS, then it doesn't have to be a hardware issue nor a Firefox bug. It could be a Windows issue that I have seen happen on all kinds of hardware and to all kinds of software, invariably after a dirty shutdown.

Comment 5

[Jan Varga [:janv]]

The patch for fixing tests will be under bug 1578177.

Comment 6

[Jan Varga [:janv]]

(In reply to Roman from comment #4)

Regarding zeroed out pages: if the "pages" are 4096 bytes long at 4096-aligned offsets, and on NTFS, then it doesn't have to be a hardware issue nor a Firefox bug. It could be a Windows issue that I have seen happen on all kinds of hardware and to all kinds of software, invariably after a dirty shutdown.

Ok, good to know, the database page size in our case is 1024 bytes.

Comment 7

[Jan Varga [:janv]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=0d087f64df7ba3271f663e539015712863346374

Comment 8

[Andrew Sutherland [:asuth] (he/him)]

Comment on attachment 9104775 [details]
Bug 1592136 - Disable LSNG in 70 release. r=janv

Beta/Release Uplift Approval Request

• User impact if declined: With LSNG enabled, QuotaManager initialization failures break LocalStorage which results in oddly broken websites for users.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: Bug 1578177
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): This is a pref disabling for a subsystem designed to support toggling the preference off. It's something we've unfortunately gotten to use several times on previous disablings and QA has tested with on/off.
• String changes made/needed: none

Comment 9

[Liz Henry (:lizzard) (relman/hg->git project)]

Taking this for tomorrow's 70.0.1 build.

Comment 10

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/releases/mozilla-release/rev/21ce5fdd3146

Comment 14

[viking4096]

I updated to Firefox 70.0.1 yesterday. Now the website my.matterport.com fails to load properly. I checked thye same webpage on Firefox 70.0 on another Linux platform, and the webpage loads fine and functions. I believe that matterport uses dynamic Javascripts, but am not completely sure. I run Elementary OS Juno with kernel 4.15.0-66-generic.

Comment 15

[Emma Humphries ☕️🎸🧞‍♀️✨ (she/they) [:emceeaich] (Pacific Time) use needinfo]

This bug has been identified as part of a pilot on determining root causes of blocking and dot release drivers.

It needs a root-cause set for it. Please see the list at https://docs.google.com/document/d/1FFEGsmoU8T0N8R9kk-MXWptOPtXXXRRIe4vQo3_HgMw/.

Add the root cause as a whiteboard tag in the form [rca - <cause> ] and remove the rca-needed keyword.

If you have questions, please contact :tmaity.

Comment 16

[Jens Stutte [:jstutte]]

Same as https://bugzilla.mozilla.org/show_bug.cgi?id=1570644#c18.