Closed Bug 1341070 Opened 8 years ago Closed 3 years ago

localStorage implementation does not handle dynamically-detected corruption, reports NS_ERROR_FILE_CORRUPTED for localStorage-using websites

Categories

(Core :: Storage: localStorage & sessionStorage, defect, P2)

Product:
Core
Component:
Storage: localStorage & sessionStorage
Type:
defect

Tracking

()

Status:
RESOLVED WORKSFORME

People

(Reporter: jwalker, Unassigned)

References

(Blocks 1 open bug)

Details

I pinned it down to a call to window.localStorage.setItem() in feedly.com, but it happens in other places too.
Have you used a profile with "future" releases and then gone back to a previous one? We sometimes have issues with schema changes. Andrew McCreight said we return this error in a lot of places. Any chance you could catch this in a debugger or give a more detailed STR? Thanks!
Flags: needinfo?(jwalker)
So it looks like any calls to window.localStorage.setItem or window.localStorage.getItem kick it off. It seems likely that any website that used either of those would fail. I don't think I've used anything other than Nightly for ages (with that profile). I used an older FF temporarily in an attempt to debug, but it was already happening by that time. My workaround is to `mv webappsstore.sqlite webappsstore.sqlite.corrupted` It looks like the DB itself is ok. `.dump webappsstore2` takes a long time to complete, but it does complete without error (as far as I can see). So my hunch is that some of the data there is not what something is expecting.
Flags: needinfo?(jwalker)
Lots of sites use localStorage and I haven't seen any other reports so I'm not sure what could have happened. We sometimes upgrade our version of SQLite ... Andrew Sutherland's been looking at localStorage a bit recently so maybe he has an idea for how to debug. (If you could give webappsstore.sqlite to :asuth or :janv privately via email, that would be best.)
Flags: needinfo?(jwalker)
Flags: needinfo?(bugmail)
(In reply to Joe Walker [:jwalker] (needinfo me or ping on irc) from comment #2) > It looks like the DB itself is ok. `.dump webappsstore2` takes a long time > to complete, but it does complete without error (as far as I can see). This suggests the origin_key_index index is corrupt then. We'd expect shell.c's run_table_dump_query to print something like "/**** ERROR: (%d) %s *****/\n" before dump_callback tries dumping the table with reverse rowid order if the table itself was busted. I guess this because if we explain the query for what localstorage does at startup in StorageDBThread::InitDatabase: ``` sqlite> EXPLAIN QUERY PLAN SELECT DISTINCT originAttributes || ':' || originKey FROM webappsstore2 ...> ; 0|0|0|SCAN TABLE webappsstore2 USING COVERING INDEX origin_key_index 0|0|0|USE TEMP B-TREE FOR DISTINCT ``` We see it uses a covering index and doesn't actually walk the table. Same deal with StorageDBThread::DBOperation::Perform on OpPreload, which is the dynamic access pattern we expect. ``` sqlite> EXPLAIN QUERY PLAN SELECT key, value FROM webappsstore2 WHERE originAttributes = "foo" AND originKey = "bar" ORDER BY key LIMIT -1 OFFSET 0; 0|0|0|SEARCH TABLE webappsstore2 USING INDEX origin_key_index (originAttributes=? AND originKey=?) ``` (Note that this is not a covering index usage; we're retrieving the value which is not part of the index. The table rows will be accessed via rowindex.) We expect the error to be flowed back into StorageCache's mLoadResult via preload, and also for dynamic calls, for mStatus to end up taking on the result too. Things could end up slightly more complicated due to a weird remote filesystem heuristic StorageDBThread has. I suspect if you do: `PRAGMA integrity_check;` it should report the index as corrupt. It may make sense to morph this bug into an enhancement for localstorage to handle corruption better. Most of our mozStorage consumers do a poor job of handling corruption reported at runtime. We've discussed improving the situation for async consumers in bug 1125157, see bug 1125157 comment 7 in particular... but for a (clever) synchronous consumer like localstorage it probably should be handling the corruption itself, although maybe using some mozStorage helpers. For :jwalker's specific scenario, it sounds like running a VACUUM might just automagically fix things. And that's something localstorage could perhaps leverage itself in an enhancement. Although I'm not sure it's worth special-casing corruption handling to that extent given that: - in many cases the scenario will not be recoverable - a situation that looks recoverable may just be a case where the corruption was undetectable in the table. AKA the data's corrupt, but the btree looks intact. cc'ing mak for general corruption-scenario awareness.
Flags: needinfo?(bugmail)
Summary: NS_ERROR_FILE_CORRUPTED in several websites → localStorage implementation does not handle dynamically-detected corruption, reports NS_ERROR_FILE_CORRUPTED for localStorage-using websites
Andrew Sutherland wrote: > I suspect if you do: `PRAGMA integrity_check;` it should report the index as corrupt. sqlite> PRAGMA integrity_check; *** in database main *** On tree page 1185 cell 159: Rowid 346481 out of order On tree page 1185 cell 157: Rowid 346452 out of order On tree page 1185 cell 155: Rowid 346453 out of order On tree page 1185 cell 154: Rowid 346456 out of order On tree page 1185 cell 153: Rowid 346482 out of order On tree page 1185 cell 151: Rowid 346483 out of order ... > For :jwalker's specific scenario, it sounds like running a VACUUM might just automagically fix things. sqlite> VACUUM; sqlite> PRAGMA integrity_check; non-unique entry in index origin_key_index non-unique entry in index origin_key_index row 3767 missing from index origin_key_index non-unique entry in index origin_key_index ... Does that indicate that we need more than a vacuum? The blow-the-DB-away-and-restart solution for me has been surprisingly annoyance free.
Flags: needinfo?(jwalker)
(In reply to Joe Walker [:jwalker] (needinfo me or ping on irc) from comment #6) > Does that indicate that we need more than a vacuum? Yeah; I wrote that bit before I realized that your problem could be specific operations failing rather than InitDatabase's index scan. Thanks for running the integrity_check and checking out the impact of VACUUM!
I think we should fix this but given it's at least somewhat hard to get into this situation, I'll mark it as P3. Jan, will your localStorage redesign have an impact here?
Flags: needinfo?(jvarga)
I have this problem with at least two sites (salesforce.com and redhat.com customer portal) and almost in a persistent way. There is a form to manually try to solve it? Currently I'm just delete all the content of .mozilla/firefox/<PROFILE>/storage/default/ the files regenerate but get the same error. It's not necessary to delete all the content, you can just delete the directory with the site that you have issues, but the last time I have troubles with 3 sites (add whatsapp web to the list). That sometimes solve the issue, but just for a few hours/days, I'm not clearly have an idea of what trigger this. Just for the record, I'm using a SSD Storage, maybe there is some issue with this kind of storage? I read that a couple of apps (like spotify) that have some bad write patterns with the SSD may push extra wear on the disk and sometimes corrupt files. Let me know if you need more info, files, or test, I think that I can reproduce this.
Per your example path scheme, you want to delete ".mozilla/firefox/<PROFILE>/webappsstore.sqlite" if localstorage corruption is your problem, which it sounds like it could be. The other possibility would be if you are trying to use very different versions of Firefox with the same profile; many storage subsystems don't support downgrades.
Would anyone like a copy of webappsstore.sqlite which seems to be exhibiting this problem? Gerv
(In reply to Gervase Markham [:gerv] from comment #11) > Would anyone like a copy of webappsstore.sqlite which seems to be exhibiting > this problem? > > Gerv Yes, I'd like to take a look.
Flags: needinfo?(jvarga)
Shared via Dropbox to your gmail address. (It's too large to email, even when gzipped.) Gerv
Priority: -- → P2
(In reply to Jan Varga [:janv] from comment #12) > (In reply to Gervase Markham [:gerv] from comment #11) > > Would anyone like a copy of webappsstore.sqlite which seems to be exhibiting > > this problem? > > > > Gerv > > Yes, I'd like to take a look. sqlite> pragma integrity_check; *** in database main *** On tree page 827 cell 73: Rowid 993240 out of order (previous was 993322) On tree page 827 cell 74: Rowid 993199 out of order (previous was 993240) On tree page 827 cell 75: Rowid 993079 out of order (previous was 993199) On tree page 827 cell 76: Rowid 992330 out of order (previous was 993079) On tree page 3365 cell 178: Rowid 991704 out of order (min less than parent min of 992446) On tree page 3556 cell 198: Rowid 998135 out of order (previous was 998349) ... sqlite> vacuum; sqlite> pragma integrity_check; row 29409 missing from index origin_key_index row 29410 missing from index origin_key_index ... row 30430 missing from index origin_key_index row 30437 missing from index origin_key_index wrong # of entries in index origin_key_index So integrity_check reports missing rows in index origin_key_index, even after a vacuum (I repeated it several times) Btw, this database has 3451 unique origins :)
(In reply to Andrew Overholt [:overholt] from comment #8) > I think we should fix this but given it's at least somewhat hard to get into > this situation, I'll mark it as P3. Jan, will your localStorage redesign > have an impact here? We consider to split the shared database into per origin databases and we also consider to replace sqlite with something else. So yes, it does have a big impact, but you know it won't happen anytime soon, it's a lot of work. Right now, I'm focusing on moving PStorage from PContent to PBackground.
(In reply to Andrew Overholt [:overholt] from comment #8) > I think we should fix this but given it's at least somewhat hard to get into > this situation, I'll mark it as P3. Jan, will your localStorage redesign > have an impact here? I'm asking to raise priority of this bug. It drives people to Chrome as web pages silently stop working and the cause is not evident, see our reports: https://bugzilla.redhat.com/show_bug.cgi?id=1497738#c1 https://bugzilla.redhat.com/show_bug.cgi?id=1497738#c3
I'm working on a new LocalStorage implementation that uses per origin LocalStorage databases and it should also handle the corruption (by deleting entire database for given origin in the worst case).
I had the same problem with linkedin.com - getting NS_ERROR_FILE_CORRUPTED messages in the developer's console and the Linkedin site not loading (the spinner spins for ever and various other errors are shown in the console). Deleting webappsstore.sqlite solved this problem for me. If a developer is interested in a copy of my webappsstore.sqlite file, please let me know.
I've had what appears to be the same issue across dozens of websites when using Nightly, sometimes dozens or more times a day the NS_ERROR_FILE_CORRUPTED will cause the failure of JavaScript on a given site. URLs I can recall it typically affecting: trello.com digitalocean.com console.aws.amazon.com Trello in particular sticks out because it was affected most frequently. There's a bunch more domains too but they were typically sites I'm visiting once-off so my workaround was enough to get me through; I worked around the issue on a given site with the following JS: // First throws an NS_ERROR_FILE_CORRUPTED error, second one succeeds localStorage.clear(); localStorage.clear() which would temporarily clear the problem. Some sites would take a day to fail again, and others would fail immediately. After finding this bug report, I've removed my webappsstore.sqlite and restarted Nightly and sites are loading okay. I have my old webappsstore.sqlite if anyone would like it -- reinstating it replicates the NS_ERROR_FILE_CORRUPTED when attempting to load JavaScript on Trello.com.
Thanks for the report. We are working on a new local storage implementation that should address this issue. If everything goes well, it will be available in FF 61.
Hi, I have the same issue with FF DE 56 and on Mac OS High Sierra. Moreover, I don't find the file webappsstore.sqlite on my disk and the workaround 'localStorage.clear()' doesn't succeed. So, how can i proceed, please ? Thnaks in advance
Depends on: 1286798
(In reply to frederic.leleu from comment #21) > Hi, > I have the same issue with FF DE 56 and on Mac OS High Sierra. Moreover, I > don't find the file webappsstore.sqlite on my disk and the workaround > 'localStorage.clear()' doesn't succeed. So, how can i proceed, please ? > Thnaks in advance See https://support.mozilla.org/en-US/kb/profiles-where-firefox-stores-user-data#firefox:mac:fx56 for information on how to locate your profile folder. webappsstore.sqlite live in the profile folder.
Blocks: 1504142
Component: DOM: Core & HTML → DOM: Web Storage

Are there any updates? I started seeing this somewhat recently on a bunch of websites -- Slack and Netflix most frequently -- and clearing both local and session storage each time I open a new tab is rather annoying.

Yes, the new LocalStorage implementation is enabled on nightly and early betas currently and should ship either in the current Firefox 68 or the next 69 release cycle.

However, I'd recommend that you do one of two things to fix the problem immediately:

  1. Delete the corrupt database directly.
  • Opening the URL about:profiles address bar
  • Locate the profile listed as "This is the profile in use and it cannot be deleted."
  • Open the profile directory by using the "Open directory" button on the "Root directory" button.
  • Using the file manager that pops up, locate webappsstore.sqlite.
  • Quit Firefox. This will make it possible to safely delete the file. You may want to wait a few seconds because Firefox does not always close down instantly.
  • Delete the webappsstore.sqlite file.
  1. open the about:support URL and use the "Refresh Firefox" button. This will create a new profile and copy over data that does not include the LocalStorage database. Unfortunately this also leaves the old profile around. Which is good in some senses (you can always go back to it) and bad in others (privacy). After you're sure the new profile didn't lose any important information, it's advisable to use about:profiles and its "Remove" button to get rid of the old profile.
Blocks: 1569296

Why have we not shipped this? It's been over a year and our current localStorage implementation has issues.

It is enabled in Nightly and early Beta and we see from telemetry still more initialization failures than we should, such that we did not enable it on release (see bug 1599979). As far as we can see, it mostly boils down to resolve bug 1594075 and bug 1621920 (ignore several types of unknown files in the profile), together with bug 1645943 (which is a regression from the long file names handling fix). All these bugs are our top priority currently, still we need to see the numbers go down from telemetry once they land.

Depends on: 1599979

Even when setting dom.storage.next_gen to true in about:config, the problem described in https://bugzilla.mozilla.org/show_bug.cgi?id=1647619, still persists. And the error always mentions the same JavaScript on:
https://spoprod-a.akamaihd.net/files/sp-client/chunk.sp-pages-social_nl-nl_4ce026042c18403fbf69.js
It seems like a very large script. Is it worthwhile to investigate this script, or are there any other helpful ideas?
TIA

(In reply to Andrew Sutherland [:asuth] (he/him) from comment #25)

Yes, the new LocalStorage implementation is enabled on nightly and early betas currently and should ship either in the current Firefox 68 or the next 69 release cycle.

However, I'd recommend that you do one of two things to fix the problem immediately:

  1. Delete the corrupt database directly.
  • Opening the URL about:profiles address bar
  • Locate the profile listed as "This is the profile in use and it cannot be deleted."
  • Open the profile directory by using the "Open directory" button on the "Root directory" button.
  • Using the file manager that pops up, locate webappsstore.sqlite.
  • Quit Firefox. This will make it possible to safely delete the file. You may want to wait a few seconds because Firefox does not always close down instantly.
  • Delete the webappsstore.sqlite file.
  1. open the about:support URL and use the "Refresh Firefox" button. This will create a new profile and copy over data that does not include the LocalStorage database. Unfortunately this also leaves the old profile around. Which is good in some senses (you can always go back to it) and bad in others (privacy). After you're sure the new profile didn't lose any important information, it's advisable to use about:profiles and its "Remove" button to get rid of the old profile.

Thank you very much for this information. Leaving out 2) worked out in my case however, as it fixes broken websites but doesn't force you to create a new profile.
I noticed this issue after the latest Firefox update (v81.0 from Sep 22), so it could be related to that and might have affected other users, too.

I'm having the same problem on quite a few websites. The locakStorage.clear() seems to fix it for a bit, then it reappears.
v.82.0.2 on Win 10 64 bit

Having the problem of localStorage failing to be updated for at least one site. If I clear storage, the app can write new keys in, but updates to the keys never happen. v83.0 on MacOS. Confirmed that the site works with Chrome.

If you need a reproducible case: codesandbox.io is always throwing this error. For example: https://codesandbox.io/s/sentry-js-sdk-unhandled-promise-fuq2h?file=/src/index.js

I didn't analyze what the test case does, but I tried it on both Firefox Nightly and Chrome and it throws in both browsers. Can you confirm that with Chrome? Are you sure the test case is valid? (If it is, it seems this is a different issue.)

Flags: needinfo?(yefremov.oleksandr)

(In reply to Simon Giesecke [:sg] [he/him] from comment #36)

I didn't analyze what the test case does, but I tried it on both Firefox Nightly and Chrome and it throws in both browsers. Can you confirm that with Chrome? Are you sure the test case is valid? (If it is, it seems this is a different issue.)

Hi Simon,
in Chrome it works (site loads normally). In Firefox (85, macos) any page on codesandbox.io throws error NS_ERROR_FILE_CORRUPTED (unfortunately, I don't have the stack trace, somehow forgot to capture it before removing webappsstore.sqlite which fixed the issue permanently). I am not sure whether it is this issue or another one, I just stumbled across this ticket and decided to add a url where it was 100% reproducible for me (and for you, as far as I understood).

We saw this problem on a Citrix-environment with 32-bit Firefox as an App-V. Because of this we migrated our users to a new Windows 2016 platform with Citrix and a 64-bit version of Firefox 68. After that this problem went away,

I have been seeing this issue on a self hosted gitlab instance with Firefox 85.0.1. The workaround in comment #32 fixes it for the time being. Will update if it reappears.

The original issue should be mitigated/fixed by shipping LSNG. The remaining cases seem to be covered by bug 1683401 (which is still open and might deserve some attention).

Status: NEW → RESOLVED
Closed: 3 years ago
Flags: needinfo?(yefremov.oleksandr)
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.