935947 - localStorage never recovers from an "NS_ERROR_FILE_NO_DEVICE_SPACE: File error: No device space" even when disk space recovers, should listen for nsIDiskSpaceWatcher

Status: RESOLVED WORKSFORME

(Core :: Storage: localStorage & sessionStorage, defect, P3)

Description

[Hallvord R. M. Steen [:hallvors]]

While I was using Firefox, the computer ran out of disk space.

After un-installing a few things and cleaning up so there are several GBs free, Firefox is still being weird: all sites that try to use localStorage fail due to this error:

[13:37:06.780] NS_ERROR_FILE_NO_DEVICE_SPACE: File error: No device space @ https://mail.mozilla.com/zimbra/js/Startup1_2_all.js.zgz?v=130524122052:22811

So Zimbra doesn't finish loading, GitHub fails - etc. There's seems to be something in the localStorage functionality that stores the "disk is full" state (to avoid checking it too often?) and doesn't un-set this flag again.

Comment 1

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

I suspect this comes up from the storage layer (and maybe even sqlite).

Honza, care to look into this?

Comment 2

[Boris Zbarsky [:bzbarsky]]

Yeah, the only place storage code produces that error code is here:

58     case SQLITE_FULL:
59     case SQLITE_TOOBIG:
60       return NS_ERROR_FILE_NO_DEVICE_SPACE;

Comment 3

[Honza Bambas (:mayhemer)]

In localStorage I only take the result of the operation that sqlite (mozStorage) gives me.  I am no deep expert to our mozStorage impl, someone who wrote those few lines of code at comment 2 should look at this.

My first raw idea is we could try 'restarting' the database.  I.e. close it and open it again in background.  But in case the disk is actually of space/quota we may just end up with lot of unnecessary IO.

Comment 4

[(no longer active)]

(In reply to Honza Bambas (:mayhemer) from comment #3)
> In localStorage I only take the result of the operation that sqlite
> (mozStorage) gives me.  I am no deep expert to our mozStorage impl, someone
> who wrote those few lines of code at comment 2 should look at this.

That might be Marco.

Comment 5

[Sylvestre Ledru [:Sylvestre]]

I just experienced the same issue. This is causing my Firefox to be mostly useless on some website ( https://travis-ci.org/ / or bill.com). For a regular user, the obvious alternative is to change browser...

Honza, do you know which Marco Ehsan is talking about?

Comment 6

[Marco Bonardo [:mak]]

I guess it's me. I assume we hit SQLITE_FULL and not SQLITE_TOOBIG, cause the latter can happen when binding too large blobs or strings, an it wouldn't persist any way.

I didn't read whether restarting the browser fixed things?
We've seen some cases where a full disk has caused database corruption, thus first of all we should exclude that. running a "PRAGMA integrity_check" query on the DB would be useful.

Is your temporary folder in the same partition or in a separate partition? If it's in a separate partition did you still had space in the temp folder? Sqlite may use the temp folder for some of its tasks and if there's no space it will fail with SQLITE_FULL even if the main partition has space.

I will verify, but off-hand I can't think of any place where we "store" an error code, we usually just call into Sqlite and forward its errors. So it may be a Sqlite internal behavior, or a db corruption.

Comment 7

[Marco Bonardo [:mak]]

if the problem is due to a full temp partition, it could be workarounded by using PRAGMA temp_store = MEMORY on the connection. And probably we could evaluate making this the default for all the new connections through the SQLITE_TEMP_STORE compile time option. We'd use a bit more memory, but operations may be faster and we wouldn't depend on having space in a temp partition.

Comment 8

[Marco Bonardo [:mak]]

Sylvestre, can you tell me if you have a separate partition for temp and whether restarting the browser fixed things?
Could you please try to check integrity of the db?

Comment 9

[Sylvestre Ledru [:Sylvestre]]

The answers:
* no /tmp/ partition (/tmp/ is in /)
* restarting the browser fixed the issue 
* which db do you want me to check?

Thanks :)

Comment 10

[Marco Bonardo [:mak]]

If it's related to DOM Storage, I think it's still webappsstore.sqlite (IIRC there's work ongoing to replace it, but it's not done yet. I'm not 100% sure, I think Jan Varga is working on the new store).

Comment 11

[Sylvestre Ledru [:Sylvestre]]

sqlite> PRAGMA integrity_check;
ok

but I guess the restart changed that

Comment 12

[Marco Bonardo [:mak]]

Ok, I don't think Storage does any kind of error caching, or at least I didn't find it. Though, we do quota management, and thus I wonder if out quota layer (that is part in the VFS and part managed in the quota manager) could get confused by this unexpected full error.
Since restarting solves the problem, it's clear something keeps thinking there's no space, and that something gets a reset on restart. Jan, I think you know the quota handling far better than me, do you think it may get confused by a full disk?

Comment 13

[Jan Varga [:janv]]

Please note that localStorage is not yet plugged into quota manager. It does it's own quota management.
That was a quick answer. I'll take a deeper look later.

Comment 14

[Dennis Schridde]

I also just hit this issue, and will likely hit it a few more times, caused by bugs in other software repeatedly filling my disk. Is there a solution in sight?

Comment 15

[Dennis Schridde]

Earlier, someone requested to make this error user-facing (when it is actually caused by a full disk), so the user has a chance to understand why Firefox suddenly starts to behave weirdly.

See-Also: https://bugzilla.mozilla.org/show_bug.cgi?id=433445

Comment 16

[Andrew Sutherland [:asuth] (he/him)]

While investigating bug 1341070 I noticed that StorageDBThread effectively latches the database status once the first error occurs, causing all future attempts to use localstorage to fail.

Good news/bad news:
- Good news: Apparently toolkit has an nsIDiskSpaceWatcher that generates "disk-space-watcher" observer events.  localStorage could listen for these events and clear the error state when the "free" notification is received.
- Bad news: It looks like nsIDiskSpaceWatcher is a b2g-only thing, with the only implementation being in hal/gonk.

So, this case could be handled by making nsIDiskSpaceWatcher work on all platforms.  It would probably make sense to address this for localStorage after it's refactored to be a quota manager consumer, and then have quota manager handle that issue.  (IndexedDB already has low-disk awareness from b2g efforts.)

Comment 17

[Jan Varga [:janv]]

Yes, let's fix this along with the refactoring to be a quota manager client.

Comment 18

[Gabriele Svelto [:gsvelto]]

Just stumbled upon this... I've removed the remnants of the disk space watcher in bug 1488401 and cleaned up all its consumers. It hadn't had a working implementation for over two years.

Comment 19

[braiamp]

(In reply to Gabriele Svelto [:gsvelto] from comment #18)

Just stumbled upon this... I've removed the remnants of the disk space
watcher in bug 1488401 and cleaned up all its consumers. It hadn't had a
working implementation for over two years.

Are you suggesting that clients should not see this issue anymore when the filesystem fills up or that localStorage should recover from using some other facility?

BTW, this is what verify integrity returns now:

> Task: checkIntegrity
+ The places.sqlite database is sane
+ The favicons.sqlite database is sane
> Task: invalidateCaches
+ The caches have been invalidated
> Task: checkCoherence
+ The database is coherent
> Task: expire
+ Database cleaned up
> Task: originFrecencyStats
+ Recalculated origin frecency stats
> Task: vacuum
+ Initial database size is 46080KiB
+ The database has been vacuumed
+ Final database size is 46080KiB
> Task: stats
+ Places.sqlite size is 46080KiB
+ Favicons.sqlite size is 26848KiB
+ pragma_user_version is 54
+ pragma_page_size is 32768
+ pragma_cache_size is -2048
+ pragma_journal_mode is wal
+ pragma_synchronous is 1
+ History can store a maximum of 116737 unique pages
+ Table moz_origins has 4170 records
+ Table moz_places has 78780 records
+ Table moz_historyvisits has 196897 records
+ Table moz_inputhistory has 170 records
+ Table moz_bookmarks has 1194 records
+ Table moz_bookmarks_deleted has 0 records
+ Table moz_keywords has 0 records
+ Table sqlite_sequence has 1 records
+ Table moz_anno_attributes has 2 records
+ Table moz_annos has 110 records
+ Table moz_items_annos has 0 records
+ Table moz_meta has 7 records
+ Table sqlite_stat1 has 17 records
+ Index sqlite_autoindex_moz_origins_1
+ Index sqlite_autoindex_moz_inputhistory_1
+ Index sqlite_autoindex_moz_bookmarks_deleted_1
+ Index sqlite_autoindex_moz_keywords_1
+ Index sqlite_autoindex_moz_anno_attributes_1
+ Index moz_places_url_hashindex
+ Index moz_places_hostindex
+ Index moz_places_visitcount
+ Index moz_places_frecencyindex
+ Index moz_places_lastvisitdateindex
+ Index moz_places_guid_uniqueindex
+ Index moz_places_originidindex
+ Index moz_historyvisits_placedateindex
+ Index moz_historyvisits_fromindex
+ Index moz_historyvisits_dateindex
+ Index moz_bookmarks_itemindex
+ Index moz_bookmarks_parentindex
+ Index moz_bookmarks_itemlastmodifiedindex
+ Index moz_bookmarks_dateaddedindex
+ Index moz_bookmarks_guid_uniqueindex
+ Index moz_keywords_placepostdata_uniqueindex
+ Index moz_annos_placeattributeindex
+ Index moz_items_annos_itemattributeindex
> Task: _refreshUI

Comment 20

[Jens Stutte [:jstutte]]

LSNG has been enabled a while ago.