Closed Bug 1425146 Opened 7 years ago Closed 6 years ago

response_padding_size doesn't exist in cache.sqlite while accessing it in storage initialization

Categories

(Core :: Storage: IndexedDB, enhancement, P2)

Product:
Core
Component:
Storage: IndexedDB
Type:
enhancement

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla65
Tracking Flags:
Tracking Status
firefox64 --- fixed
firefox65 --- fixed

People

(Reporter: rhubscher, Assigned: tt)

References

(Blocks 1 open bug,
URL
)

Details

Attachments

(3 files, 13 obsolete files)

demo.png
54.76 KB, image/png
Details
Bug 1425146 - P1 - Ensure the schema for the cache.sqlite is up to date beforing accessing it; r=janv
1.93 KB, patch
tt
: review+
jcristau
: approval-mozilla-beta+
Details | Diff | Splinter Review
Bug 1425146 - P2 - Add a test reproducing the reported issue to verify that a Cache directory without a padding file and with an old version of the database doesn't break the storage initialization;
6.76 KB, patch
tt
: review+
jcristau
: approval-mozilla-beta+
Details | Diff | Splinter Review 
The only output we get is:

```
   IndexedDB UnknownErr: ActorsParent.cpp:598 (unknown)
   UnknownError kinto.js:5546:22
```

The code calling IndexedDB is here: https://github.com/Kinto/kinto.js/blob/35b0b0d0c2ea27ce280cced2c7678e72914be95f/src/adapters/IDB.js#L148

The dbname should be something like: default/notes here.

Do you know how we can check permissions for the IDB file or directory?
(In reply to Rémy Hubscher (:natim) from comment #0)
> Do you know how we can check permissions for the IDB file or directory?

What do you mean by that? You want to check it in the profile directory?

So AFAIK there's no good way to debug this in an optimized (release) build.
Well, maybe there are some messages from QuotaManager in the console before you get "IndexedDB UnknownErr: ActorsParent.cpp:598 (unknown)". It would help if you provided some QuotaManager related messages from the console.

The other option is to run a debug build, in that case you get much more "warnings" in the system console with source code file names and line numbers.
> What do you mean by that? You want to check it in the profile directory?

Yes that was my guess but I might be wrong.

> So AFAIK there's no good way to debug this in an optimized (release) build.

Ok

> Well, maybe there are some messages from QuotaManager in the console before you get "IndexedDB UnknownErr:

I haven't see them in the output that was provided by the user [0]

[0] https://github.com/mozilla/notes/issues/512
It is apparently not a permission problem according to the user.
When we investigate the unknown error coming from indexedDB.open() we usually end up seeing a profile that was used with newer Firefox that upgraded the storage directory in the profile and then was used again with an older release.
See bug 1246615.
Priority: -- → P3
I ran with debug and I get:


[Parent 15601, QuotaManager IO] WARNING: An unexpected file exists in the storage area: "~/.mozilla/firefox/x0wifmbi.default/storage/default/moz-extension+++a38e6f78-f775-4371-979f-394e085c1e28/idb/3350367306aesgyanrcoSt.": file ~/hg/mozilla-central/dom/indexedDB/ActorsParent.cpp, line 18152
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/indexedDB/ActorsParent.cpp, line 17740
[Parent 15601, QuotaManager IO] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x8000FFFF: file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 4365
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 4277
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 5248
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 5168
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 5130
[Parent 15601, IPDL Background] WARNING: 'aRv.Failed()', file ~/hg/mozilla-central/dom/cache/CacheOpParent.cpp, line 171
[Child 15762, Main Thread] WARNING: 'aRv.Failed()', file ~/hg/mozilla-central/dom/cache/CacheOpChild.cpp, line 114
[Child 15762, Main Thread] WARNING: Could not open the existing cache.: file ~/hg/mozilla-central/dom/workers/ServiceWorkerScriptCache.cpp, line 1345
[Child 15762, Main Thread] WARNING: 'NS_FAILED(aStatus)', file ~/hg/mozilla-central/dom/workers/ServiceWorkerUpdateJob.cpp, line 355
[Parent 15601, QuotaManager IO] WARNING: An unexpected file exists in the storage area: "~/.mozilla/firefox/x0wifmbi.default/storage/default/moz-extension+++a38e6f78-f775-4371-979f-394e085c1e28/idb/3350367306aesgyanrcoSt.": file ~/hg/mozilla-central/dom/indexedDB/ActorsParent.cpp, line 18152
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/indexedDB/ActorsParent.cpp, line 17740
[Parent 15601, QuotaManager IO] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x8000FFFF: file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 4365
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 4277
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 5248
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 5168
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 5130
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/indexedDB/ActorsParent.cpp, line 21649
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/indexedDB/ActorsParent.cpp, line 21493
[Parent 15601, IPDL Background] WARNING: Converting non-IndexedDB error code (0x8000FFFF) to NS_ERROR_DOM_INDEXEDDB_UNKNOWN_ERR: file ~/hg/mozilla-central/dom/indexedDB/ActorsParent.cpp, line 593
--DOCSHELL 0x7fdd7d91c000 == 16 [pid = 15601] [id = {1d4cc73a-e6c9-44c1-9dcb-60728fac3ac3}]
[Child 15678, Main Thread] WARNING: NS_ENSURE_TRUE(!mHasOrHasHadOwnerWindow || mOwnerWindow) failed: file ~/hg/mozilla-central/obj-x86_64-pc-linux-gnu/dist/include/mozilla/DOMEventTargetHelper.h, line 128
[Parent 15601, Main Thread] WARNING: NS_ENSURE_TRUE(aSecondURI) failed: file ~/hg/mozilla-central/dom/base/ThirdPartyUtil.cpp, line 98
Promise resolved after context unloaded
[Parent 15601, Main Thread] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80070057: file ~/hg/mozilla-central/netwerk/base/nsChannelClassifier.cpp, line 344
[Parent 15601, QuotaManager IO] WARNING: An unexpected file exists in the storage area: "~/.mozilla/firefox/x0wifmbi.default/storage/default/moz-extension+++a38e6f78-f775-4371-979f-394e085c1e28/idb/3350367306aesgyanrcoSt.": file ~/hg/mozilla-central/dom/indexedDB/ActorsParent.cpp, line 18152
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/indexedDB/ActorsParent.cpp, line 17740
[Parent 15601, QuotaManager IO] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x8000FFFF: file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 4365
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 4277
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 5248
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 5168
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 5130
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/indexedDB/ActorsParent.cpp, line 21649
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/indexedDB/ActorsParent.cpp, line 21493
[Parent 15601, IPDL Background] WARNING: Converting non-IndexedDB error code (0x8000FFFF) to NS_ERROR_DOM_INDEXEDDB_UNKNOWN_ERR: file ~/hg/mozilla-central/dom/indexedDB/ActorsParent.cpp, line 593
JavaScript error: undefined, line 0: Error: Frame not found, or missing host permission
[Parent 15601, Main Thread] WARNING: NS_ENSURE_TRUE(aSecondURI) failed: file ~/hg/mozilla-central/dom/base/ThirdPartyUtil.cpp, line 98
[Parent 15601, Main Thread] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80004005: file ~/hg/mozilla-central/chrome/nsChromeRegistry.cpp, line 210
++DOMWINDOW == 11 (0x7efcd552c000) [pid = 15728] [serial = 11] [outer = 0x7efce1aa19c0]
[Parent 15601, Main Thread] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80004005: file ~/hg/mozilla-central/chrome/nsChromeRegistry.cpp, line 210
[Child 15762, Main Thread] WARNING: '!aSelection->RangeCount()', file ~/hg/mozilla-central/editor/libeditor/EditorBase.cpp, line 4069
[Parent 15601, Main Thread] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80004005: file ~/hg/mozilla-central/chrome/nsChromeRegistry.cpp, line 210
[Parent 15601, Main Thread] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80004005: file ~/hg/mozilla-central/chrome/nsChromeRegistry.cpp, line 210
++DOMWINDOW == 12 (0x7efcd552e400) [pid = 15728] [serial = 12] [outer = 0x7efce1aa17d0]
--DOCSHELL 0x7fdd63cc7800 == 15 [pid = 15601] [id = {2751bac9-2641-40da-86e7-852419336290}]
[Parent 15601, QuotaManager IO] WARNING: An unexpected file exists in the storage area: "~/.mozilla/firefox/x0wifmbi.default/storage/default/moz-extension+++a38e6f78-f775-4371-979f-394e085c1e28/idb/3350367306aesgyanrcoSt.": file ~/hg/mozilla-central/dom/indexedDB/ActorsParent.cpp, line 18152
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/indexedDB/ActorsParent.cpp, line 17740
[Parent 15601, QuotaManager IO] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x8000FFFF: file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 4365
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 4277
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 5248
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 5168
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 5130
[Parent 15601, IPDL Background] WARNING: 'aRv.Failed()', file ~/hg/mozilla-central/dom/cache/CacheOpParent.cpp, line 171
[Child 15745, Main Thread] WARNING: 'aRv.Failed()', file ~/hg/mozilla-central/dom/cache/CacheOpChild.cpp, line 114
[Child 15745, Main Thread] WARNING: Could not open the existing cache.: file ~/hg/mozilla-central/dom/workers/ServiceWorkerScriptCache.cpp, line 1345
[Child 15745, Main Thread] WARNING: 'NS_FAILED(aStatus)', file ~/hg/mozilla-central/dom/workers/ServiceWorkerUpdateJob.cpp, line 355
[Parent 15601, Main Thread] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80070057: file ~/hg/mozilla-central/netwerk/base/nsChannelClassifier.cpp, line 344
++DOMWINDOW == 13 (0x7f7ca801b800) [pid = 15678] [serial = 13] [outer = 0x7f7ca9488c30]
[Parent 15601, QuotaManager IO] WARNING: An unexpected file exists in the storage area: "~/.mozilla/firefox/x0wifmbi.default/storage/default/moz-extension+++a38e6f78-f775-4371-979f-394e085c1e28/idb/3350367306aesgyanrcoSt.": file ~/hg/mozilla-central/dom/indexedDB/ActorsParent.cpp, line 18152
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/indexedDB/ActorsParent.cpp, line 17740
[Parent 15601, QuotaManager IO] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x8000FFFF: file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 4365
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 4277
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 5248
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 5168
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 5130
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/indexedDB/ActorsParent.cpp, line 21649
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/indexedDB/ActorsParent.cpp, line 21493
[Parent 15601, IPDL Background] WARNING: Converting non-IndexedDB error code (0x8000FFFF) to NS_ERROR_DOM_INDEXEDDB_UNKNOWN_ERR: file ~/hg/mozilla-central/dom/indexedDB/ActorsParent.cpp, line 593
JavaScript error: https://platform.twitter.com/widgets.js, line 9: Error: Network error.
Promise resolved while context is inactive
Promise resolved after context unloaded
JavaScript error: https://platform.twitter.com/widgets.js, line 9: Error: Network error.
[Parent 15601, Main Thread] WARNING: NS_ENSURE_TRUE(aSecondURI) failed: file ~/hg/mozilla-central/dom/base/ThirdPartyUtil.cpp, line 98
[Parent 15601, QuotaManager IO] WARNING: An unexpected file exists in the storage area: "~/.mozilla/firefox/x0wifmbi.default/storage/default/moz-extension+++a38e6f78-f775-4371-979f-394e085c1e28/idb/3350367306aesgyanrcoSt.": file ~/hg/mozilla-central/dom/indexedDB/ActorsParent.cpp, line 18152
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/indexedDB/ActorsParent.cpp, line 17740
[Parent 15601, QuotaManager IO] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x8000FFFF: file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 4365
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 4277
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 5248
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 5168
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 5130
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/indexedDB/ActorsParent.cpp, line 21649
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/indexedDB/ActorsParent.cpp, line 21493
[Parent 15601, IPDL Background] WARNING: Converting non-IndexedDB error code (0x8000FFFF) to NS_ERROR_DOM_INDEXEDDB_UNKNOWN_ERR: file ~/hg/mozilla-central/dom/indexedDB/ActorsParent.cpp, line 593
--DOMWINDOW == 3 (0x7f7962aa1bb0) [pid = 15745] [serial = 4] [outer = (nil)] [url = about:blank]
++DOMWINDOW == 40 (0x7fdd640ab000) [pid = 15601] [serial = 67] [outer = 0x7fdd8c196e90]
++DOCSHELL 0x7fdd7f828000 == 20 [pid = 15601] [id = {00f3eb24-1e8a-453a-a2fa-1a7d7efdf719}]
++DOMWINDOW == 41 (0x7fdd7d9d98c0) [pid = 15601] [serial = 68] [outer = (nil)]
++DOMWINDOW == 42 (0x7fdd7f884800) [pid = 15601] [serial = 69] [outer = 0x7fdd7d9d98c0]
++DOMWINDOW == 43 (0x7fdd7fd62400) [pid = 15601] [serial = 70] [outer = 0x7fdd7d9d98c0]
[Parent 15601, Main Thread] WARNING: Failed to open external DTD: publicId "-//W3C//DTD SVG 1.1//EN" systemId "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd" base "jar:file://~/.mozilla/firefox/x0wifmbi.default/extensions/notes@mozilla.com.xpi!/sidebar/close.svg" URL "resource://gre/res/dtd/svg11.dtd": file ~/hg/mozilla-central/parser/htmlparser/nsExpatDriver.cpp, line 588
[Parent 15601, Main Thread] WARNING: '!aSelection->RangeCount()', file ~/hg/mozilla-central/editor/libeditor/EditorBase.cpp, line 4069
[Parent 15601, QuotaManager IO] WARNING: An unexpected file exists in the storage area: "~/.mozilla/firefox/x0wifmbi.default/storage/default/moz-extension+++a38e6f78-f775-4371-979f-394e085c1e28/idb/3350367306aesgyanrcoSt.": file ~/hg/mozilla-central/dom/indexedDB/ActorsParent.cpp, line 18152
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/indexedDB/ActorsParent.cpp, line 17740
[Parent 15601, QuotaManager IO] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x8000FFFF: file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 4365
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 4277
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 5248
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 5168
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 5130
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/indexedDB/ActorsParent.cpp, line 21649
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/indexedDB/ActorsParent.cpp, line 21493
[Parent 15601, IPDL Background] WARNING: Converting non-IndexedDB error code (0x8000FFFF) to NS_ERROR_DOM_INDEXEDDB_UNKNOWN_ERR: file ~/hg/mozilla-central/dom/indexedDB/ActorsParent.cpp, line 593
--DOMWINDOW == 2 (0x7f7955a9c000) [pid = 15745] [serial = 7] [outer = (nil)] [url = about:blank]
[Parent 15601, Main Thread] WARNING: Partial match in a table without a valid completer, ignoring partial match.: file ~/hg/mozilla-central/toolkit/components/url-classifier/nsUrlClassifierDBService.cpp, line 1140
[Parent 15601, Main Thread] WARNING: Partial match in a table without a valid completer, ignoring partial match.: file ~/hg/mozilla-central/toolkit/components/url-classifier/nsUrlClassifierDBService.cpp, line 1140
[Parent 15601, Main Thread] WARNING: Partial match in a table without a valid completer, ignoring partial match.: file ~/hg/mozilla-central/toolkit/components/url-classifier/nsUrlClassifierDBService.cpp, line 1140
[Parent 15601, Main Thread] WARNING: Partial match in a table without a valid completer, ignoring partial match.: file ~/hg/mozilla-central/toolkit/components/url-classifier/nsUrlClassifierDBService.cpp, line 1140
[Parent 15601, Main Thread] WARNING: Partial match in a table without a valid completer, ignoring partial match.: file ~/hg/mozilla-central/toolkit/components/url-classifier/nsUrlClassifierDBService.cpp, line 1140
[Parent 15601, QuotaManager IO] WARNING: An unexpected file exists in the storage area: "~/.mozilla/firefox/x0wifmbi.default/storage/default/moz-extension+++a38e6f78-f775-4371-979f-394e085c1e28/idb/3350367306aesgyanrcoSt.": file ~/hg/mozilla-central/dom/indexedDB/ActorsParent.cpp, line 18152
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/indexedDB/ActorsParent.cpp, line 17740
[Parent 15601, QuotaManager IO] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x8000FFFF: file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 4365
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 4277
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 5248
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 5168
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 5130
[Parent 15601, IPDL Background] WARNING: 'aRv.Failed()', file ~/hg/mozilla-central/dom/cache/CacheOpParent.cpp, line 171
[Child 15762, Main Thread] WARNING: 'aRv.Failed()', file ~/hg/mozilla-central/dom/cache/CacheOpChild.cpp, line 114
[Child 15762, Main Thread] WARNING: Could not open the existing cache.: file ~/hg/mozilla-central/dom/workers/ServiceWorkerScriptCache.cpp, line 1345
[Child 15762, Main Thread] WARNING: 'NS_FAILED(aStatus)', file ~/hg/mozilla-central/dom/workers/ServiceWorkerUpdateJob.cpp, line 355
[Parent 15601, Main Thread] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80070057: file ~/hg/mozilla-central/netwerk/base/nsChannelClassifier.cpp, line 344
[Parent 15601, Main Thread] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80070057: file ~/hg/mozilla-central/netwerk/base/nsChannelClassifier.cpp, line 344
[Parent 15601, Main Thread] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80070057: file ~/hg/mozilla-central/netwerk/base/nsChannelClassifier.cpp, line 344
[Parent 15601, Main Thread] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80070057: file ~/hg/mozilla-central/netwerk/base/nsChannelClassifier.cpp, line 344
++DOMWINDOW == 44 (0x7fdd641e6800) [pid = 15601] [serial = 71] [outer = 0x7fdd8c196e90]
[Parent 15601, Main Thread] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80070057: file ~/hg/mozilla-central/netwerk/base/nsChannelClassifier.cpp, line 344
[Parent 15601, Main Thread] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80070057: file ~/hg/mozilla-central/netwerk/base/nsChannelClassifier.cpp, line 344
[Parent 15601, QuotaManager IO] WARNING: An unexpected file exists in the storage area: "~/.mozilla/firefox/x0wifmbi.default/storage/default/moz-extension+++a38e6f78-f775-4371-979f-394e085c1e28/idb/3350367306aesgyanrcoSt.": file ~/hg/mozilla-central/dom/indexedDB/ActorsParent.cpp, line 18152
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/indexedDB/ActorsParent.cpp, line 17740
[Parent 15601, QuotaManager IO] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x8000FFFF: file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 4365
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 4277
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 5248
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 5168
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 5130
[Parent 15601, IPDL Background] WARNING: 'aRv.Failed()', file ~/hg/mozilla-central/dom/cache/CacheOpParent.cpp, line 171
[Child 15762, Main Thread] WARNING: 'aRv.Failed()', file ~/hg/mozilla-central/dom/cache/CacheOpChild.cpp, line 114
[Child 15762, Main Thread] WARNING: Could not open the existing cache.: file ~/hg/mozilla-central/dom/workers/ServiceWorkerScriptCache.cpp, line 1345
[Child 15762, Main Thread] WARNING: 'NS_FAILED(aStatus)', file ~/hg/mozilla-central/dom/workers/ServiceWorkerUpdateJob.cpp, line 355
[Parent 15601, QuotaManager IO] WARNING: An unexpected file exists in the storage area: "~/.mozilla/firefox/x0wifmbi.default/storage/default/moz-extension+++a38e6f78-f775-4371-979f-394e085c1e28/idb/3350367306aesgyanrcoSt.": file ~/hg/mozilla-central/dom/indexedDB/ActorsParent.cpp, line 18152
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/indexedDB/ActorsParent.cpp, line 17740
[Parent 15601, QuotaManager IO] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x8000FFFF: file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 4365
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 4277
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 5248
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 5168
[Parent 15601, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file ~/hg/mozilla-central/dom/quota/ActorsParent.cpp, line 5130
[Parent 15601, IPDL Background] WARNING: 'aRv.Failed()', file ~/hg/mozilla-central/dom/cache/CacheOpParent.cpp, line 171
[Child 15762, Main Thread] WARNING: 'aRv.Failed()', file ~/hg/mozilla-central/dom/cache/CacheOpChild.cpp, line 114
[Child 15762, Main Thread] WARNING: Could not open the existing cache.: file ~/hg/mozilla-central/dom/workers/ServiceWorkerScriptCache.cpp, line 1345
[Child 15762, Main Thread] WARNING: 'NS_FAILED(aStatus)', file ~/hg/mozilla-central/dom/workers/ServiceWorkerUpdateJob.cpp, line 355
(In reply to Rémy Hubscher (:natim) from comment #5)
> I ran with debug and I get:
> 
> 
> [Parent 15601, QuotaManager IO] WARNING: An unexpected file exists in the
> storage area:
> "~/.mozilla/firefox/x0wifmbi.default/storage/default/moz-
> extension+++a38e6f78-f775-4371-979f-394e085c1e28/idb/3350367306aesgyanrcoSt.

Wow, can you post full directory listing of ""~/.mozilla/firefox/x0wifmbi.default/storage/default/moz-extension+++a38e6f78-f775-4371-979f-394e085c1e28/idb" ?

Also, what is the name of IDB database ?
> ls ~/.mozilla/firefox/x0wifmbi.default/storage/default/moz-extension+++a38e6f78-f775-4371-979f-394e085c1e28/idb
> 
> 3350367306aesgyanrcoSt.files  3350367306aesgyanrcoSt.sqlite 3350367306aesgyanrcoSt.
> 

> what is the name of IDB database ?

I assume it would be something like `default/notes` but I wasn't able to check in the debug tools of the extension debugger.
Depends on: 1425533
Ok, so the question is why "3350367306aesgyanrcoSt." exists, it shouldn't ...
Is it a directory or file ?
If it's a file, what's inside ?
Yes it is an empty file.
If I remove it it starts working again.
Empty file ? Hm, I don't know why we would create that. We 3350367306aesgyanrcoSt.files directory for stored blobs and 3350367306aesgyanrcoSt.sqlite file for actual database.
We used to have a bug when the directory name was ending with "." which was a problem on windows and we fixed it by using the ".files" suffix.

Is it possible that something else creates this file ?
(In reply to Jan Varga [:janv] from comment #12)
> Empty file ? Hm, I don't know why we would create that. We
We create ...
> 3350367306aesgyanrcoSt.files directory for stored blobs and
(In reply to Rémy Hubscher (:natim) from comment #8)
> I assume it would be something like `default/notes` but I wasn't able to
> check in the debug tools of the extension debugger.

Can you find out what exactly is passed to indexedDB.open() ?
Attached image demo.png — 
Apparently it is called: default/notes
We have multiple users affected by this issue. I will try to get them to comment more in https://github.com/mozilla/notes/issues/512 to get details
It appears some Lockbox users are running into a very similar issue, and have the same struggles troubleshooting it: https://github.com/mozilla-lockbox/lockbox-extension/issues/428
To be honest I am really worried about this.

It might be a bug when closing the add-on while the sqlite database is still open and not flushed? But as Jan told me this is just guessing and we need to figure out a way to debug it.

Can we ship a better error logging story in Firefox 58? (IndexedDB open, corrupted file, etc)
> The operation failed for reasons unrelated to the database itself and not covered by any other error code.
> actorsParent.cpp#599

Jan: Any idea what that could be?
IDB implementation normally creates 3350367306aesgyanrcoSt.files directory and 3350367306aesgyanrcoSt.sqlite database.
The empty file "3350367306aesgyanrcoSt." shouldn't exist and if find an unknown file ("3350367306aesgyanrcoSt." in this case) we fail the open operation.
Yes I don't think the error people are having is the same (i.e the file)
(In reply to Rémy Hubscher (:natim) from comment #15)
> Created attachment 8937273 [details]
> demo.png
> 
> Apparently it is called: default/notes

When I try to open a database called "default/notes" I get this physical database:
4213852584dseeftaounlFt2%.sqlite

Are you sure it's "default/notes" ?
According to the screenshot it is isn't it? Let me try to use a console.log where we open it in the add-on.
> console.log("IndexedDB", "'"+ this.dbname+"'");

IndexedDB 'default/notes' kinto.js:5546:7
And yes I have this file: 
> ./foobar.default/storage/default/moz-extension+++8307a5d6-421c-4cd7-a1e0-c16629c59117/idb/4213852584dseeftaounlFt2%.sqlite
Can you send me 3350367306aesgyanrcoSt.sqlite ?
Or open it in sqlite3 and do "select name from database;"
And paste the output here.
Ok, I reverse engineered the name, it's "asyncStorage".
When I create it locally using indexedDB.open(), I see two directoru entries:
3350367306aesgyanrcoSt.files
3350367306aesgyanrcoSt.sqlite
which is expected behavior.

I guess an extension or the user manually creates "3350367306aesgyanrcoSt." which then breaks IDB initialization.
> I guess an extension or the user manually creates "3350367306aesgyanrcoSt." which then breaks IDB initialization.

Yes for this one, I tend to agree. But apparently other users don't have the file ending with a dot.
So you want to cover it all here ?
Yes at least being able to have some debug information to help people when they encounter an error with IndexedDB.
What I could do here is to expose the "An unexpected file exists in the storage area" message to the browser console.
In the future, if someone has broken quotamanager-controlled storage, they can check if things are just broken for them with https://firefox-storage-test.glitch.me/.
I'm one of the people for whom this happens with Lockbox.  The above site yields "Totally Broken".  Is there a way to recover from this that doesn't involve a new profile?


Overview:
Storage is broken. This is the same version (59) as the last time you loaded this page.
Specific Subsystem Statuses:

LocalStorage
    Good: Totally Working. (fullyOperational)
QuotaManager
    Bad: Totally Broken. (fullyBroken)
IndexedDB
    Bad: Totally Broken. (fullyBroken)
Cache API
    Bad: Totally Broken. (fullyBroken)

Debug Info:

storage.estimate() threw: The expression cannot be converted to return the specified type.
Failed to create "persistent" IDB.
Failed to create "transient" IDB.

{
  "v": 1,
  "curVersion": 59,
  "prevVersion": 59,
  "ls": {},
  "qm": {
    "lastWorkedIn": 0
  },
  "idb": {
    "persistentCreatedIn": 0,
    "persistentLastOpenedIn": 0,
    "clearDetectedIn": 0
  },
  "cache": {
    "firstCacheCreatedIn": 0,
    "unpaddedOpaqueCreatedIn": 0,
    "paddedOpaqueCreatedIn": 0
  }
}
Joe, for now, can you run a debug build and paste console output here, as Remy did in comment 5 ?
Here are the steps I followed to build a debug build on Linux:

1. Clone mozilla-central or gecko-dev:

> hg clone https://hg.mozilla.org/mozilla-central/ firefox
> cd firefox/

or 

> git clone https://github.com/mozilla/gecko-dev.git firefox
> cd firefox/

2. Download and run bootstrap.py to install system dependencies

> wget https://hg.mozilla.org/mozilla-central/raw-file/default/python/mozboot/bin/bootstrap.py
> python bootstrap.py

3. Create a .mozconfig file in the firefox/ cloned directory: (without the > at the start line)

> nano .mozconfig

> . ./browser/config/mozconfig
> ac_add_options --enable-debug

4. Run mach build

> ./mach build


5. Run mach run (make sure you stopped any running firefox first)

> ./mach run

or (to select the profile to run) 

> ./obj-x86_64-pc-linux-gnu/dist/bin/firefox -P


The logs Jan is talking about will be displayed in the console.
I've got a log file, but it's 125k, and I don't know if there are personal details in it.  What parts do you need?
Just the lines with "QuotaManager" string.
Here you go:


[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/cache/FileUtils.cpp, line 737
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(mozilla::dom::cache:: LockedDirectoryPaddingGet(dir, &paddingSize))', file /Users/jhildebrand/track/gecko-dev/dom/cache/QuotaClient.cpp, line 182
[Parent 10504, QuotaManager IO] WARNING: The SQL statement 'SELECT response_padding_size FROM entries WHERE response_padding_size IS NOT NULL;' could not be compiled due to an error: no such column: response_padding_size: file /Users/jhildebrand/track/gecko-dev/storage/mozStorageConnection.cpp, line 1260
[Parent 10504, QuotaManager IO] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80004005: file /Users/jhildebrand/track/gecko-dev/storage/mozStorageConnection.cpp, line 1827
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/cache/DBSchema.cpp, line 753
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/cache/FileUtils.cpp, line 906
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/cache/QuotaClient.cpp, line 103
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/cache/QuotaClient.cpp, line 184
[Parent 10504, QuotaManager IO] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80004005: file /Users/jhildebrand/track/gecko-dev/dom/quota/ActorsParent.cpp, line 4365
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/quota/ActorsParent.cpp, line 4277
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/quota/ActorsParent.cpp, line 5248
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/quota/ActorsParent.cpp, line 5168
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/quota/ActorsParent.cpp, line 5130
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/indexedDB/ActorsParent.cpp, line 21649
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/indexedDB/ActorsParent.cpp, line 21493
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/cache/FileUtils.cpp, line 737
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(mozilla::dom::cache:: LockedDirectoryPaddingGet(dir, &paddingSize))', file /Users/jhildebrand/track/gecko-dev/dom/cache/QuotaClient.cpp, line 182
[Parent 10504, QuotaManager IO] WARNING: The SQL statement 'SELECT response_padding_size FROM entries WHERE response_padding_size IS NOT NULL;' could not be compiled due to an error: no such column: response_padding_size: file /Users/jhildebrand/track/gecko-dev/storage/mozStorageConnection.cpp, line 1260
[Parent 10504, QuotaManager IO] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80004005: file /Users/jhildebrand/track/gecko-dev/storage/mozStorageConnection.cpp, line 1827
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/cache/DBSchema.cpp, line 753
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/cache/FileUtils.cpp, line 906
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/cache/QuotaClient.cpp, line 103
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/cache/QuotaClient.cpp, line 184
[Parent 10504, QuotaManager IO] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80004005: file /Users/jhildebrand/track/gecko-dev/dom/quota/ActorsParent.cpp, line 4365
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/quota/ActorsParent.cpp, line 4277
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/quota/ActorsParent.cpp, line 5248
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/quota/ActorsParent.cpp, line 5168
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/quota/ActorsParent.cpp, line 5130
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/indexedDB/ActorsParent.cpp, line 21649
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/indexedDB/ActorsParent.cpp, line 21493
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/cache/FileUtils.cpp, line 737
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(mozilla::dom::cache:: LockedDirectoryPaddingGet(dir, &paddingSize))', file /Users/jhildebrand/track/gecko-dev/dom/cache/QuotaClient.cpp, line 182
[Parent 10504, QuotaManager IO] WARNING: The SQL statement 'SELECT response_padding_size FROM entries WHERE response_padding_size IS NOT NULL;' could not be compiled due to an error: no such column: response_padding_size: file /Users/jhildebrand/track/gecko-dev/storage/mozStorageConnection.cpp, line 1260
[Parent 10504, QuotaManager IO] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80004005: file /Users/jhildebrand/track/gecko-dev/storage/mozStorageConnection.cpp, line 1827
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/cache/DBSchema.cpp, line 753
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/cache/FileUtils.cpp, line 906
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/cache/QuotaClient.cpp, line 103
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/cache/QuotaClient.cpp, line 184
[Parent 10504, QuotaManager IO] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80004005: file /Users/jhildebrand/track/gecko-dev/dom/quota/ActorsParent.cpp, line 4365
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/quota/ActorsParent.cpp, line 4277
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/quota/ActorsParent.cpp, line 5248
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/quota/ActorsParent.cpp, line 5168
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/quota/ActorsParent.cpp, line 5130
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/indexedDB/ActorsParent.cpp, line 21649
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/indexedDB/ActorsParent.cpp, line 21493
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/cache/FileUtils.cpp, line 737
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(mozilla::dom::cache:: LockedDirectoryPaddingGet(dir, &paddingSize))', file /Users/jhildebrand/track/gecko-dev/dom/cache/QuotaClient.cpp, line 182
[Parent 10504, QuotaManager IO] WARNING: The SQL statement 'SELECT response_padding_size FROM entries WHERE response_padding_size IS NOT NULL;' could not be compiled due to an error: no such column: response_padding_size: file /Users/jhildebrand/track/gecko-dev/storage/mozStorageConnection.cpp, line 1260
[Parent 10504, QuotaManager IO] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80004005: file /Users/jhildebrand/track/gecko-dev/storage/mozStorageConnection.cpp, line 1827
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/cache/DBSchema.cpp, line 753
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/cache/FileUtils.cpp, line 906
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/cache/QuotaClient.cpp, line 103
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/cache/QuotaClient.cpp, line 184
[Parent 10504, QuotaManager IO] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80004005: file /Users/jhildebrand/track/gecko-dev/dom/quota/ActorsParent.cpp, line 4365
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/quota/ActorsParent.cpp, line 4277
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/quota/ActorsParent.cpp, line 5248
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/quota/ActorsParent.cpp, line 7106
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/quota/ActorsParent.cpp, line 6239
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/quota/ActorsParent.cpp, line 6103
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/cache/FileUtils.cpp, line 737
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(mozilla::dom::cache:: LockedDirectoryPaddingGet(dir, &paddingSize))', file /Users/jhildebrand/track/gecko-dev/dom/cache/QuotaClient.cpp, line 182
[Parent 10504, QuotaManager IO] WARNING: The SQL statement 'SELECT response_padding_size FROM entries WHERE response_padding_size IS NOT NULL;' could not be compiled due to an error: no such column: response_padding_size: file /Users/jhildebrand/track/gecko-dev/storage/mozStorageConnection.cpp, line 1260
[Parent 10504, QuotaManager IO] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80004005: file /Users/jhildebrand/track/gecko-dev/storage/mozStorageConnection.cpp, line 1827
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/cache/DBSchema.cpp, line 753
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/cache/FileUtils.cpp, line 906
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/cache/QuotaClient.cpp, line 103
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/cache/QuotaClient.cpp, line 184
[Parent 10504, QuotaManager IO] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80004005: file /Users/jhildebrand/track/gecko-dev/dom/quota/ActorsParent.cpp, line 4365
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/quota/ActorsParent.cpp, line 4277
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/quota/ActorsParent.cpp, line 5248
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/quota/ActorsParent.cpp, line 5168
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/quota/ActorsParent.cpp, line 5130
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/indexedDB/ActorsParent.cpp, line 21649
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/indexedDB/ActorsParent.cpp, line 21493
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/cache/FileUtils.cpp, line 737
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(mozilla::dom::cache:: LockedDirectoryPaddingGet(dir, &paddingSize))', file /Users/jhildebrand/track/gecko-dev/dom/cache/QuotaClient.cpp, line 182
[Parent 10504, QuotaManager IO] WARNING: The SQL statement 'SELECT response_padding_size FROM entries WHERE response_padding_size IS NOT NULL;' could not be compiled due to an error: no such column: response_padding_size: file /Users/jhildebrand/track/gecko-dev/storage/mozStorageConnection.cpp, line 1260
[Parent 10504, QuotaManager IO] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80004005: file /Users/jhildebrand/track/gecko-dev/storage/mozStorageConnection.cpp, line 1827
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/cache/DBSchema.cpp, line 753
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/cache/FileUtils.cpp, line 906
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/cache/QuotaClient.cpp, line 103
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/cache/QuotaClient.cpp, line 184
[Parent 10504, QuotaManager IO] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80004005: file /Users/jhildebrand/track/gecko-dev/dom/quota/ActorsParent.cpp, line 4365
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/quota/ActorsParent.cpp, line 4277
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/quota/ActorsParent.cpp, line 5248
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/quota/ActorsParent.cpp, line 5168
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/quota/ActorsParent.cpp, line 5130
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/indexedDB/ActorsParent.cpp, line 21649
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/indexedDB/ActorsParent.cpp, line 21493
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/cache/FileUtils.cpp, line 737
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(mozilla::dom::cache:: LockedDirectoryPaddingGet(dir, &paddingSize))', file /Users/jhildebrand/track/gecko-dev/dom/cache/QuotaClient.cpp, line 182
[Parent 10504, QuotaManager IO] WARNING: The SQL statement 'SELECT response_padding_size FROM entries WHERE response_padding_size IS NOT NULL;' could not be compiled due to an error: no such column: response_padding_size: file /Users/jhildebrand/track/gecko-dev/storage/mozStorageConnection.cpp, line 1260
[Parent 10504, QuotaManager IO] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80004005: file /Users/jhildebrand/track/gecko-dev/storage/mozStorageConnection.cpp, line 1827
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/cache/DBSchema.cpp, line 753
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/cache/FileUtils.cpp, line 906
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/cache/QuotaClient.cpp, line 103
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/cache/QuotaClient.cpp, line 184
[Parent 10504, QuotaManager IO] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80004005: file /Users/jhildebrand/track/gecko-dev/dom/quota/ActorsParent.cpp, line 4365
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/quota/ActorsParent.cpp, line 4277
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/quota/ActorsParent.cpp, line 5248
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/quota/ActorsParent.cpp, line 5168
[Parent 10504, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /Users/jhildebrand/track/gecko-dev/dom/quota/ActorsParent.cpp, line 5130
Ok, so it seems to be a DOM cache issue. response_padding_size column doesn't exist.
This is fallout from the great bug 1404344 "profile from the future" compensating "make it possible to downgrade from 57 to 56" hack.  It seems almost certain that the profile from comment 39 was used in 56 after being used in a hack-fixed-57.  The GetUsageForOrigin accesses the DB at a slightly lower level than if it was opened using the API, bypassing the upgrade logic and any schema version guards.  That made sense for the code because the patch was inducing the (downgrade-breaking) schema version change which would run prior to getting the origin usage.  I can provide a minor fix to the cache quotaclient to check the (hack-corrected schema version) so it can know that if it's not the right version that it shouldn't try and use the DB.

That may want to be spun off to a different bug since that sounds different from the badly named file.

And of course all of this wants QM to stop failing to initialize when it experiences a problem with an origin.
NI on me for post-holiday fix.
Flags: needinfo?(bugmail)
> Ok, so it seems to be a DOM cache issue. response_padding_size column doesn't exist.

Is there a way we can tell people to clear the cache and get back on track? A sqlite file to remove maybe?
Flags: needinfo?(jvarga)
Here are some more logs that another user was able to figure out:

[Parent 12625, Main Thread] WARNING: Failed to open external DTD: publicId "-//W3C//DTD SVG 1.1//EN" systemId "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd" base "jar:file:///home/fer/.mozilla/firefox/n476vrxq.sept2014/extensions/notes@mozilla.com.xpi!/sidebar/close.svg" URL "resource://gre/res/dtd/svg11.dtd": file /media/creaciones/aplicaciones_externas/firefox/parser/htmlparser/nsExpatDriver.cpp, line 588
[Parent 12625, Main Thread] WARNING: '!aSelection->RangeCount()', file /media/creaciones/aplicaciones_externas/firefox/editor/libeditor/EditorBase.cpp, line 4019
[Parent 12625, QuotaManager IO] WARNING: '!databaseFilenames.GetEntry(subdirNameBase)', file /media/creaciones/aplicaciones_externas/firefox/dom/indexedDB/ActorsParent.cpp, line 17758
[Parent 12625, QuotaManager IO] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x8000FFFF: file /media/creaciones/aplicaciones_externas/firefox/dom/quota/ActorsParent.cpp, line 4365
[Parent 12625, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /media/creaciones/aplicaciones_externas/firefox/dom/quota/ActorsParent.cpp, line 4277
[Parent 12625, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /media/creaciones/aplicaciones_externas/firefox/dom/quota/ActorsParent.cpp, line 5248
[Parent 12625, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /media/creaciones/aplicaciones_externas/firefox/dom/quota/ActorsParent.cpp, line 5168
[Parent 12625, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /media/creaciones/aplicaciones_externas/firefox/dom/quota/ActorsParent.cpp, line 5130
[Parent 12625, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /media/creaciones/aplicaciones_externas/firefox/dom/indexedDB/ActorsParent.cpp, line 21649
[Parent 12625, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /media/creaciones/aplicaciones_externas/firefox/dom/indexedDB/ActorsParent.cpp, line 21493
[Parent 12625, IPDL Background] WARNING: Converting non-IndexedDB error code (0x8000FFFF) to NS_ERROR_DOM_INDEXEDDB_UNKNOWN_ERR: file /media/creaciones/aplicaciones_externas/firefox/dom/indexedDB/ActorsParent.cpp, line 593
++DOMWINDOW == 7 (0x7fb305f4fc00) [pid = 12704] [serial = 13] [outer = 0x7fb321552010]
[Parent 12625, Main Thread] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80070057: file /media/creaciones/aplicaciones_externas/firefox/netwerk/base/nsChannelClassifier.cpp, line 344
Janv dou you have a link for a debug version of Firefox for Windows 10? Maybe from treeherder?
Will this be fixed by bug 1423917?
(In reply to Andrew Overholt [:overholt] from comment #46)
> Will this be fixed by bug 1423917?

(Or at least the cache padding issue)
Yes probably. I will let you know as soon as it lands.
(In reply to Andrew Overholt [:overholt] from comment #47)
> (In reply to Andrew Overholt [:overholt] from comment #46)
> > Will this be fixed by bug 1423917?
> 
> (Or at least the cache padding issue)

Unfortunately, they are two different issues.

Bug 1423917 is an upgrade failure in QuotaManager due to an unexpected directory for a chrome origin (not able to be parsed by OriginParser in QuotaManager).   

Comment 39 & 41 looks like a failure happened while QuotaManager was initializing origins & directories. QuotaManager couldn't find the padding file to get the padding size from a cache directory, so it tried to get the size from Cache DB. Then, it failed because the Cache DB (cache.sqlite) couldn't find the "response_padding_size" column.
Are either of those recoverable?  I wouldn't mind losing data, because my only other option is to trash my profile and lose *all* of my data.  I'll have to do something if I ever want to run Lockbox, and I don't want end users to end up in this situation one day.
(In reply to Joe Hildebrand  [:hildjj] (UTC-6) from comment #50)
> Are either of those recoverable?  I wouldn't mind losing data, because my
> only other option is to trash my profile and lose *all* of my data.  I'll
> have to do something if I ever want to run Lockbox, and I don't want end
> users to end up in this situation one day.

The most friendly option is about:support's "Refresh Firefox" option as documented at:
  https://support.mozilla.org/en-US/kb/refresh-firefox-reset-add-ons-and-settings
with the caveat that it doesn't bring all your data across, but it should bring your most important data across (passwords, bookmarks, other places contents).  It leaves your old profile on your disk, so if it turns out the refresh did leave some stuff behind, you can still mess with your profiles.ini or use -p to try and use your old profile or try the options below.

If you're comfortable nuking folders yourself, your options are, for each thing:

1) Bug 1423917 (not your problem, per comment 39): Remove any directory beneath PROFILE/storage/{permanent,temporary} that starts with "chrome++++".

2) Your comment 39 problem.  The overzealous fix is to remove any cache directory located at PROFILE/storage/{permanent,temporary}/*/cache.  Less overzealous is to not do that for directories with padding files in them.

Aside: You likely encountered this problem due to trying the beta versions of Firefox we were pushing.  Please see Bug 1373244 for the great hope to avoid this profile footgun.
Flags: needinfo?(bugmail)
Priority: P3 → P2
:asuth one of our users tried your manual nuking:

- I can verify that there are no folders beneath PROFILE/storage/{permanent,temporary} that start with "chrome++++".

- There are no folders named "cache" under PROFILE/storage/{permanent,temporary}/*. The only folders are named "idb" (under PROFILE/storage/permanent/*) or "asmjs" (under PROFILE/storage/temporary/*).

Additionally, I've never used a beta version of Firefox on this PC. So that couldn't be the cause in my case.

Do you have a link for a debug version of Firefox on Windows that the user could use to help us get more logs?
For latest release build:
1. Go to https://treeherder.mozilla.org/#/jobs?repo=mozilla-release
2. Check the latest changeset (in the top)
3. Find "Windows 2012 debug" or "Windows 2012 x64 debug" row
4. Click on "tc (+3)"
5. Click on "B"
6. Find "artifact uploaded: target.zip" in the "Job Details" tab
7. Click on target.zip
8. Download target.zip
9. Unpack target.zip
Flags: needinfo?(jvarga)
Thanks Janv, I will get back to you with the logs :)
I also have the following logs from another user:

[Parent 16916, Main Thread] WARNING: Need TabChild to get the nativeWindow from!: file z:/build/build/src/widget/PuppetWidget.cpp, line 1212
[Parent 16916, Main Thread] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80070057: file z:/build/build/src/netwerk/base/nsChannelClassifier.cpp, line 344
[Parent 16916, Main Thread] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80070057: file z:/build/build/src/netwerk/base/nsChannelClassifier.cpp, line 344
[Parent 16916, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file z:/build/build/src/dom/cache/FileUtils.cpp, line 739
[Parent 16916, Main Thread] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80070057: file z:/build/build/src/netwerk/base/nsChannelClassifier.cpp, line 344
[Parent 16916, QuotaManager IO] WARNING: 'NS_FAILED(mozilla::dom::cache:: LockedDirectoryPaddingGet(dir, &paddingSize))', file z:/build/build/src/dom/cache/QuotaClient.cpp, line 179
[Parent 16916, Main Thread] WARNING: Suboptimal indexes for the SQL statement 0x25f8f2141c0 (http://mzl.la/1FuID0j).: file z:/build/build/src/storage/mozStoragePrivateHelpers.cpp, line 114
[Parent 16916, QuotaManager IO] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80520015: file z:/build/build/src/storage/mozStorageService.cpp, line 721
[Parent 16916, Main Thread] WARNING: Suboptimal indexes for the SQL statement 0x25f8f214090 (http://mzl.la/1FuID0j).: file z:/build/build/src/storage/mozStoragePrivateHelpers.cpp, line 114
[Parent 16916, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file z:/build/build/src/dom/cache/DBAction.cpp, line 234
[Parent 16916, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file z:/build/build/src/dom/cache/QuotaClient.cpp, line 94
[Parent 16916, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file z:/build/build/src/dom/cache/QuotaClient.cpp, line 181
[Parent 16916, QuotaManager IO] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80520015: file z:/build/build/src/dom/quota/ActorsParent.cpp, line 4365
[Parent 16916, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file z:/build/build/src/dom/quota/ActorsParent.cpp, line 4277
[Parent 16916, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file z:/build/build/src/dom/quota/ActorsParent.cpp, line 5248
[Parent 16916, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file z:/build/build/src/dom/quota/ActorsParent.cpp, line 5168
[Parent 16916, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file z:/build/build/src/dom/quota/ActorsParent.cpp, line 5130
[Parent 16916, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file z:/build/build/src/dom/indexedDB/ActorsParent.cpp, line 21648
[Parent 16916, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file z:/build/build/src/dom/indexedDB/ActorsParent.cpp, line 21492
Could reproduce this bug today, having the following logs : 

[Parent 28239, QuotaManager IO] WARNING: 'NS_FAILED(rv)', file /builds/worker/workspace/build/src/dom/cache/FileUtils.cpp, line 739
[Parent 28239, QuotaManager IO] WARNING: 'NS_FAILED(mozilla::dom::cache:: LockedDirectoryPaddingGet(dir, &paddingSize))', file /builds/worker/workspace/build/src/dom/cache/QuotaClient.cpp, line 179
[Parent 28239, QuotaManager IO] WARNING: The SQL statement 'SELECT response_padding_size FROM entries WHERE response_padding_size IS NOT NULL;' could not be compiled due to an error: no such column: response_padding_size: file /builds/worker/workspace/build/src/storage/mozStorageConnection.cpp, line 1205
[Parent 28239, QuotaManager IO] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80004005: file /builds/worker/workspace/build/src/storage/mozStorageConnection.cpp, line 1728
> Comment 39 & 41 looks like a failure happened while QuotaManager was
> initializing origins & directories. QuotaManager couldn't find the padding
> file to get the padding size from a cache directory, so it tried to get the
> size from Cache DB. Then, it failed because the Cache DB (cache.sqlite)
> couldn't find the "response_padding_size" column.

Can we safely remove the caches.sqlite file? How do we know which one is broken?
Flags: needinfo?(shes050117)
me on mbp.local in ~/Library/Application Support/Firefox/Profiles/vbjl4t4j.Default User
$ find . -name '*.sqlite'
./chrome_debugger_profile/webappsstore.sqlite
./chrome_debugger_profile/cookies.sqlite
./chrome_debugger_profile/storage/permanent/chrome/idb/3561288849sdhlie.sqlite
./chrome_debugger_profile/permissions.sqlite
./chrome_debugger_profile/favicons.sqlite
./chrome_debugger_profile/storage.sqlite
./chrome_debugger_profile/places.sqlite
./kinto.sqlite
./webappsstore.sqlite
./cookies.sqlite
./formhistory.sqlite
./storage/default/http+++localhost+3000/idb/2595388511s3e2vne.sqlite
./storage/default/http+++localhost+3000/cache/caches.sqlite
./storage/permanent/moz-safe-about+home/idb/818200132aebmoouht.sqlite
./storage/permanent/chrome/idb/3561288849sdhlie.sqlite
./storage/permanent/chrome/idb/2029012401EexhtceanCspiuotnrSat.sqlite
./storage/permanent/chrome/idb/2918063365piupsah.sqlite
./storage/permanent/indexeddb+++fx-devtools/idb/478967115deegvatroootlss--cans.sqlite
./storage/permanent/indexeddb+++fx-devtools/idb/4268914080AsptpcPerjo.sqlite
./content-prefs.sqlite
./permissions.sqlite
./favicons.sqlite
./storage.sqlite
./places.sqlite

After deleting folder https+++localhost+3000 ($ rm -rf storage/default/http+++localhost+3000), add-on is back to normal.
Ok I tried to put together a script to detect broken cache files:


> #!/bin/bash
> for file in $(find "$1" -name 'caches.sqlite')
> do
> 	if ([ -d $(dirname "$file")/../idb ] && sqlite3 "$file" '.schema entries' | egrep -v response_padding_size | egrep -v entries_request_match_index > /dev/null)
> 	then
> 		echo "$file"
> 	fi
> done
> Can we safely remove the caches.sqlite file?

Apparently not because then an empty file is created and Firefox cannot find the entries table anymore.

Is there a better way than to remove the storage folder altogether?
(In reply to Rémy Hubscher (:natim) from comment #60)
> > Can we safely remove the caches.sqlite file?
> 
> Apparently not because then an empty file is created and Firefox cannot find
> the entries table anymore.
> 
> Is there a better way than to remove the storage folder altogether?

You should be able to remove it using storage preferences (UI).
:hildjj, :m_and_m, we have a new script that might help debug this if you have a broken profile. Please see https://github.com/mozilla/notes/issues/512#issuecomment-363822635
(In reply to Rémy Hubscher (:natim) from comment #57)
> Can we safely remove the caches.sqlite file? How do we know which one is
> broken?

The cache.sqlite file is bound with the files in morgue folder, so remove the whole cache folder is probably a good option. As :janv said in comment 61, the safest way is remove it by using the storage preference.

You could find it via finding error messages in the web console or debug messages. However, I'm not sure whether there is a better way to distinguish if the caches.sqlite is broken.

(In reply to Rémy Hubscher (:natim) from comment #60)
> > Can we safely remove the caches.sqlite file?
> 
> Apparently not because then an empty file is created and Firefox cannot find
> the entries table anymore.
> 
> Is there a better way than to remove the storage folder altogether?

Or you can remove the cache folder (PROFILE/storage/{permanent,temporary}/*/cache) as :asuth suggested in comment 51.
Flags: needinfo?(shes050117)
> Or you can remove the cache folder (PROFILE/storage/{permanent,temporary}/*/cache) as :asuth suggested in comment 51.

This is really useful I am going to try that.
Do you think we could detect this errors and ask Firefox to remove all cache folder to self recover from this issues?
Flags: needinfo?(jvarga)
Meeting notes:

```
Causes of the issue?

* Can be various
* Profile Switching between channels
* If you open page not using Localhost but file:/// protocol



Can we use LocalStorage instead?

* No, need transactions, etc



Can we detect that IndexedDB failed?

* Try the `onError`, it should throw an `Unknown Error`



Can we have a fix for this?

* The team is busy refactoring LocalStorage for Firefox 61, so any fixes to this issue would have to be done after that. (Only 3 people knowing the codebase and we other projects are going on)
* Lockbox targeted shipping day is May and needs the fix



What’s the workaround?

* Try deleting caches
* It is not risky to remove the DOM cache cache/caches.sqlite
* *Quit the browser* then remove all cache subfolder in: PROFILE/storage/{default,permanent,temporary}/*/cache
* Lockbox or Notes are **not** planning to modify their bootstrap scripts for workaround purposes

```

Please excuse me if I missed something or rephrased too much.
It seems this is caused mostly by users going up and down Firefox versions in the profile. It's a pain if you do that, but this is one of the many issues you'll encounter. Making the assumption that this is a rare event for release users.

At this point dropping it down in priority to a P3 from the Lockbox / FxA point of view.
Priority: P2 → P3
(In reply to Andy McKay [:andym] from comment #67)
> It seems this is caused mostly by users going up and down Firefox versions
> in the profile. It's a pain if you do that, but this is one of the many
> issues you'll encounter. Making the assumption that this is a rare event for
> release users.

Exactly because blaming users and throwing your testers under the bus is a great idea. So, yes https://firefox-storage-test.glitch.me/ claims that

LocalStorage
    Good: Totally Working. (fullyOperational)
QuotaManager
    Bad: Totally Broken. (fullyBroken)
IndexedDB
    Bad: Totally Broken. (fullyBroken)
Cache API
    Bad: Totally Broken. (fullyBroken)

and I have no idea, what to do to fix my profile. Wiping out 1.4GB profile is probably not a nice option, I have removed all zero-byte files in $PROFILE/storage/ directory, but of course it didn't help.

(coming here from https://github.com/KaiRo-at/lantea/issues/3 ).
So, yes, after rm -rf $PROFILE/storage/ I got https://lantea.kairo.at/ working again. Zip archive of my original (broken) storage/ directory could be available upon request (I won't put URL here not knowing what's there).
Thank you for your feedback Matěj.

I am glad you fixed your profile. We are planning to fix this bug as soon as we land some LocalStorage optimization that the QuotaManager team is working on.

Only a few people know how to fix this in Gecko and they have a lot on their plate so we are waiting.
See Also: → 1377539
It looks like the storage is totally broken...

Overview:
Storage is broken. This is your first visit or all storage was automatically cleared.
Specific Subsystem Statuses:

LocalStorage
Good: Totally Working. (fullyOperational)
QuotaManager
Good: Totally Working. (fullyOperational)
IndexedDB
Bad: Our test logic is broken, please copy and paste the contents of 'Debug Info' below and anything in the devtools console and send to :asuth. (unexpectedBreakage)
Cache API
Bad: Totally Broken. (fullyBroken)

Debug Info:

A mutation operation was attempted on a database that did not allow mutations.
The operation is insecure.

{
"v": 1,
"curVersion": 60,
"prevVersion": 0,
"ls": {},
"qm": {
"lastWorkedIn": 60
},
"idb": {
"persistentCreatedIn": 0,
"persistentLastOpenedIn": 0,
"clearDetectedIn": 0
},
"cache": {
"firstCacheCreatedIn": 0,
"unpaddedOpaqueCreatedIn": 0,
"paddedOpaqueCreatedIn": 0
}
}
Looks like my Notes broke again. All I get is a blank canvas with everything I had entered previously has disappeared. Any suggestions are welcome. Thank you.
See bug1455707 for one possible (forward-looking only!) mitigation.
Blocks: 1482662
Tom is back.
Assignee: nobody → shes050117
Flags: needinfo?(jvarga)
As Andrew's comment in comment #41, I also believe the problem originated from that. Some users upgraded from FF 56 directly to some version after FF57-hack.

To fix that, there are two things we need to address in this bug:
1. Create another DOM Cache upgrade to ensure all the cache.sqlite have the response_padding column.
2. Loosen the check while initializing Cache. Right now, it assumes all the directory must have a readable padding file or have padding column in cache.sqlite. Otherwise, it breaks the whole QM initialization. However, I think we might consider fixing the padding file and padding column or at least only break Cache while executing Cache's operation for that directory.
(In reply to Tom Tung [:tt] from comment #75)
> from that. Some users upgraded from FF 56 directly to some version after
I meant users had applied the hack in FF 56 and then upgraded to the version after that.
(In reply to Tom Tung [:tt] from comment #75)
> As Andrew's comment in comment #41, I also believe the problem originated
> from that. Some users upgraded from FF 56 directly to some version after
> FF57-hack.

Actually, I misunderstood the comment. It's because LockedGetPaddingSizeFromDB() doesn't check the db schema and users haven't upgraded the cache db yet. It assumed every user are at least upgraded their cache db so that having the response_padding_size col.

So the solution might be:
1. Let LockedGetPaddingSizeFromDB() run db::CreateOrMigrateSchema(aConn) before acutally checking the db.
2. Still the same, loosen the check while initializing Cache.
Attached patch mitigation and fix (obsolete) — — Splinter Review 
task 1
Attached patch A test to verify (obsolete) — — Splinter Review 

    
    

    
  

      
      
      
      

        Attached patch
          
          
        fix (obsolete)
        — 
          — Splinter Review
      

    


  
Added comment

        Attachment #9007250 -
        Attachment is obsolete: true

    
    

    
  

      
      
      
      

        Attached patch
          
          
        test (obsolete)
        — 
          — Splinter Review
      

    


  

        Attachment #9007252 -
        Attachment is obsolete: true

    
    

    
  


  
Elaborated the comment.

Hi Jan,

This patch mainly fixes the condition described in comment 39. The reason for failure was because somehow the padding file didn't exist or broken and the database schema for cache.sqlite is too old (response_padding_size hasn't been updated to the main table) so that it showed the error message and the origins initialization was broken.

An obvious fix should be ensuring cache.sqlite is update-to-date before accessing it, and this patch does that.

The following patch is a test to verify this case is captured and origins initialization doesn't fail in this case.

I also reckon for a midterm goal we might want to have a mechanism to mark these kinds of origins as a broken origin and delete then at the end of the origins initialization if their persistence type is default instead of breaking all the origins initialization. We would probably want to capture how they fail by telemetry or on Nightly channel, but I believe at least we can do that on Release.

        Attachment #9007260 -
        Attachment is obsolete: true

        Attachment #9007707 -
        Flags: review?(jvarga)

    
    

    
  


  
Patch for verifying.

I tested it locally, and it failed if P1 wasn't applied and succeeded if P1 was.

        Attachment #9007261 -
        Attachment is obsolete: true

        Attachment #9007708 -
        Flags: review?(jvarga)
Priority: P3 → P2

    
    

    
  
Comment on attachment 9007707 [details] [diff] [review]
Bug 1425146 - P1 - Ensure the schema for the cache.sqlite is up to date beforing accessing it;

Review of attachment 9007707 [details] [diff] [review]:
-----------------------------------------------------------------

r=me once I see the comment

::: dom/cache/QuotaClient.cpp
@@ +89,5 @@
>      return NS_OK;
>    }
>    if (NS_WARN_IF(NS_FAILED(rv))) { return rv; }
>  
> +  rv = mozilla::dom::cache::db::CreateOrMigrateSchema(conn);

Ok, this makes sense, but we should add a comment here explaining that this wouldn't be necessary if we did full storage major scheme upgrade and that this fixes the case when the the padding column doesn't exist, so the db scheme needs to be checked and potentially upgraded.

    
    

    
  
Comment on attachment 9007708 [details] [diff] [review]
Bug 1425146 - P2 - A test to verify initialization of origins doesn't fail while the padding file is missing and the schema in cache.sqlite is too old;

Review of attachment 9007708 [details] [diff] [review]:
-----------------------------------------------------------------

I need to apply this patch locally, will do some modifications and send an interdiff then.

    
    

    
  
Comment on attachment 9007708 [details] [diff] [review]
Bug 1425146 - P2 - A test to verify initialization of origins doesn't fail while the padding file is missing and the schema in cache.sqlite is too old;

Review of attachment 9007708 [details] [diff] [review]:
-----------------------------------------------------------------

Should this test be places in dom/cache/test/xpcshell ?

::: dom/quota/test/unit/test_cacheOldDBInitialize.js
@@ +28,5 @@
> +  // Since the QuotaManager will create a padding file to each cache directory
> +  // for a new profile in the storage initialization, this test does a trick to
> +  // reproduce it:
> +  //   Let the padding file be created in the intialization, and then remove it
> +  // manually.

Hm, this will only happen if you put older storage.sqlite into the zip (storage version 2.0)
If you use storage.sqlite with storage version 2.1, then quota manager shouldn't create a padding file, so you don't have to remove it in this test.
And that actually simulates the bug that we are fixing here:
1. A new profile is created or used in 57 (storage version 2.1)
2. The profile is used in 56 (storage version 2.0), an origin dir is created with the cache subdir and cache db version is 25 (no padding file and no padding column is created)
3. The profile is used in 57 again and during storage initialization getUsage fails because there's no padding file and the column doesn't exist either

Is that right ?

    
    

    
  
(In reply to Jan Varga [:janv] from comment #86)
> Should this test be places in dom/cache/test/xpcshell ?

Hmm, I put in dom/quota/test/unit for reasons:
- The fix is in GetUsageForOrigin() which is called when initializing temporary storage or getting usage, so it's part of behaviors needing to be ensured in QM.
- The testing framework in QM makes this test simpler since all we need here is call initTemporaryStorage() (the test uses initOrigin() but it should be changed).

> ::: dom/quota/test/unit/test_cacheOldDBInitialize.js
> @@ +28,5 @@
> > +  // Since the QuotaManager will create a padding file to each cache directory
> > +  // for a new profile in the storage initialization, this test does a trick to
> > +  // reproduce it:
> > +  //   Let the padding file be created in the intialization, and then remove it
> > +  // manually.
> 
> Hm, this will only happen if you put older storage.sqlite into the zip
> (storage version 2.0)
> If you use storage.sqlite with storage version 2.1, then quota manager
> shouldn't create a padding file, so you don't have to remove it in this test.
> And that actually simulates the bug that we are fixing here:
> 1. A new profile is created or used in 57 (storage version 2.1)
> 2. The profile is used in 56 (storage version 2.0), an origin dir is created
> with the cache subdir and cache db version is 25 (no padding file and no
> padding column is created)
> 3. The profile is used in 57 again and during storage initialization
> getUsage fails because there's no padding file and the column doesn't exist
> either
> 
> Is that right ?

Yes, I think so. Let me try to make a test like this.

    
    

    
  

      
      
      
      

        Attached patch
          
          
        P2 (v1) (obsolete)
        — 
          — Splinter Review
      

    


  
Addressed comment

        Attachment #9007708 -
        Attachment is obsolete: true

        Attachment #9007708 -
        Flags: review?(jvarga)

    
    

    
  
Comment on attachment 9019403 [details] [diff] [review]
P2 (v1)

Jan, I did:
- Moved the test to dom/cache/test/xpcshell
- Made a profile by what you suggested (Create and init in FF57, Downgrade to FF56 and open website using DOM Cache, and upgrade back to FF57)

Could you help me to review it? Thanks!

        Attachment #9019403 -
        Flags: review?(jvarga)

    
    

    
  
(In reply to Tom Tung [:tt] from comment #89)
> Comment on attachment 9019403 [details] [diff] [review]
> P2 (v1)
> 
> Jan, I did:
> - Moved the test to dom/cache/test/xpcshell
> - Made a profile by what you suggested (Create and init in FF57, Downgrade
> to FF56 and open website using DOM Cache, and upgrade back to FF57)
> 
> Could you help me to review it? Thanks!

I also verified that the test failed without applying P1 and succeeded with applying P1.

    
    

    
  


  
Addressed the comment so far.

        Attachment #9007707 -
        Attachment is obsolete: true

        Attachment #9007707 -
        Flags: review?(jvarga)

        Attachment #9019406 -
        Flags: review?(jvarga)

    
    

    
  
Comment on attachment 9019406 [details] [diff] [review]
Bug 1425146 - P1 - Ensure the schema for the cache.sqlite is up to date beforing accessing it;

Review of attachment 9019406 [details] [diff] [review]:
-----------------------------------------------------------------

r=me if you change the comment

::: dom/cache/QuotaClient.cpp
@@ +91,5 @@
>    if (NS_WARN_IF(NS_FAILED(rv))) { return rv; }
>  
> +  // This fixes the case when the the padding column doesn't exist, so the
> +  // db scheme needs to be checked and potentially upgraded. Note that it isn't
> +  // necessary if we did full storage major scheme upgrade.

So I was thinking about this a bit more and I changed my mind a bit.
1.
We shouldn't mention that a major storage upgrade would help here.
It can happen, that even after a major storage upgrade, the padding
file disappeared and we need to restore it from the database.

2.
If we know that we are accessing the database before DOM cache is
actually used, we should just always check/upgrade the schema.
Not just for this particular case, when the padding column is missing.

So, I would just add this comment:
  // Make sure that the database has the latest schema before we try to read
  // from it. We have to do this because LockedGetPaddingSizeFromDB is called
  // by QuotaClient::GetUsageForOrigin which may run at any time (there's no
  // guarantee that SetupAction::RunSyncWithDBOnTarget already checked the
  // schema for the given origin).

        Attachment #9019406 -
        Flags: review?(jvarga) → review+

    
    

    
  
(In reply to Jan Varga [:janv] from comment #92)

Thanks for the review and feedback!

> So I was thinking about this a bit more and I changed my mind a bit.
> 1.
> We shouldn't mention that a major storage upgrade would help here.
> It can happen, that even after a major storage upgrade, the padding
> file disappeared and we need to restore it from the database.

Indeed. That's the case that I didn't handle properly while implementing the cache padding stuff. It could happen if somehow the padding file can't be read (e.g. file disappear or just broken) and the schema of cache db hasn't be upgraded yet, though it might rare to see.

> 2.
> If we know that we are accessing the database before DOM cache is
> actually used, we should just always check/upgrade the schema.
> Not just for this particular case, when the padding column is missing.

True.

> So, I would just add this comment:
>   // Make sure that the database has the latest schema before we try to read
>   // from it. We have to do this because LockedGetPaddingSizeFromDB is called
>   // by QuotaClient::GetUsageForOrigin which may run at any time (there's no
>   // guarantee that SetupAction::RunSyncWithDBOnTarget already checked the
>   // schema for the given origin).

Will do

    
    

    
  

      
      
      
      

        Attached patch
          
          
        interdiff for P2 (obsolete)
        — 
          — Splinter Review
      

    


  
I think this deserves better comments in the test since description of the problem is quite hard to find in bugzilla comments. You can also use this in future to create tests like this.
I'll also send review comments for P2.

    
    

    
  
Comment on attachment 9019403 [details] [diff] [review]
P2 (v1)

Review of attachment 9019403 [details] [diff] [review]:
-----------------------------------------------------------------

The package contains stuff that is not needed at all, for example:
__MACOSX directory
.DS_Store files
storage/temporary directory
storage/permanent directory
storage/default/https+++firefox-storage-test.glitch.me/idb directory

storage/default/https+++firefox-storage-test.glitch.me/cache/caches.sqlite can be just empty (nothing stored in the database)

storage/default/https+++firefox-storage-test.glitch.me/cache/morgue can be empty too

::: dom/cache/test/xpcshell/test_padding_column_missing.js
@@ +1,3 @@
> +/**
> + *  This test is mainly to verify cache actions work as usual even there exists
> + *  an unexpected padding file.

This doesn't belong here, it was probably copied from test_padding_error_handle.js.
We should rather mention the license here.

@@ +2,5 @@
> + *  This test is mainly to verify cache actions work as usual even there exists
> + *  an unexpected padding file.
> + */
> +
> +// This test is mainly to fix the unexpected condition reporting in

This test doesn't "fix" it.

@@ +4,5 @@
> + */
> +
> +// This test is mainly to fix the unexpected condition reporting in
> +// bug 1425146 comment 39. The test below test is trying to reproduce the
> +// condition and ensure at least it won't break the origins initialization.

As I mentioned, I believe this deserves a more detailed comment since the problem is not easy to understand from bugzilla comments and multiple storage aspects are involved like storage versions, cache database schema, etc.

@@ +15,5 @@
> +  //    the storage.
> +  // 2. Downgrade to FF56 (storage version 2.0), and go to
> +  //    "https://firefox-storage-test.glitch.me/" to generate a cache storage
> +  //    without a directory padding file.
> +  // 3. Upgrade to FF57 or later.

Again, it's not trivial to build a profile like this so it's better to describe it here in more details.

::: dom/cache/test/xpcshell/xpcshell.ini
@@ +6,5 @@
>  head = head.js
>  support-files =
>    schema_15_profile.zip
>    schema_25_profile.zip
> +  versionBounce_profile.zip

This sounds too generic (a version bounce).

@@ +13,5 @@
>  [make_profile.js]
>    skip-if = true
>  
>  [test_migration.js]
> +[test_padding_column_missing.js]

The problem is a bit more complex, it's not just missing column. I think we can just use the bug number in this case.

        Attachment #9019403 -
        Flags: review?(jvarga)

    
    

    
  
Tom, I'm a bit lost in this bug, do we need to fix something else besides the problem with missing padding file and padding column during origin initialization ?
Flags: needinfo?(shes050117)

    
    

    
  
(In reply to Jan Varga [:janv] from comment #97)
> Tom, I'm a bit lost in this bug, do we need to fix something else besides
> the problem with missing padding file and padding column during origin
> initialization ?

If there's nothing else, we probably need to do something with bug description here and maybe file a new bug for "How to debug and recover from Unknown IndexedDB errors?"

    
    

    
  
(In reply to Jan Varga [:janv] from comment #97)
> Tom, I'm a bit lost in this bug, do we need to fix something else besides
> the problem with missing padding file and padding column during origin
> initialization ?

I think it's the main issue here.

Another one is that having a better error message for an unexcepted file, but it's not really related to storage initialization. Thus, I want to open another bug for that.

Other than them, there seems to have other storage initialization issues (but the information wasn't sufficient information to identify the root cause). I guess they might also be reported in bug 944918, and bug 1423917, so I planned to solve them all in bug 1423917. We could probably see the result after bug 1436188.
Flags: needinfo?(shes050117)

    
    

    
  
(In reply to Jan Varga [:janv] from comment #96)

Thanks for the feedback!

> Comment on attachment 9019403 [details] [diff] [review]
> P2 (v1)
> 
> Review of attachment 9019403 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> The package contains stuff that is not needed at all, for example:
> __MACOSX directory
> .DS_Store files
> storage/temporary directory
> storage/permanent directory
> storage/default/https+++firefox-storage-test.glitch.me/idb directory
> 
> storage/default/https+++firefox-storage-test.glitch.me/cache/caches.sqlite
> can be just empty (nothing stored in the database)
> 
> storage/default/https+++firefox-storage-test.glitch.me/cache/morgue can be
> empty too

Sure, will clean them out.

> ::: dom/cache/test/xpcshell/test_padding_column_missing.js
> @@ +1,3 @@
> > +/**
> > + *  This test is mainly to verify cache actions work as usual even there exists
> > + *  an unexpected padding file.
> 
> This doesn't belong here, it was probably copied from
> test_padding_error_handle.js.
> We should rather mention the license here.

I guess I accidentally modified the string for the copyright. Sorry about that.

> @@ +2,5 @@
> > + *  This test is mainly to verify cache actions work as usual even there exists
> > + *  an unexpected padding file.
> > + */
> > +
> > +// This test is mainly to fix the unexpected condition reporting in
> 
> This test doesn't "fix" it.

Will correct it.

> @@ +4,5 @@
> > + */
> > +
> > +// This test is mainly to fix the unexpected condition reporting in
> > +// bug 1425146 comment 39. The test below test is trying to reproduce the
> > +// condition and ensure at least it won't break the origins initialization.
> 
> As I mentioned, I believe this deserves a more detailed comment since the
> problem is not easy to understand from bugzilla comments and multiple
> storage aspects are involved like storage versions, cache database schema,
> etc.

Will correct it.

> @@ +15,5 @@
> > +  //    the storage.
> > +  // 2. Downgrade to FF56 (storage version 2.0), and go to
> > +  //    "https://firefox-storage-test.glitch.me/" to generate a cache storage
> > +  //    without a directory padding file.
> > +  // 3. Upgrade to FF57 or later.
> 
> Again, it's not trivial to build a profile like this so it's better to
> describe it here in more details.

Will do.

> ::: dom/cache/test/xpcshell/xpcshell.ini
> @@ +6,5 @@
> >  head = head.js
> >  support-files =
> >    schema_15_profile.zip
> >    schema_25_profile.zip
> > +  versionBounce_profile.zip
> 
> This sounds too generic (a version bounce).
> @@ +13,5 @@
> >  [make_profile.js]
> >    skip-if = true
> >  
> >  [test_migration.js]
> > +[test_padding_column_missing.js]
> 
> The problem is a bit more complex, it's not just missing column. I think we
> can just use the bug number in this case.

Will do.

    
    

    
  
Comment on attachment 9020040 [details] [diff] [review]
Bug 1425146 - P2 - A test to verify that origin initializations don't fail while a padding file is missing and the schema in cache.sqlite is too old; r=janv

Thanks for providing a template for standardizing the xpcshell test!

This patch applied the inter-diff and addressed the comment. Jan, could you help me to review it? Thanks!

        Attachment #9020040 -
        Flags: review?(jvarga)

    
    

    
  
Comment on attachment 9020040 [details] [diff] [review]
Bug 1425146 - P2 - A test to verify that origin initializations don't fail while a padding file is missing and the schema in cache.sqlite is too old; r=janv

Review of attachment 9020040 [details] [diff] [review]:
-----------------------------------------------------------------

>Bug 1425146 - P2 - A test to verify that origin initializations don't fail while a padding file is missing and the schema in cache.sqlite is too old; r=janv

s/initializations/initialization
s/don't/doesn't

>It was originally reported at bug 1425146 comment 39. This test tries to
>reporduce the steps and ensure the initialization is success by using a cache
>storage operation without throwing an exception.

This sounds very confusing to me, can you rephrase it ?

    
  
See Also:  → 1502077
Summary: How to debug and recover from Unknown IndexedDB errors? → response_padding_size doesn't exist in cache.sqlite while accessing it in storage initialization

    
    

    
  
Comment on attachment 9020057 [details] [diff] [review]
Bug 1425146 - P2 - A test to verify that origin initialization doesn't fail while a padding file is missing and the schema in cache.sqlite is too old; r=janv

Change the message in bugzilla

        Attachment #9020057 -
        Attachment description: Bug 1425146 - P2 - A test to verify that origin initializations don't fail while a padding file is missing and the schema in cache.sqlite is too old; r=janv → Bug 1425146 - P2 - A test to verify that origin initialization doesn't fail while a padding file is missing and the schema in cache.sqlite is too old; r=janv

    
    

    
  
Comment on attachment 9020057 [details] [diff] [review]
Bug 1425146 - P2 - A test to verify that origin initialization doesn't fail while a padding file is missing and the schema in cache.sqlite is too old; r=janv

Sorry for the back and forth, but I think it's better to check it again and send it to review tomorrow.

        Attachment #9020057 -
        Flags: review?(jvarga)

    
    

    
  

      
      
      
      

        Attached patch
          
          
        bug1425146-p2.patch (obsolete)
        — 
          — Splinter Review
      

    


  

        Attachment #9020057 -
        Attachment is obsolete: true

    
    

    
  
Comment on attachment 9020289 [details] [diff] [review]
Bug 1425146 - P2 - Add a test reproducing the reported issue to verify that a Cache directory without a padding file and an upgraded db doesn't break the storage initialization; r=janv

Review of attachment 9020289 [details] [diff] [review]:
-----------------------------------------------------------------

Much better!

>Bug 1425146 - P2 - Add a test reproducing the reported issue to verify that a Cache directory without a padding file and an upgraded db doesn't break the storage initialization; >r=janv

"... and an upgraded db ..."
Should this be "... and not yet upgraded db ..." ?


>An issue for missing the "response_padding_size" column in cache.sqlite was
>reported in bug 1425146 comment 39. This test reproduces it and is mainly to
>ensure that a Cache directory which lacks of a directory padding file and
>contains a old version of cache.sqlite can still be initialized successfully.

I think "which lacks a" sounds better than "which lacks of a", but I may be wrong.
Nit: "a old version" -> "an old version"

        Attachment #9020289 -
        Flags: review?(jvarga) → review+

    
    

    
  
(In reply to Jan Varga [:janv] from comment #109)
> Comment on attachment 9020289 [details] [diff] [review]
> Bug 1425146 - P2 - Add a test reproducing the reported issue to verify that
> a Cache directory without a padding file and an upgraded db doesn't break
> the storage initialization; r=janv
> 
> Review of attachment 9020289 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Much better!
> 
> >Bug 1425146 - P2 - Add a test reproducing the reported issue to verify that a Cache directory without a padding file and an upgraded db doesn't break the storage initialization; >r=janv
> 
> "... and an upgraded db ..."
> Should this be "... and not yet upgraded db ..." ?

I was thinking that "without a padding file" + "without an upgraded db" so that I wrote "without a padding file and an upgraded db", but it's confusing.

What about "without both a padding file and an upgraded db" or "without a padding file and with an old version db"?

> >An issue for missing the "response_padding_size" column in cache.sqlite was
> >reported in bug 1425146 comment 39. This test reproduces it and is mainly to
> >ensure that a Cache directory which lacks of a directory padding file and
> >contains a old version of cache.sqlite can still be initialized successfully.
> 
> I think "which lacks a" sounds better than "which lacks of a", but I may be
> wrong.
> Nit: "a old version" -> "an old version"

Thanks, will correct it.

    
    

    
  
I guess it should match the logic used in "which lacks a directory padding file and contains an old version of cache.sqlite".
What about:
"without a padding file and with an old version of the database"

    
    

    
  
(In reply to Jan Varga [:janv] from comment #112)
> I guess it should match the logic used in "which lacks a directory padding
> file and contains an old version of cache.sqlite".
> What about:
> "without a padding file and with an old version of the database"

That sounds better. Thanks!

    
    

    
  
Pushed by shes050117@gmail.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/52646a13f5de
P1 - Ensure the schema for the cache.sqlite is up to date beforing accessing it; r=janv
https://hg.mozilla.org/integration/mozilla-inbound/rev/78f441521243
P2 - Add a test reproducing the reported issue to verify that a Cache directory without a padding file and with an old version of the database doesn't break the storage initialization; r=janv

    
    

    
  
https://hg.mozilla.org/mozilla-central/rev/52646a13f5de
https://hg.mozilla.org/mozilla-central/rev/78f441521243
Status: NEW → RESOLVED
Closed: 6 years ago

          status-firefox65:
          --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla65

    
    

    
  
Comment on attachment 9019951 [details] [diff] [review]
Bug 1425146 - P1 - Ensure the schema for the cache.sqlite is up to date beforing accessing it; r=janv

[Beta/Release Uplift Approval Request]

Feature/Bug causing the regression: Bug 1290481

User impact if declined: If a directory padding file in a cache directory somehow cannot be read and its database hasn't been upgraded to the current version, then the profile will be broken.

Is this code covered by automated tests?: Yes

Has the fix been verified in Nightly?: Yes

Needs manual test from QE?: Yes

If yes, steps to reproduce: 

List of other uplifts needed: None

Risk to taking this patch: Low

Why is the change risky/not risky? (and alternatives if risky): It's verified by P2 and the patches have stayed in Nightly for a week. Also, it's executed only in the error case (missing padding file or file cannot be read).

String changes made/needed:

        Attachment #9019951 -
        Flags: approval-mozilla-beta?

    
    

    
  
Comment on attachment 9020727 [details] [diff] [review]
Bug 1425146 - P2 - Add a test reproducing the reported issue to verify that a Cache directory without a padding file and with an old version of the database doesn't break the storage initialization;

[Beta/Release Uplift Approval Request]

Feature/Bug causing the regression: None

User impact if declined: 

Is this code covered by automated tests?: Yes

Has the fix been verified in Nightly?: Yes

Needs manual test from QE?: Yes

If yes, steps to reproduce: 

List of other uplifts needed: None

Risk to taking this patch: Low

Why is the change risky/not risky? (and alternatives if risky): 

String changes made/needed:

        Attachment #9020727 -
        Flags: approval-mozilla-release?

    
    

    
  
Tom, why do you think this patch should be uplifted to beta and release? We have shipped with this bug since at least Firefox 57 and it is a P2 not marked as a regression so it seems like regular bug work, is there a pressing reason now to not let just it ride the trains to 65? Thanks
Flags: needinfo?(shes050117)

    
    

    
  
Ah, I made a wrong request for P2. It should just be uplifted to Beta. Sorry about that...

However, I do think both patches should be uplifted to the Beta if the risk is low. The reason is that it's a regression (not handling the error case properly in the DOM Cache initialization) and what the P1 does is to fix the issue described was in comment #39. That issue was reported at Nightly 58. 
The issue might happen if either:
- A user used Nightly 57 and downgraded to Nightly 56 and then switch to some version that is greater than 57
- A user's directory padding file in DOM Cache directory is broken and the database in that directory hasn't been upgraded to the newest version yet.

If someone somehow traps into this, then their profile is broken (which means they cannot use indexedDB, QuotaManager, Cache, ServiceWork ... etc). 

I haven't found any reporting issues that are related this, so it seems that both cases happened rarely, but I reckon it's an issue that has to be taken care in the older version of Firefox.
Flags: needinfo?(shes050117)

    
    

    
  
Comment on attachment 9020727 [details] [diff] [review]
Bug 1425146 - P2 - Add a test reproducing the reported issue to verify that a Cache directory without a padding file and with an old version of the database doesn't break the storage initialization;

[Beta/Release Uplift Approval Request]

Feature/Bug causing the regression: None

User impact if declined: It's the test for verifying P1

Is this code covered by automated tests?: No

Has the fix been verified in Nightly?: Yes

Needs manual test from QE?: No

If yes, steps to reproduce: 

List of other uplifts needed: None

Risk to taking this patch: Low

Why is the change risky/not risky? (and alternatives if risky): It's a test

String changes made/needed:

        Attachment #9020727 -
        Flags: approval-mozilla-release? → approval-mozilla-beta?

    
    

    
  
(In reply to Tom Tung [:tt] from comment #120)
> Ah, I made a wrong request for P2. It should just be uplifted to Beta. Sorry

s/P2/patch 2/g

> about that...
> 
> However, I do think both patches should be uplifted to the Beta if the risk
> is low. The reason is that it's a regression (not handling the error case
> properly in the DOM Cache initialization) and what the P1 does is to fix the
> issue described was in comment #39. That issue was reported at Nightly 58. 
> The issue might happen if either:
> - A user used Nightly 57 and downgraded to Nightly 56 and then switch to
> some version that is greater than 57
> - A user's directory padding file in DOM Cache directory is broken and the
> database in that directory hasn't been upgraded to the newest version yet.
> 
> If someone somehow traps into this, then their profile is broken (which
> means they cannot use indexedDB, QuotaManager, Cache, ServiceWork ... etc). 
> 
> I haven't found any reporting issues that are related this, so it seems that
> both cases happened rarely, but I reckon it's an issue that has to be taken
> care in the older version of Firefox.

From my point of view, I don't have a pressing reason for uplifting and also it seems that it happens rarely. However, I think it's a regression so it should be uplifted.

    
    

    
  
Comment on attachment 9019951 [details] [diff] [review]
Bug 1425146 - P1 - Ensure the schema for the cache.sqlite is up to date beforing accessing it; r=janv

cache corruption fix, approved for 64.0b8

        Attachment #9019951 -
        Flags: approval-mozilla-beta? → approval-mozilla-beta+

        Attachment #9020727 -
        Flags: approval-mozilla-beta? → approval-mozilla-beta+

    
    

    
  
(In reply to Tom Tung [:tt] from comment #121)
> Needs manual test from QE?: No
Marking as qe-verify- per Tom's assessment.
Flags: qe-verify-

          You need to log in
          before you can comment on or make changes to this bug.
        






  

    
  







  

    

      
Attachment

      

      
      
      
      
    

    

      

        

          

            
General

            

              Creator: 



            

            
Created: 

            
Updated: 

            
Size: