1595913 - Extension storage gets corrupted on Firefox update

Status: RESOLVED DUPLICATE of bug 1536796

(Core :: Storage: Quota Manager, defect)

Description

[Alexei]

User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/78.0.3904.87 Safari/537.36

Steps to reproduce:

Firefox got updated.

Actual results:

Sometimes extensions break because their storage database get corrupted.

Extension background pages seem filled with "The operation failed for reasons unrelated to the database itself and not covered by any other error code." errors coming from ExtensionStorageIDB.jsm.

Please see https://github.com/EFForg/privacybadger/issues/2506 for an example report. Note that this problem is not exclusive to Privacy Badger.

Expected results:

Extension storage does not get corrupted.

Comment 1

[Peter Macej]

Hi,

I'm the one who reported the original problem at https://github.com/EFForg/privacybadger/issues/2506 mentioned above. Finally, I found a reason and fix for my problem. I didn't want to loose all my setting and customization by "Refresh Firefox". So I went to my FF profile folder and opened the folder with indexed DB files, which was:
C:\Users\<USER>\AppData\Roaming\Mozilla\Firefox\Profiles\<DEFAULT_PROFILE>\storage\default\

I closed FF and deleted all sub-folders in it. When I started FF, everything worked (of course, all my extension settings were lost). So I added back half of the original sub-folders. If it worked, I added half of the missing folders. If it didn't work, I deleted half of the folders. With this kind of "binary search" I quickly discovered, that there's only one sub-folder that causes the problems. It was for a local HTML page and it was empty. The sub-folder name was:
file++++C++Zdrojaky+Helixoft+VSdocman+src+OfficialSample+SampleClassLibraryCsharp+VSdoc+HTML+topic_0000000000000076_props--.html

I don't know whether the folder complicated name or its emptiness was the problem, but when I deleted just this sub-folder, everything worked fine and I didn't lost any of my settings.

So it seems that the problem was not related to the update to FF 70 and the corrupted empty subfolder was perhaps caused by FF crash. I don't know. Anyway, FF should detect such problems with single DB and not cause entire store to fail.

Comment 2

[Luca Greco [:rpl] [:luca] [:lgreco]]

Hi Tom,
based on comment 1 this looks one of the issues related to the underlying components (QuotaManagerService and IndexedDB),
do you mind to double-check if this particular one already have a duplicate that you are already taking care of or if it is a new issue to add to the tracking bug related to this class of bugs?

Comment 3

[Tom Tung [:tt, :ttung]]

(In reply to Luca Greco [:rpl] [:luca] [:lgreco] from comment #2)

Hi Tom,
based on comment 1 this looks one of the issues related to the underlying components (QuotaManagerService and IndexedDB),
do you mind to double-check if this particular one already have a duplicate that you are already taking care of or if it is a new issue to add to the tracking bug related to this class of bugs?

Hi,

Sorry for the late reply.

This looks like the file name too long issue (bug 1536796) to me.

Peter, could you help me to verify that by manually create a file with more than ".metadata-v2" (12 char) on that directory (file++++C++Zdrojaky+Helixoft+VSdocman+src+OfficialSample+SampleClassLibraryCsharp+VSdoc+HTML+topic_0000000000000076_props--.html)? If you can create one successfully, then it's probably another issue.

Thanks!

Comment 4

[Peter Macej]

Attachment: Creating file with too long name.png

(In reply to Tom Tung [:tt, :ttung] from comment #3)

Peter, could you help me to verify that by manually create a file with more than ".metadata-v2" (12 char) on that directory (file++++C++Zdrojaky+Helixoft+VSdocman+src+OfficialSample+SampleClassLibraryCsharp+VSdoc+HTML+topic_0000000000000076_props--.html)? If you can create one successfully, then it's probably another issue.

Thanks!

I created the ".metadata-v2" without problems. Then I created another one with longer name "012345679abcdefghij012345679.metadata-v2", again, no problems. Then I tried to rename it to even longer name "012345679abcdefghij012345679abcdefghij012345679abcdefghij012345679abcdefghij.metadata-v2" and then I got the warning from Total commander:

attachment

I'll be glad to perform more tests if needed.

Comment 5

[Peter Macej]

(In reply to Peter Macej from comment #4)

One more info. After the warning from Total commander, the "012345679abcdefghij012345679abcdefghij012345679abcdefghij012345679abcdefghij.metadata-v2" was successfully created. But then I opened it in PSPad and added some text in it. When I tried to save it, it failed with an error like "Error saving the file".

Comment 6

[Peter Macej]

(In reply to Peter Macej from comment #5)

More clarification as my previous findings were a bit misleading because I created some files with Windows File explorer which brings no warnings for long names and some files were created with Total commander (TC) which brings the warnings.

Here's the summary. The ".metadata-v2" file name is OK and I can write and save it with e.g. PSPad. The "0.metadata-v2" name is also OK. But with "01.metadata-v2" TC gives a warning that it is 260 chars long and I cannot edit and save it.

Comment 7

[Tom Tung [:tt, :ttung]]

(In reply to Peter Macej from comment #6)
Sorry, I edited the bug and left some comments, but forgot to submit...

(In reply to Peter Macej from comment #5)

More clarification as my previous findings were a bit misleading because I created some files with Windows File explorer which brings no warnings for long names and some files were created with Total commander (TC) which brings the warnings.

Here's the summary. The ".metadata-v2" file name is OK and I can write and save it with e.g. PSPad. The "0.metadata-v2" name is also OK. But with "01.metadata-v2" TC gives a warning that it is 260 chars long and I cannot edit and save it.

So, this is not related to file name too long issue, but it definitely related to storage initialization (because everything recovers back after removing the origin directories).

This could be hidden once bug 1594080 is fixed.

Comment 8

[Tom Tung [:tt, :ttung]]

(In reply to Peter Macej from comment #6)

(In reply to Peter Macej from comment #5)

More clarification as my previous findings were a bit misleading because I created some files with Windows File explorer which brings no warnings for long names and some files were created with Total commander (TC) which brings the warnings.

Here's the summary. The ".metadata-v2" file name is OK and I can write and save it with e.g. PSPad. The "0.metadata-v2" name is also OK. But with "01.metadata-v2" TC gives a warning that it is 260 chars long and I cannot edit and save it.

Hi Peter, could you share the result of ls -al for that folder? I want to check if there is an unknown file/directory inside so that the initialization is broken (this is expected to be fixed by bug 1594075 and bug 1592204).
Normally, there should only have:

• .metadata-v2 [file]
• idb/cache/ls [folders]

Or it would be better if you can share the stdout & stderr of the debug build. So that we can get more information to analyze. Thanks!

Comment 9

[Peter Macej]

(In reply to Tom Tung [:tt, :ttung] from comment #8)

Hi Peter, could you share the result of ls -al for that folder? I want to check if there is an unknown file/directory inside so that the initialization is broken (this is expected to be fixed by bug 1594075 and bug 1592204).
Normally, there should only have:

• .metadata-v2 [file]
• idb/cache/ls [folders]

Or it would be better if you can share the stdout & stderr of the debug build. So that we can get more information to analyze. Thanks!

I'm on Windows 10, so no 'ls' command. But that folder is empty for sure, no hidden or system files or folders, event the DIR command returns no results.

What do you mean by the debug build? I'm using the latest official FF version.

Comment 10

[Tom Tung [:tt, :ttung]]

(In reply to Peter Macej from comment #9)
Thanks for the quick reply!

I'm on Windows 10, so no 'ls' command. But that folder is empty for sure, no hidden or system files or folders, event the DIR command returns no results.

I see. That helps us to understand that it's also not related to the unknown files/directories issues.

What do you mean by the debug build? I'm using the latest official FF version.

I meant the build with debug symbol. Note that running this might have some issues and it's definately okay if you don't want to do that.
If you wouldn't mind that, please ensure you copy your profile before proceeding.

The link below is from our treeherder server:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-release&selectedJob=276757626&searchStr=Build

You can find your build in "Windows 2012 x64 debug" "B" symbol (I assume you are x64). Then, you could see in the "Job Details" column below, there is an item "artifact uploaded: target.zip".

• Download the Firefox release x64 Release version via that link.
• Unzip the file and you should be able to find the debug version of the Firefox.
• Ensure you have made a copy of your profile in another place.
• Open that Firefox with your profile [1]
• Visit the website "https://firefox-storage-test.glitch.me/"
• Copy the log inside the console and maybe filter the data with "dom/quota" (because this log might leak your personal information and all we need is the information related to QuotaManager)
• Share the log in this bug or email that privately to me if you have some concerns.

Thanks!

[1] Profile I meant the folder of the profile. You can find yours in the "about:profiles" page

Comment 11

[Peter Macej]

Attachment: The log from debug build console

This is the debug log from the procedure you asked above. I returned the problematic empty folder to my profile. Then I started the debug build and from about:profiles I switched to that profile in a new browser. Then I went to https://firefox-storage-test.glitch.me/

Comment 12

[Jan Varga [:janv]]

It seems .metadata-v2-tmp couldn't be created. So it's probably the issue with long file path.

Comment 13

[Peter Macej]

(In reply to Jan Varga [:janv] from comment #12)

It seems .metadata-v2-tmp couldn't be created. So it's probably the issue with long file path.

Could be. When I try to create .metadata-v2-tmp manually, I get a warning that the path is too long (262 chars).

Comment 14

[Tom Tung [:tt, :ttung]]

(In reply to Jan Varga [:janv] from comment #12)

It seems .metadata-v2-tmp couldn't be created. So it's probably the issue with long file path.

Ah, I totally forgot to check this longest file name.

(In reply to Peter Macej from comment #11)

Created attachment 9110611 [details]
The log from debug build console

This is the debug log from the procedure you asked above. I returned the problematic empty folder to my profile. Then I started the debug build and from about:profiles I switched to that profile in a new browser. Then I went to https://firefox-storage-test.glitch.me/

Thanks for the log! It verifies it's the file name too long issue (same error from the same function). I'm marking this issue as a duplicate. And this issue could be fixed/improved by QuotaManager v3.