1539139 - nightly failing A LOT of pages with NS_ERROR_UNEXPECTED

Status: RESOLVED INACTIVE

(Core :: Storage: Quota Manager, defect, P2)

Description

[Peter Gervai]

User Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Firefox/68.0

Steps to reproduce:

Updated the browser. Pages stop loading or fail. Console always show NS_ERROR_UNEXPECTED, related to various parts.
Example: proxmox GUI (local), RIPE LIRPORTAL

Actual results:

Javascript crash.

Expected results:

Was working until 2019-03-24

Comment 1

[Peter Gervai]

Can be fixed by using private mode.
Cannot be fixed by removing extensions, nor -safe-mode.
"Refresh firefox" (aka. factory reset) fixes it but clears all extensions and session save, so it's not useful at all. Present in 68.0a1 (2019-03-26) (64-bit).

Comment 2

[Matthias Versen [:Matti]]

It something in your settings or the profile if the refresh function fixes this.
What do you mean by "Javascript crash" ?

It's difficult for another person to guess what is wrong in your Firefox profile.

This bug is only actionable if:
a) You provide clear steps top reproduce (currently missing)
b) What setting in Firefox or which file in the profile is also required to trigger the bug.

Comment 3

[Peter Gervai]

(In reply to Matthias Versen [:Matti] from comment #2)

It something in your settings or the profile if the refresh function fixes this.

As far as I am able to do manually -- setting the settings the same in the "nonworking" profile as the "fresh" profile does not resolve the bug; or to put it in other way I don't seem to be able to resolve the bug by doing anything on the settings page.
It looks like some accumulated cruft or setting causes it but I don't quite see where to look. I try to see about:config, since this is a decade old profile...

What do you mean by "Javascript crash" ?

Well there's an error where there wasn't one last time, and the JS execution halts there. And it is present in many places, I actually thought it's a bug of the website until I have experienced it in 5-6th times elsewhere.

It's difficult for another person to guess what is wrong in your Firefox profile.

I'm sure, but simply diffing prefs.js result 660 lines, and I'm not even sure that's all what there is.

This bug is only actionable if:
a) You provide clear steps top reproduce (currently missing)

Probably that's not possible, since doing that would mean I already know what the bug is. :-)

b) What setting in Firefox or which file in the profile is also required to trigger the bug.

I try to fiddle with the prefs, though I see little hope.

Comment 4

[Peter Gervai]

Additional info: completely removing prefs.js (and restart) doesn't fix it.
Any idea what shall I destroy next? I would appreciate a list of gradual destruction, where I can walk and remove files to see what's changed.

Comment 5

[Peter Gervai]

(Why "has str" [steps to reproduce, as I guess] is true? I mean I can reliably test it but can't give external instructions.)

Comment 6

[Peter Gervai]

This wasn't easy.
It is the storage directory. Removing it fixes the problem.

I am not familiar with firefox storages, and this one seems to be either kind of JS local storage or general site-specific storage like cookies. What would you like me to do next?

Comment 7

[Matthias Versen [:Matti]]

nice find Peter !
I guess that the directory belongs to Core/Dom:Webstorage.
I will move this report to that component and the responsible developer/QA will hopefully show up in this report :-)

Comment 8

[Peter Gervai]

I don't really understand it but:

It was "Stylish", an extension, which does have a directory in the storage called moz-extension+++d9f72c26-c6ee-4859-b54e-95dc71952e84.

However this extension is disabled since long time ago!
Removing its .metadata and .metadata-v2 resolves the problem.

I do not think a disabled extension shall have any effect on Firefox itself, but that's really your expertise.
If you need anything more from me, just ask. I have moved the files away.

Comment 9

[Tom Tung [:tt, :ttung]]

It sounds like a problem on QuotaManager and initialization stuff. I'm moving it to QuotaManager for now and ni myself to check it tomorrow.

Comment 10

[Tom Tung [:tt, :ttung]]

(In reply to Peter Gervai from comment #8)

I don't really understand it but:

It was "Stylish", an extension, which does have a directory in the storage called moz-extension+++d9f72c26-c6ee-4859-b54e-95dc71952e84.

However this extension is disabled since long time ago!
Removing its .metadata and .metadata-v2 resolves the problem.

I do not think a disabled extension shall have any effect on Firefox itself, but that's really your expertise.
If you need anything more from me, just ask. I have moved the files away.

Peter, would you mind attaching the "moz-extension+++d9f72c26-c6ee-4859-b54e-95dc71952e84" folder or emailing that to me? I guess it'll help me to find the problem more quickly if you don't mind doing that. Thanks!

Comment 11

[Tom Tung [:tt, :ttung]]

Attachment: bug-1539139

I made a simple test for that and it passed. So I guess it's not related to its filename. (Note that I didn't make metadata files for them to force them generating new metadata files)

Comment 12

[Peter Gervai]

Attachment: storage.bug.extension.dir.tgz

It's not related to the name of the directory, in fact renamed to "buggy" and it still produced the error, had to move away from .../storage/ altogether.
Attaching, I believe there's no private info in them.

Comment 13

[Tom Tung [:tt, :ttung]]

(In reply to Peter Gervai from comment #12)

Thanks for providing that directory!

Created attachment 9054155 [details]
storage.bug.extension.dir.tgz

It's not related to the name of the directory, in fact renamed to "buggy" and it still produced the error, had to move away from .../storage/ altogether.
Attaching, I believe there's no private info in them.

So, renaming it to "buggy" is expected to break your storage because such file name is not allowed to exist in that directory. And just make sure there is nothing unexpected, did you mean you move it from ".../stroage/default/"? I expected this should live under either "default", "temporary", or "permanent" directory.

I cannot reproduce the issue by adding to the current storage test for upgrading from 2_1 to 2_2 [1]. (The test will pass).

Running with that with a debug build which has been upgraded to 2_2 already would still successfully initialize the storage. However, I got the following message:

[Parent 7814, QuotaManager IO] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80630001: file /Users/tomtung/Work/mozilla-central/storage/mozStorageConnection.cpp, line 698
[Parent 7814, QuotaManager IO] WARNING: NS_ENSURE_SUCCESS(rv, rv) failed with result 0x80630001: file /Users/tomtung/Work/mozilla-central/storage/mozStorageService.cpp, line 671
[Parent 7814, QuotaManager IO] WARNING: Received NS_ERROR_STORAGE_BUSY when attempting to open database '3870112724rsegmnoittet-es.sqlite', retrying for up to 10 seconds: file /Users/tomtung/Work/mozilla-central/dom/indexedDB/ActorsParent.cpp, line 4081

Comment 14

[Peter Gervai]

The problem was fixed by "removing" the directory from the firefox directory structure ("moved from storage/... into /tmp/").

Since it's an installed (but disabled) extension there could be other, related files or configuration, but that's too deep internals for me to see readily. I can pick whatever you point to.

(It may be a problem that by removing the directory changes internal states elsewhere [so I wouldn't be able to reproduce just by reinserting the directory], but, again, I can't tell or see. In fact, let me see whether I can break it again by reinserting.)

Comment 15

[Peter Gervai]

Bad news: reinserting will not break it anymore. :-(

Comment 16

[Tom Tung [:tt, :ttung]]

(In reply to Peter Gervai from comment #14)

Thanks! I guess I have a few questions.

The problem was fixed by "removing" the directory from the firefox directory structure ("moved from storage/... into /tmp/").

Do you mean from "storage/default" or "storage"?

Since it's an installed (but disabled) extension there could be other, related files or configuration, but that's too deep internals for me to see readily. I can pick whatever you point to.

Could you go to the "https://firefox-storage-test.glitch.me/" website to test whether you have a problem related to initialization? If everything is green, then it means your storage is in a good status.

(It may be a problem that by removing the directory changes internal states elsewhere [so I wouldn't be able to reproduce just by reinserting the directory], but, again, I can't tell or see. In fact, let me see whether I can break it again by reinserting.)

(In reply to Peter Gervai from comment #15)

Bad news: reinserting will not break it anymore. :-(

Hmm, that's indeed bad news for us, but thank you for letting us know there is an issue on storage initialization. It sounds like a problem with permission stuff.

Comment 17

[Tom Tung [:tt, :ttung]]

(In reply to Tom Tung [:tt, :ttung] from comment #16)

(In reply to Peter Gervai from comment #15)

Bad news: reinserting will not break it anymore. :-(

Hmm, that's indeed bad news for us, but thank you for letting us know there is an issue on storage initialization. It sounds like a problem with permission stuff.

Note to myself: It might also because the problem code (e.g. the upgrade) won't be executed in the second try.

Comment 18

[Peter Gervai]

Do you mean from "storage/default" or "storage"?

From "~/.mozilla/firefox/userdir/storage/default/".

Could you go to the "https://firefox-storage-test.glitch.me/" website to test whether you have a problem related to initialization? If everything is green, then it means your storage is in a good status.

It's all green.

Note to myself: It might also because the problem code (e.g. the upgrade) won't be executed in the second try.

I am not sure whether it's relevant or not but the problem persisted throughout restarts until I have actually removed the extension directory.

Comment 19

[Tom Tung [:tt, :ttung]]

Attachment: storage.sqlite.zip

Comment 20

[Tom Tung [:tt, :ttung]]

Peter, could you do me one more favor? I want to test the upgrade code in your profile because I cannot reproduce it by my computer.

In the storage.sqlite.zip, it's compressed storage.sqlite file with storage version 2_1 (right now it's 2_2). Could you put it to your profile (.mozilla/firefox/userdir/), insert "moz-extension+++d9f72c26-c6ee-4859-b54e-95dc71952e84" folder and run the debug build in [1] and check the debugging message? You should find something like "[Parent xxx, QuotaManager IO] xxxx".

You should be able to check if you get the same problem by visiting the "https://firefox-storage-test.glitch.me/" website. If everything is still green, then I guess the problem is somehow not being able to reproduce.

Note that please remember to backup your current "storage.sqlite" file and all other files and folder under "storage" folder before doing all the tests. Thanks!

[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=9ce3ccc51ae4f832f018794d331d1c0c43cf7048

Comment 21

[Tom Tung [:tt, :ttung]]

Since the information is not clear enough to take any action at this moment, I'm going to close this bug. Please feel free to reopen it!