Unlimited storage is not unlimited

NEW
Assigned to

Status

()

P3
normal
3 years ago
6 months ago

People

(Reporter: brunoaiss, Assigned: janv)

Tracking

({regression})

44 Branch
x86_64
Windows 7
regression
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

3 years ago
I have been using a DB in IndexedDB for quite a while.
As time passed, I increased its permissions allowing it to store unlimited data.
Currently the DB I'm using is using ~680MB of storage.

This started happening in firefox 41.0.1 but I can still reproduce it in Nightly.

When I try to open the DB (even before I try doing anything) I get the QuotaExceededError error with the message:
"The current transaction exceeded its quota limitations."

This wasn't happening until I updated to firefox 41.
Please do answer this ASAP as I haven't been warned about this happening.

Comment 1

3 years ago
Hai Brinoais,
             Can you ask this question here :- https://support.mozilla.org
Flags: needinfo?(brunoaiss)
(Reporter)

Comment 2

3 years ago
Done
Flags: needinfo?(brunoaiss)

Updated

3 years ago
Status: UNCONFIRMED → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → INVALID
Whiteboard: Closed
(Reporter)

Comment 3

3 years ago
Without a reason specified, I cannot accept this to be just closed.
Why are you closing this bug report?
If this is a feature and not a bug, please do point to where that is mentioned.
If this is an undocumented change, then I cannot consider this as deliberate because it broke code and it didn't give any time to find a workaround!
Status: RESOLVED → UNCONFIRMED
Flags: needinfo?(cskumaresan)
Resolution: INVALID → ---
(Reporter)

Updated

3 years ago
Whiteboard: Closed

Comment 4

3 years ago
(In reply to brunoais from comment #3)
> Without a reason specified, I cannot accept this to be just closed.
I close this bug because this bug belongs to this place https://support.mozilla.org
> Why are you closing this bug report?
I show you the right place to ask your question 
> If this is a feature and not a bug, please do point to where that is
> mentioned.
> If this is an undocumented change, then I cannot consider this as deliberate
> because it broke code and it didn't give any time to find a workaround!
Ok , if you want to workaround on this here, then you take the bug...
Flags: needinfo?(cskumaresan)
> I close this bug because this bug belongs to this place https://support.mozilla.org

How does that follow?  An IDB database worked in 40, doesn't work in 41, how is that a support issue?  Sounds like a regression bug to me.

brunoais, I've spun up a try build with the printfs we discussed on IRC.  The build, once it exists, should be in https://archive.mozilla.org/pub/firefox/try-builds/bzbarsky@mozilla.com-9348ce2bd16bd16e0ce5248471db806577bf93df/

Please let me know what it says, and whether mozregression tells you anything useful.
Flags: needinfo?(brunoaiss)
Keywords: regression
Jan speculated this was related to bug 1089764 but that landed in 35 ...
Assignee: nobody → Jan.Varga
(Reporter)

Updated

3 years ago
Flags: needinfo?(brunoaiss)
(Reporter)

Comment 7

3 years ago
Just an update before I go away for some more hours:
FYI and so that I don't lose these:
Mozregression says:
Latest good from mozilla-central: 2015-06-24
Earliest bad from mozilla-central: 2015-06-25
Latest good from mozilla-inbound: ?
Earliest bad from mozilla-inbound: 1019348c77860c34354e5674e21cbba09877c891

Inbound test interval: c319f262ce3ea3db52aad981abb9f33e36857c3d - 5b38df79819f9a6d45ec057dad29cc000c19a425

(I'll add myself to needinfo here so that I can make sure not to forget coming back here today)
I hope these are useful for a start
Flags: needinfo?(bzbarsky)
Flags: needinfo?(brunoaiss)
(In reply to brunoais from comment #7)
> Earliest bad from mozilla-inbound: 1019348c77860c34354e5674e21cbba09877c891

http://hg.mozilla.org/integration/mozilla-inbound/rev/1019348c77860c34354e5674e21cbba09877c891

but that's bug 1174381 which seems unrelated.
(Reporter)

Comment 9

3 years ago
Seems like so, yes. I won't blame that one because it was unable to find the latest good in the inbound list.
IMO, The commit that caused this was a commit made between the one tagged as 2015-06-24 and commit 1019348c77860c34354e5674e21cbba09877c891.
I'm unsure how I do more testing in between those, though.
> Latest good from mozilla-central: 2015-06-24
> Earliest bad from mozilla-central: 2015-06-25

Assuming those are nightlies, that corresponds to this range: http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=be81b8d6fae9&tochange=0b2f5e8b7be5

That contains the IDB change from http://hg.mozilla.org/mozilla-central/rev/a6f67ef45731 but that would not affect this, I would think.
Flags: needinfo?(bzbarsky)
(Reporter)

Comment 11

3 years ago
Please disregard my previous comment about latest good and earliest bad. I probably made a mistake in the bisection that I hadn't noticed.
This time I made the bisection 3 times and it gave the same result.
These data now should be correct:

Latest good from mozilla-central: 2015-03-31
Earliest bad from mozilla-central: 2015-04-01
Latest good from mozilla-inbound: 9bf6a6c8404561bf9b29dbb1a69518cc14eba994
Earliest bad from mozilla-inbound: d4fdf509f24254ad07730f481c3763b136e6aa44

comparison link: https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=9bf6a6c8404561bf9b29dbb1a69518cc14eba994&tochange=d4fdf509f24254ad07730f481c3763b136e6aa44

This one makes much more sense than the previous one.
:bz please assist me here. Feel free to make test builds for me.
I can't check the code now to see if I can find anything that calls for my attention...
Flags: needinfo?(bzbarsky)
Jan's probably in a better position to comment here than Boris is.

Jan, see comment 11.
Flags: needinfo?(bzbarsky) → needinfo?(Jan.Varga)
(Assignee)

Comment 13

3 years ago
(In reply to brunoais from comment #11)
> comparison link:
> https://hg.mozilla.org/integration/mozilla-inbound/
> pushloghtml?fromchange=9bf6a6c8404561bf9b29dbb1a69518cc14eba994&tochange=d4fd
> f509f24254ad07730f481c3763b136e6aa44
> 

That's a big landing for IDB implementation, part of that was the switch to WAL which also affected quota handling.

Can you tell me how do you use unlimited storage ? I guess you must pass |{ storage: "persistent" }| to indexedDB.open() and then confirm it in the dialog launched by the browser.
Flags: needinfo?(Jan.Varga)
(Reporter)

Comment 14

3 years ago
> var req = indexedDB.open(DB_NAME, DB_VERSION);
Where
> const DB_NAME = "ConfidentialString";
> const DB_VERSION = 1;

It is doing to the "storage/default/" directory in the profile.

The dialog had long been confirmed about 2 years ago. I believe it was when storage reached 50MB.
Flags: needinfo?(brunoaiss) → needinfo?(Jan.Varga)
(Assignee)

Comment 15

3 years ago
So, your database is treated as default (which is actually temporary in your case), not as persistent. I'm not going to explain whole story in details since it's rather complicated. Basically, there were two prompts, the first one was used to allow storing any data for given origin and the second one when data reached 50 MB. The first one was later disabled, so I guess you just saw the second one and the permission for it was stored. However several months ago we removed the second prompt and re-enabled the first one, but only for new explicit persistent storage. (data stored as persistent was migrated to default storage)

So you just need to modify your code
from:
var request = indexedDB.open(DB_NAME, DB_VERSION);
to:
var request = indexedDB.open(DB_NAME, { version: DB_VERSION, storage: "persistent" });

The fact that you hit this problem now is that your 680 MB database(s) were already big enough that all modifications fit in free blocks of the database and the journal file was not quota tracked. That changed with the switch to WAL (write ahead log). For example sqlite needs to write into the WAL even when it's just opening a database and since WAL is now quota tracked indexedDB.open() fails.

I'll keep this bug open until you verify that |storage: "persistent"| fixes it for you.
Flags: needinfo?(Jan.Varga)
(Reporter)

Comment 17

3 years ago
Please wait a while I finish testing everywhere.
So far, so good on my testing profile.
(Assignee)

Comment 18

3 years ago
Unfortunately there's no easy way how to migrate your data from default to explicit persistent storage. I hope it's not a big problem for you.
(Reporter)

Comment 19

3 years ago
It is a problem, though...
I'll see if I can make a script to do that. If I can, I'll post it here for someone who stumbles upon this.
Then I'll be able to fully test (and now all those data incongruences make sense)
(Reporter)

Comment 20

3 years ago
Is it OK if I just issue a move on the directory inside "storage/default" to "storage/persistent"?
Flags: needinfo?(Jan.Varga)
(Assignee)

Comment 21

3 years ago
(In reply to brunoais from comment #20)
> Is it OK if I just issue a move on the directory inside "storage/default" to
> "storage/persistent"?

Yeah, but it's "storage/permanent". "storage/persistent" was used before we introduced default storage.
You might also want to check if the directory already exists in "storage/permanent".
Flags: needinfo?(Jan.Varga)
(Reporter)

Comment 22

3 years ago
UnknownError <unknown>
IndexedDB UnknownErr: ActorsParent.cpp:576 <unknown>

name:"UnknownError"

Message:
"The operation failed for reasons unrelated to the database itself and not covered by any other error code."

This is what happens when I copy the DB files to the directory "storage/permanent" then I try to open it in firefox 42 (stable version) using:

> var request = indexedDB.open(DB_NAME, { version: DB_VERSION, storage: "persistent" });

Anything I'm missing? Anything I can use to help you debug?
Flags: needinfo?(Jan.Varga)
(Assignee)

Comment 23

3 years ago
Make sure you also copy the hidden ".metadata" file and the subdirectory structure is the same:

storage/default/http+++127.0.0.1/idb
storage/default/http+++127.0.0.1/.metadata
Flags: needinfo?(Jan.Varga)
(Assignee)

Comment 24

3 years ago
And also make sure there's nothing else like a custom file, a backup file or directory.
(Reporter)

Comment 25

3 years ago
I copied that whole:
storage/default/http+++127.0.0.1 -> storage/permanent/http+++127.0.0.1
(Reporter)

Comment 26

3 years ago
P.S. It is confirmed that only the "official" files are in the corresponding directories
(Reporter)

Updated

3 years ago
Flags: needinfo?(Jan.Varga)
(Assignee)

Comment 27

3 years ago
Sorry it was late yesterday and I'm packaging for travel today...
It would be helpful if you could download a debug build of firefox and try it with it.
You should see some warnings in the console.
Flags: needinfo?(Jan.Varga)
(Reporter)

Comment 28

3 years ago
I'm trying with latest firefox nightly (in both single process mode and multiple process mode) but I'm getting nothing actually usable in the console:

> JavaScript strict warning: resource://gre/modules/commonjs/toolkit/loader.js -> resource://gre/modules/devtools/toolkit/webconsole/utils.js, line 549: ReferenceError: reference to undefined property aGrip.actor
> JavaScript strict warning: resource://gre/modules/commonjs/toolkit/loader.js -> resource://gre/modules/devtools/toolkit/webconsole/utils.js, line 229: ReferenceError: reference to undefined property aSourceURL[(aSourceURL.length - 1)]
> JavaScript strict warning: resource://gre/modules/commonjs/toolkit/loader.js -> resource:///modules/devtools/framework/toolbox.js, line 298: ReferenceError: reference to undefined property this._performance

But here's what I experience:

According to the OS's tools, when I try to open the DB in the latest nightly the disc traffic skyrockets to the maximum in use by firefox and SYSTEM. Both are using the:

> storage/permanent/http+++127.0.0.1/idb/2714291041GBMD__[code]_s.sqlite-shm

In a super aggressive way. It starts with firefox but then it's SYSTEM (I believe that it is because writes are set as async).
After some minutes, with my computer nearly locked down (being unable to use it due to resource hog firefox), this appears in the console:

> QuotaExceededError <unknown>

... and the resource hogging stops.
Flags: needinfo?(Jan.Varga)
(Reporter)

Comment 29

3 years ago
P.S.
After closing firefox in the console I had opened using CTRL + C, I just won a **21.1GB* size
"2714291041GBMD__[code]_s.sqlite-wal"
file.
(Reporter)

Comment 30

3 years ago
After thinking a bit...
Could it be that that 21GB file actually gets even bigger up to filling my disc producing the quota exceeded error thing?
(Reporter)

Updated

3 years ago
Flags: needinfo?(jvarga)
(Reporter)

Updated

3 years ago
Flags: needinfo?(jvarga)
(Reporter)

Updated

3 years ago
Flags: needinfo?(bzbarsky)
(Reporter)

Comment 31

3 years ago
Is it normal to wait this long?
Flags: needinfo?(bzbarsky)
(Reporter)

Updated

3 years ago
Flags: needinfo?(bzbarsky)
It depends.  If people are on vacation (tends to happen this time of year), then possibly yes.
Flags: needinfo?(bzbarsky)
(Assignee)

Comment 33

3 years ago
(In reply to brunoais from comment #30)
> After thinking a bit...
> Could it be that that 21GB file actually gets even bigger up to filling my
> disc producing the quota exceeded error thing?

Yeah that could be a problem, can you share your database with me ?
I would try to debug the internal upgrade process, but if you can't then I will have to give you instructions how to debug it on your side.
Flags: needinfo?(jvarga)
(Reporter)

Comment 34

3 years ago
My confidentiality contract prevents me from sharing it.
Still,
I may be able to make a clone of it with bogus data.

Is there a way to create a 2nd database file for the same domain? If so, I could make a script that takes in all the original data and transforms it into bogus data and then I can send you that database file.
(Assignee)

Comment 35

3 years ago
(In reply to brunoais from comment #34)
> My confidentiality contract prevents me from sharing it.
> Still,
> I may be able to make a clone of it with bogus data.
> 
> Is there a way to create a 2nd database file for the same domain? If so, I
> could make a script that takes in all the original data and transforms it
> into bogus data and then I can send you that database file.

Using the old version of firefox ? Yeah I think so.
Just pass a different database name to indexedDB.open()
Also, if you want to investigate locally, you can use the NSPR_LOG_MODULES and NSPR_LOG_FILE environment variables (https://developer.mozilla.org/en-US/docs/Mozilla/Projects/NSPR/Reference/NSPR_LOG_MODULES) to dump what IndexedDB is doing at a high level and what it is doing at a lower level by also logging the SQL operations occurring.  It looks like you are on Windows, so I think the commands would look something like this in a cmd shell prior to starting firefox:

set NSPR_LOG_MODULES=timestamp,mozStorage:5,IndexedDB:5
set NSPR_LOG_FILE=C:\database-stuff.log

If you want the environment values persisted (you probably don't), it looks like Windows 7 onwards now support a "setx" command of the form "setx variable value".

This could potentially help make it clear what is going into a crazy writing loop since the IndexedDB logging will capture all the requests your code makes and the mozStorage logging will capture what is being done to the database, including things happening during any upgrade process, etc.

Note that the log generated may include data that may or may not have privacy ramifications, so please check it before sharing the logs.  Namely, I think IndexedDB keys that are not made up of arrays will be included.  Turning down the IndexedDB logging level will eliminate those, but I think it will make the logs less useful.
(Reporter)

Comment 37

3 years ago
Thanks for the information Andrew but I still prefer someone who is more knowledgeable to do it in addition to myself.

______________________________

An e-mail has been sent to :janv with the link to where he can download the database. I'm not sharing the database publicly because I'm unsure if how I transformed the data is safe enough to prevent information leakage. Sorry about that... But I trust that :janv will only share with Mozillians whom I trust to use such data correctly.
Flags: needinfo?(jvarga)
(Assignee)

Comment 38

3 years ago
It's an internal upgrade of the database that causes all these troubles. I'm looking at the upgrade from internal version 17 to 18 which happened for firefox 40 bug 1131776.
I'll look if it's possible to optimize the upgrade.
Flags: needinfo?(jvarga)
(Assignee)

Updated

3 years ago
Status: UNCONFIRMED → NEW
Ever confirmed: true
(Assignee)

Comment 39

3 years ago
Anyway, the original issue that was reported here is actually fixed by using the extra argument for indexedDB.open(). Problems with internal upgrade is a separate issue and we'll need to file a new bug for it probably.
(Reporter)

Comment 40

3 years ago
Feel free to file a new bug if needed. Just don't forget to make this one as related please.

Updated

6 months ago
Priority: -- → P3
You need to log in before you can comment on or make changes to this bug.