1083852 - Humble Bundle games get QuotaExceededError at 50mb, no UI prompt

Status: RESOLVED FIXED

(Core :: Storage: IndexedDB, defect)

Description

[Luke Wagner [:luke]]

IIUC, the current policy is that you can store up to 50mb of data in IDB per eTLD+1 and that goes into persistent storage.  If you try to go beyond that, you get an async user prompt and then you get unlimited storage after that.

What I'm seeing is:
 - new profile, so empty storage/ dir
 - visit http://humblebundle.com/
 - click "Play" on Super Hexagon (then the big button on the canvas to start)
 - the game downloads then stores about 18mb in persistent storage in storage/persistent/https++www.humblebundle.com
 - reload the page and restart Super Hexagon and notice no download; it's cached.
 - now click "Play" on "Aaaaa! for the Awesome" (then the big button to start)
 - the game downloads, there is no storage prompt
 - there is a QuotaExceededErrors in the JS console and several more once the game starts up
 - if you reload the game, it redownloads, more QuotaExceededErrors

Also, the total size of storage/persistent/https+++www.humblebundle.com is left at 50mb, suspiciously close to the persistent storage prompt limit.  So I'm guessing something is inhibiting the prompt.  Can this possibly be the application's fault?

Comment 1

[Luke Wagner [:luke]]

I should add that, once storage/persistent/https+++www.humblebundle.com hits 50mb, all other games (other than the first game that successfully cached) fail to cache.

Comment 2

[Jukka Jylänki]

On my Windows computer running Firefox 33.0 (C:\ drive has 13.8GB of 111GB free), the browser is able to successfully cache all the games once I let them download. Closing the browser, relogging in to HB site, and the caches have persisted. The prompt did appear only on the startup of the first game. I'll try other systems as well.

Comment 3

[Luke Wagner [:luke]]

(In reply to Jukka Jylänki from comment #2)
I have 14gb available space; I assume that is plenty for persistent storage (which, iirc, doesn't use the whole fraction-of-free-space quota thing).  Just to be sure, could you try again with a new profile in current FF Nightly?

If I end up being the only one that can reproduce, I can try to dig in more to see where the failure is coming.  Jan: can you tell me a good pinchpoint to catch QuotaExceededErrors?

Comment 4

[Jukka Jylänki]

I can verify that on the same Windows desktop computer and current Nightly 36.0a1 (2014-10-16), I don't get a dialog pop up to ask me if I want to allow the page to cache, and subsequently I am getting QuotaExceededErrors in the log.

STR:
1. Create a new Firefox profile (start up with firefox -P)
2. Go to https://www.humblebundle.com/?sel=zenbound2_asm_demo and click Play.

Observed: No prompt that asks whether the user wants to store the data in the cache, and the console log shows QuotaExceededErrors.

Expected: A prompt should show up to ask if the user wants to store the game data file in IDBFS cache.

Like mentioned in comment 2, Firefox 33.0 works properly as expected on the same computer.

Comment 5

[Ben Turner (not reading bugmail, use the needinfo flag!)]

Sorry, we broke the UI prompt for quota stuff on purpose as part of bug 994190. We're working on fixing this in bug 1018787.

Comment 6

[Jukka Jylänki]

Firefox Aurora 34.0a2 (2014-10-13) works ok and does ask for the prompt, and so does Firefox Beta 34.

Comment 7

[Jukka Jylänki]

Oh interesting, thanks! It is very important that we don't slip in a browser release to stable channel where this was broken, as it pretty much destroys the experience for the site. Reading bug 994190, it looks like it's targeting 35 release, so it's all good? Can I request access to bug 1018787?

Comment 8

[Jan Varga [:janv]]

Are you guys talking about asm.js caching or caching using IndexedDB ?

Comment 9

[Ben Turner (not reading bugmail, use the needinfo flag!)]

(In reply to Jukka Jylänki from comment #7)
> Can I request access to bug 1018787?

Apologies, I meant bug 1083927 (copy-paste goof).

We will do something smaller and more specific for Firefox 35 here.

Comment 10

[Ben Turner (not reading bugmail, use the needinfo flag!)]

One option for you is to switch your idb usage to temporary. Instead of doing:

  indexedDB.open(NAME, VERSION);

You can do:

  indexedDB.open(NAME, VERSION, { storage: "temporary" });

That should work on nightly and aurora today.

Comment 11

[Ben Turner (not reading bugmail, use the needinfo flag!)]

(In reply to ben turner [:bent] (use the needinfo? flag!) from comment #10)

Sorry, the syntax is actually:

  indexedDB.open(NAME, { version: VERSION, storage: "temporary" });

Comment 12

[Luke Wagner [:luke]]

(In reply to Jan Varga [:janv] from comment #8)
> Are you guys talking about asm.js caching or caching using IndexedDB ?

Normal (non-asm.js) caching using IDB.

(In reply to ben turner [:bent] (use the needinfo? flag!) from comment #9)
> We will do something smaller and more specific for Firefox 35 here.

Just to make sure I understand: your plan is to fix this by the time FF35/36 are released?  If so, that'd be great!  Is there a bug we can block this on?

Comment 13

[Ben Turner (not reading bugmail, use the needinfo flag!)]

This will be the bug for fixing FF35. Fixing 36 will hopefully be bug 1083927, though whatever we do here will also ride that train.

Comment 14

[Ben Turner (not reading bugmail, use the needinfo flag!)]

[Tracking Requested - why for this release]:
We broke Humble Bundle :-/

Comment 15

[Lukas Blakk [:lsblakk] use ?needinfo]

(In reply to ben turner [:bent] (use the needinfo? flag!) from comment #14)
> [Tracking Requested - why for this release]:
> We broke Humble Bundle :-/

That's not good - who can work on this?

Comment 16

[Ben Turner (not reading bugmail, use the needinfo flag!)]

Jan has started working on this. We're going to do a small targeted fix for trunk/aurora here, then the full fix will be in bug 1083927.

Comment 17

[Jan Varga [:janv]]

Attachment: WIP patch

Comment 19

[Jan Varga [:janv]]

(In reply to Luke Wagner [:luke] from comment #0)
> IIUC, the current policy is that you can store up to 50mb of data in IDB per
> eTLD+1 and that goes into persistent storage.

Just for your information, the 50mb limit is per origin, not per eTLD+1 in the case of persistent storage. However, with default storage this changes and you won't see any prompts at 50mb.
Just like in the case of temporary storage.

Comment 20

[Luke Wagner [:luke]]

I'm sure you're right but: I thought the persistent 50mb limit was per-eTLD+1 to prevent a single eTLD+1 consuming unbounded space using tons of origins to shard the total payload into 50mb hunks.

Also: sorry to ask again, but could you remind me of how 'default' differs from 'temporary'?

Comment 21

[Jan Varga [:janv]]

(In reply to Luke Wagner [:luke] from comment #20)

> 
> Also: sorry to ask again, but could you remind me of how 'default' differs
> from 'temporary'?

see bug 1083927
basically, origins with app status != NOT_INSTALLED (apps) are treated as persistent storage and everything else as temporary storage

Comment 22

[Luke Wagner [:luke]]

Btw, is there anything left to do or is this WFM?

Comment 23

[Jan Varga [:janv]]

We're just going to request an approval for aurora approximately 1 week before the uplift to beta.
Humble bundle should just work with the current nightly. However the 50mb prompt shouldn't appear at all, since most of persistent storage is now treated as temporary storage.

Comment 24

[Maxime RETY]

Hello,

Sorry to bother, but I'm not sure of having understood your plans: are you planning to add a fix for the UI prompt in Firefox 35?

I'm worried to see a patch landing in Firefox 35 because the "temporary" fix:

    indexedDB.open(NAME, VERSION, { storage: "temporary" });

would not be applicable when users already have data stored in their "permanent" folder.

In my (offline capable) application, the users may have data stored in the "permanent" folder which is unsynched (yet) with my servers. This data would be lost, or at least unreachable after the switch to "temporary".

The changes proposed in https://bugzilla.mozilla.org/show_bug.cgi?id=1083927 would be fine though, since there's a migration path for this case ("permanent" becomes "default", previous data is migrated to the "default" folder).

Thanks for the clarification.

Comment 25

[Jan Varga [:janv]]

(In reply to Maxime RETY from comment #24)
> Hello,
> 
> Sorry to bother, but I'm not sure of having understood your plans: are you
> planning to add a fix for the UI prompt in Firefox 35?
> 
> I'm worried to see a patch landing in Firefox 35 because the "temporary" fix:
> 
>     indexedDB.open(NAME, VERSION, { storage: "temporary" });
> 
> would not be applicable when users already have data stored in their
> "permanent" folder.

no, that was just an earlier suggestion for those who can't wait for a fix in Firefox

> 
> In my (offline capable) application, the users may have data stored in the
> "permanent" folder which is unsynched (yet) with my servers. This data would
> be lost, or at least unreachable after the switch to "temporary".
> 
> The changes proposed in https://bugzilla.mozilla.org/show_bug.cgi?id=1083927
> would be fine though, since there's a migration path for this case
> ("permanent" becomes "default", previous data is migrated to the "default"
> folder).
> 

Yes, but for Firefox 35 (current Aurora), we plan to land a fix that just treats persistent storage as temporary storage. So there won't be any changes in directory/folder structure on disk and the 50mb prompt won't be displayed anymore, we will just evict the least recently used origins when temporary storage reaches its limit (50% of free disk space).
For Firefox 36, we plan to land a fix for bug 1083927 which moves data from persistent to default folder and introduces explicit persistent storage. Explicit persistent storage will have the first prompt (before any data is stored for particular origin) and we will later decide if we also enable the 50mb prompt for explicit persistent storage.

Explicit persistent storage:
indexedDB.open(NAME, { version: VERSION, storage: "persistent" });

Default storage:
indexedDB.open(NAME, VERSION);
or
indexedDB.open(NAME, { version: VERSION, storage: "default" });

Explicit temporary storage:
indexedDB.open(NAME, { version: VERSION, storage: "temporary" });

Comment 26

[Maxime RETY]

Thanks Jan for your reply.

> Yes, but for Firefox 35 (current Aurora), we plan to land a fix that just treats persistent
> storage as temporary storage. So there won't be any changes in directory/folder structure on
> disk and the 50mb prompt won't be displayed anymore, we will just evict the least recently
> used origins when temporary storage reaches its limit (50% of free disk space).

Ok I get it now, seems fine.

Do you know if the new syntax for version/storage options will be standardized in the spec at some point?

Comment 27

[Jan Varga [:janv]]

(In reply to Maxime RETY from comment #26)
> Do you know if the new syntax for version/storage options will be
> standardized in the spec at some point?

Yes, it should be, in IDB v2 probably.

Comment 28

[Ben Turner (not reading bugmail, use the needinfo flag!)]

This landed, http://hg.mozilla.org/mozilla-central/rev/2266676732ef

Comment 29

[Lukas Blakk [:lsblakk] use ?needinfo]

Can we get an uplift nomination here for Beta/Aurora so this can get into the last beta of the cycle on Mon Dec 29th?

Comment 30

[Jan Varga [:janv]]

Oh, sorry for the confusion, this was fixed on m-c and aurora in bug 1089764.

Comment 31

[Lukas Blakk [:lsblakk] use ?needinfo]

Thanks!