648064 - Application cache should not bother user

Status: RESOLVED WONTFIX

(Core :: Networking: Cache, defect)

Description

[Benjamin Stover (:stechz)]

Assuming that we should be much more aggressive about caching and we want to give web developers the power to have pages load fast, we should stop asking for the user's permission to use the application cache. Appcache allows for a model where the developer can immediately show content without touching the network, asking questions about whether the content is fresh later.

Comment 1

[Louis-Rémi BABE]

Attachment: Patch v0 -- Remove user prompt & enable app-cache by default

Hey, I wrote this patch, and it should solve our problem!

Comment 2

[Andrew Sutherland [:asuth] (he/him)]

Comment on attachment 548519 [details] [diff] [review]
Patch v0 -- Remove user prompt & enable app-cache by default

I'm definitely not the right reviewer for this, guessing bz based on the modules list and his general high levels of wisdom.

Comment 3

[Boris Zbarsky [:bzbarsky]]

Seems like a UX review is what you're really looking for there, no?  Failing that, probably Honza, not me.

Comment 4

[Honza Bambas (:mayhemer)]

I'm probably also not the best one to review this (except giving r-).

Yes, I would love to remove the prompt my self.

Yes, application cache can be used for load optimizations by actually any web site, if we don't bother users with the prompt.

But, there are problems:

- by agreeing, user gives the site a special permission, so called "offline" permission, that changes the behavior of evicting persistent storages (localStorage, application cache) ; we don't know, how to recognize a web site has this offline permission to change the behavior correctly, unless it is automatically and silently given it when the manifest attribute is found

- evicting offline enabled site's local stored data is done a different way then evicting cookies+localStorage for a regular web site ; let's say that a web site having the manifest attribute immediately gets the permission - then its localStorage data will not be evicted along with cookies (hidden privacy issue) 

- offline enabled site can allocate about 50 MB of disk space to cache stuff, that can only be deleted manually by user (buried deep in the options dialog) or by the web site admin (an easy DoS attack)


These issues must be addressed first to allow change like this.  I have received calls from few people, and I have read such calls my self, the behavior of offline application caching should change to allow a better and wider usage for it as an optimization.

That, with high probability, means to change the spec.  At the moment I have no concrete suggestions, but one day I want to collect them.

Comment 5

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

I believe Chrome does this by restricting sites to N MB of usage (where N is 5, according to some slides from a Google developer engagement person that I've seen).  We'd need to figure out what we want to do for sites with more than N MB of data to store ...

Anyways, removing this prompt is on blizzard's wishlist if we can make it happen.

Comment 6

[Robert O'Callahan (:roc) (email my personal email if necessary)]

Chrome gets away with no prompt by treating the appcache as temporary storage so they can evict whenever they need to. Which IMHO demolishes the original, and very important use-case for appcache --- reliably available offline apps.

However, I think we can solve this adequately without a prompt.

I think we should let applications use appcache and other persistent local storage freely without prompts until we determine that data is using "too much" storage (depending on the device capacity I guess), at which time we involve the user in making eviction decisions. For example, we could prompt the user (non-modally) saying "hey, we're going to remove the storage for these sites (greediest/least recently used); click here to change this decision". Clicking takes the user to a page where we can show the installed apps, application and storage usage over time, screenshots, etc and the user can evict whatever. "Clear Recent History" should not clear persistent local storage by default, but it should expose a way to do so (e.g. the eviction dialog suggested above).

Comment 7

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

We could have a smaller appcache limit (like Chrome), and a larger limit that is enabled by making the page/site an app tab and/or by installing an open web app. Using a percentage of free disk space doens't work well, especially on a mobile device, but even on a desktop/laptop; eventually users get short on space. IMO, we need to have some kind of automatic eviction of this cache based on LRU.

Comment 8

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

Perhaps I remember incorrectly, but I thought there was also a privacy concern that motivated the dialog box. Can this feature be used as a way to install un-removable tracking "cookies"? I am pretty sure that we can discount the privacy benefits of the dialog box (which I also hate), by showing there are easier ways to track the user without this feature in place. But, we should keep this in mind.

Comment 9

[Robert O'Callahan (:roc) (email my personal email if necessary)]

(In reply to Brian Smith (:bsmith) from comment #7)
> open web app. Using a percentage of free disk space doens't work well,
> especially on a mobile device, but even on a desktop/laptop;
> eventually users get short on space. IMO, we need to have some kind of automatic
> eviction of this cache based on LRU.

We could default to that but tell the user what we're going to do and let them override it (permanently), like I suggested above.

Comment 10

[Honza Bambas (:mayhemer)]

Offline-enabled host can store cookies via localStorage that won't get deleted when using "delete history" dialog or when user deletes all/per-host cookies via the cookies manager.  If we allow this w/o noticing the user then it's a privacy issue.

What Robert suggests sounds good.  However I would limit that prompt only after a site, that has the manifest= attribute, attempts to store user data, e.g. to localStorage.  Until user accept it, we will treat localStorage (et al) for that host same way as cookies.

Then we would have:
- automatic permission to use offline application cache, whom eviction might have different rules then the regular cache
- manual permission to store use data (via interfaces that are normally used to store cookies-like data) having different eviction rules after user accept it (with a hint how to delete them; btw, we don't have a UI to delete user-data now)

Comment 12

[Daniel Veditz [:dveditz]]

Comment on attachment 548519 [details] [diff] [review]
Patch v0 -- Remove user prompt & enable app-cache by default

Review of attachment 548519 [details] [diff] [review]:
-----------------------------------------------------------------

Flipping the existing prefs is so far from an acceptable solution that this patch is only a distraction from the design work that needs to be done.

Comment 13

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

See http://jclaes.blogspot.com/2012/03/how-web-application-can-download-and.html ("How a web application can download and store over 2GB without you even knowing it"), which discusses a problem with existing implementations.

Also, here is my comment from bug 715789, which was dup'd to this one:

Any time we allow a website to automatically use some of the user's resource, we have to also implement sensible limits for that usage, and implement an automated way of freeing those resources. Without any user interaction here, it seems to me that the disk space used by the app cache has to be garbage collected in the same way, and with the same priority, that we would garbage collect disk space in the (non-appcache) disk cache. If so, then I don't see why we shouldn't implement the app cache as just "preload all these resources into the disk cache." And, then, why would a website want to use appcache at all?

With a user action, then it makes sense to manage the appcache specially so that those resources are permanent or at least much less likely to be deleted. This seems to be the intention of the appcache mechanism. So, I wonder if the problem is the existence of any prompt, of if the problem is the *way* we prompt.

Without the prompt, would it be able to automatically eat up all the user's disk space with appcache, either with a single domain, or by creating a bunch of subdomains that each request appcache access?

How do we prevent greedy apps from hogging all the space we allocate for the app cache, pushing out other apps?

Do the limits we set for the app cache make sense for mobile? My understanding is that Firefox itself, without any apps installed, is already using too much disk space on mobile. This is why we limit the disk cache to 10MB. Assuming we only have 10MB of disk space total to spend, how much should we allocate for the appcache and take away from the disk cache? (Actually, I think appcache should steal this disk space from places--i.e. we should shrink the size of the places database by whatever amount we want to reserve for the app cache.) This is a problem that is somewhat "solved" by the prompt currently, because the user is explicitly choosing to have the additional storage space consumed.

Comment 14

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

Also, see the following comment from Hacker News user njs12345:

"I wonder if you could still perform a DOS by doing the following:

  - register 1000 domains
  - when the browser navigates to the first domain, store 5Mb
  - once the store has finished, redirect to the next domain
  - repeat steps 2-3 ad infinitum"

Comment 16

[Bryan Clark (DevTools PM) [:clarkbw]]

Here are some related links:

Discussion for Google Chrome app cache / storage:
http://code.google.com/p/chromium/issues/detail?id=61676

Chrome HTML5 Storage guidelines:
https://developers.google.com/chrome/whitepapers/storage

Comment 17

[cscott]

I suggest that appcache space should be requested/allocated in a webapp's manifest, so that the user knows how much disk space they are allocating when they install the app: bug 792651 comment 2.

Comment 18

[Honza Bambas (:mayhemer)]

Also bug 744713 could be related.

Comment 19

[olivier]

A failing use case with the current implementation:

We have an application with pluggable webapp called expressions that everyone can use to post microblogging style content. The expression will define the way you will add and mix content and the generated output.
Web developpers can develop and publish new expressions that will let other create post in a directed (and cool!) way.

Those webapp are served into iframes from a CDN. To improve performances, as you will reuse expressions over and over, we generate a cache-manifest for each of them.

On Firefox, users are prompted the message multiple time:
1) this is annoying as it breaks the flow of the navigation,
2) this is suspect because the declared website URL in the iframe is the one of our CDN instead of the website.

Today, this limitation is a no-go for us to use cache-manifest, so sad.

We are actually satisfied with a cache limited to 5MB like in others browsers, even if its break the 'reliable offline cache' use case.

The presented solution where the question is asked later if the amount of data become too big is also a problem as long as we cannot control what is loaded or not. But if we have a way of asking for the amount of data currently stored, we can also live with this proposition.

In a perfect world, I would have a small amount of available space and I can know what is 'my available quota' before deciding to load cached resources. I would also be able to query upfront to know if a specific URL is currently stored using a cache manifest.

Comment 20

[john smith]

Is there a way of telling firefox to ignore the cache manifest?

That would solve all the problems in a easy way.

Comment 21

[Honza Bambas (:mayhemer)]

Hasn't this been fixed with bug 892488?

Comment 22

[Patrick McManus [:mcmanus]]

app cache is going away - so if this isn't already fixed, wontfix