928438 - Need in-app downloads of additional content

Status: NEW

(Web Apps :: General, enhancement)

Description

[Ben Bucksch (:BenB)]

I am writing a packaged, offline app that contains text content, which is the focus of the app and to which the app is essentially an interactive access. The text is available in several languages (currently 6, more coming), and the there is a lot of content: 20-40MB zipped *per language*. Thus, I can't put 6*40 = 240 MB in the app package. (I could make one app per lang, but I'd have 10 apps, and that's cumbersome for me and users, see bug 928428.)

The same problem exists for dictionaries, mapping apps (anything with geographical data), and many other apps that need a lot of data and that want to work offline. It's therefore a generic problem.

The best solution would be an in-app downloader. However, that's non-trivial and I don't want to write this myself, it should be part of the OS API. Most importantly, the additional resources should be accessible by the app in the same way (same kind of URLs) as internal app resources. Also, the resources should be stored in the same space as apps - the user might not have an SD card inserted, might change it etc..

As for download implementation, we need:
* Selection UI - shows list of extra packages, with title, description, download size. Allows to select several packages at once, and trigger download.
* Download progress UI - Progress meter per package. One download at a time. Possibility to background the download UI and continue on another app or even the same app.
* Download logic - Should be able to work in background. Needs to track progress. Need to be able to resume aborted downloads (network problems, user needs to use the network for something else more urgent).
* The download target is determined by OS, just like for apps. The user might get the option in OS preferences to set a specific location, e.g. SD card.
* File (XML or JSON) on server lists current content available for download and the download URLs and possibly a hash. App contains the URL to the listing file.
* UI to uninstall extra packages UI, by OS, e.g. in settings UI.

As for access implementation, we need:
* A URL that allows to access these extra resources. E.g. the URL path could contain the extra package name, which is e.g. specified in the listing file. The scheme and "origin" must be the same as in-app resources.
* Some way for the app to get a list of installed extra resources and their base URL.

Comment 1

[Ben Bucksch (:BenB)]

Importance:
* I need to have same-origin access to these extra resources. Otherwise I need to re-write half the app.
* If a user doesn't have an SD card inserted, but sufficient internal storage, a requirement to download to SD would block him from using my app.
* It's not reasonable to have all that code above as part of my app. All that functionality has nothing to do with my app topic, which is a book.
* As part of the OS, it gives a consistent UI to manage such large resources and disk space. Much better user experience.
* The same or a similar system could be used for in-app purchases.

Comment 2

[Kumar McMillan [:kumar]]

even though it's non-trivial I think this would be better as a third party library that an app could include. I don't see a strong use case for adding it to the platform.

Comment 3

[Ben Bucksch (:BenB)]

I gave those reasons in comment 1.
* A third-party implementation cannot make the additional resources available under the same "origin" as app resources, can it?
* Can an app write to device space (not SD card)?
* The user should be able to manage disk space in a central OS place. These extra downloads are probably the first things that a user should remove when the storage space is low, not whole apps. Also, the OS could give the user could have the option of moving it from device space to SD card space and vise versa, which an app cannot.

Comment 4

[Ian Bicking (:ianbicking)]

I expect a third-party library would be based on IndexedDB, which would at least satisfy your first two requirements (same-origin access, available via the URL object, and device space), though not your third.  If you used the device storage API you'd also effectively be okay on the first and third points – though because the data would be blobs and not a "resource" per se, you might have to restructure your app some.

Ideally to get access to the list of available resources you'd have a CORS-accessible list (https://developer.mozilla.org/en-US/docs/HTTP/Access_control_CORS) – and individual items would also be accessible that way.  You'd request the items via XMLHttpRequest, and then save them locally in IndexedDB and use URL.createObjectURL (https://developer.mozilla.org/en-US/docs/Web/API/URL.createObjectURL) if you needed to access those resources via a URL (or if you just need to load the data into memory, then load the data from IndexedDB).

Comment 5

[Ben Bucksch (:BenB)]

Thanks for the pointers. Unfortunately, none of this fulfills all the needs stated in comment 1 (and 3).

> though because the data would be blobs and not a "resource" per se, you might have to restructure your app some.

* Right. The API to IndexedDB is very different to that of files in the assets/resources, which I load via XHR.
* The user can't selectively remove some packages from IndexedDB, but not other packages and leaving user data. You can't make a generic UI for this, because it's a DB. You can with my proposal.
* I am an unprivileged app, and I don't want to ask for any privileges whatsoever (it's just a book!).

I think it is a common need for many apps, otherwise I wouldn't ask for it:
* dictionaries
* interactive books / library apps / reference works (like mine)
* maps
and others

Comment 6

[Kumar McMillan [:kumar]]

(In reply to Ben Bucksch (:BenB) from comment #3)
> * The user should be able to manage disk space in a central OS place. 

The Device Storage API can be used for this: https://wiki.mozilla.org/WebAPI/DeviceStorageAPI

Comment 7

[Ian Bicking (:ianbicking)]

> I am an unprivileged app, and I don't want to ask for any privileges whatsoever (it's just a book!).

If the resources you are downloading are served with CORS headers (https://developer.mozilla.org/en-US/docs/HTTP/Access_control_CORS) then you do not need any special privileges to fetch them.

You are correct that if you use Device Storage you'll need extra permissions.  Arguably there should be a way to get access to application-private device storage that does not require extra permissions, but allows other application (that do have heightened permissions) the ability to manage those files (or at least delete them).  A proposal along those lines might be more productive.

Comment 8

[Ben Bucksch (:BenB)]

You all focus on different aspects separately: Selection, download, storage, access. But they all work together. I just want to separate a part of my app's resources in a separate download, but *still install and access them exactly the same way as the in-app resources*.
* The user should have roughly the same UI as the app store, (including selection of packages, download meter, suspending and resuming)
* the storage location should be the same as the app package, and
* the URLs the app accesses the resources should be the same form and the same origin as in-app resources.
A solution must meet *all* the requirements at once, not just 2 of 3.

I get that you don't want to add new APIs, but use existing ones, but this one is needed.

This is something that Android gets totally wrong.
* I hate it when lots of apps need access to my SD card, and
* I hate it when they all create directories there that I don't want, just to download their maps, their dictionaries, their books.
* I also hate that each app has its own thrown-together downloader where I don't know how to select and download several modules at once,
* nor can I pause and resume the download when I notice that the download takes to long and I need the network connection for something else, or my network connection keeps dropping out
This is because an *API is lacking*, and the result is horrible for both the end user (above usability problems) and the developer.

This can't be solved with existing APIs. This isn't about file storage and download, it's about splitting up packages into several components. Right now, you have only monolithic packages that must contain everything. That's good for tiny, simple apps, but fails as soon as things get a tiny little bit more complex. You can't rewrite the whole app just because you need to separate out one component.

Comment 9

[Ben Bucksch (:BenB)]

> You are correct that if you use Device Storage you'll need extra permissions.
> Arguably there should be a way to get access to application-private device storage that
> does not require extra permissions, but allows other application (that do have heightened
> permissions) the ability to manage those files (or at least delete them).
> A proposal along those lines might be more productive.

Yes, I think that's missing, too. I filed bug 929462 about this.

This doesn't solve this problem here, though, unless my app can access these files using URLs with the same "origin" as in-package files that are part of the app. And even with that, and with a third-party lib that creates the selection and download UI, the user still couldn't centrally uninstall individual components of apps (e.g. maps, dictionaries), only whole apps (all maps).

The advantage in my proposal here is that the OS can notice that the user's disk space is low, and open an OS app that allows to uninstall stuff, and can allow to individually uninstall maps for US, but not those for Europe, not whole apps. This is the most reasonable recourse when disk space gets low, and given that you aim at low end devices, this is going to be very much relevant in real life. This isn't possible with any of your proposals.