Last Comment Bug 704128 - Maybe implement File API: Directories and System
: Maybe implement File API: Directories and System
Status: NEW
: dev-doc-needed
Product: Core
Classification: Components
Component: DOM (show other bugs)
: Trunk
: All All
: -- normal with 10 votes (vote)
: ---
Assigned To: Nobody; OK to take it and work on it
:
: Andrew Overholt [:overholt]
Mentors:
: 843439 (view as bug list)
Depends on:
Blocks: 832732
  Show dependency treegraph
 
Reported: 2011-11-21 07:16 PST by Ben Francis [:benfrancis]
Modified: 2013-10-18 02:35 PDT (History)
47 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments
Mega's a word on browsers (348.69 KB, image/png)
2013-01-19 16:44 PST, mgtemp
no flags Details

Description Ben Francis [:benfrancis] 2011-11-21 07:16:25 PST
Mail sent to dev-webapi...

While working on gallery & camera apps for B2G I've come across the use case where the user wants to copy photos to another device.

Most digital cameras and camera phones currently follow the DCF standard for the storage of photographs, which specifies a particular directory structure, file names and file formats for storage in a filesystem. When the user wants to copy their photos to another device they can usually just plug it in to a USB host and mount the file system via USB mass storage in order to copy or move files using a desktop file manager.

Currently the only option available in B2G for local storage of photographs is storing the base64 encoded in IndexedDB, although storing files in IndexedDB is coming. However, this does not make it possible to mount the files as a filesystem via USB so does not satisfy this use case in the usual way.

The W3C's File API: Writer specification is currently being implemented, but the use case described here is outside the scope of this spec. The W3C has a working draft of another specification called File API: Directories and System, which would enable use cases more similar to the one given here.

Should File API: Directories and System be implemented, and implemented in such a way that files written via this API could be mounted as a filesystem over USB?

Alternatively, should B2G provide some other mechanism for moving media to another device in a more web-like fashion over an IP network rather than USB?
Comment 1 Olli Pettay [:smaug] 2011-11-21 07:33:05 PST
Last time I looked at File API:D&S it wasn't a very good API, but I didn't care too much about it
since it wasn't clear at all whether we would ever implement it.
If we now decide to implement it, better to make sure the API is reasonable.
Comment 2 Kyle Huey [:khuey] (Exited; not receiving bugmail, email if necessary) 2011-11-21 07:34:10 PST
I really don't want to implement D&S, and we should explore alternative solutions for B2G.
Comment 3 Ben Francis [:benfrancis] 2012-01-05 03:47:08 PST
Jonas,

This is the thread discussing a potential "Gallery API" we talked about in the B2G meeting https://groups.google.com/d/topic/mozilla.dev.webapi/ODGDpW8Rs6M/discussion but I'm still not sure that's the right direction, it might be better to have something lower level for general file access.
Comment 4 Marco Castelluccio [:marco] 2012-01-05 04:01:12 PST
If we don't want to use something like the filesystem (that could be a good idea, to get rid of this old way of managing files), we should investigate a way to let two devices share files.
For example give the possibility to (in some way) "mount" IndexedDB as a filesystem?

I think we shouldn't create an API for each single application, but something more generally usable.
Comment 5 Chris Jones [:cjones] inactive; ni?/f?/r? if you need me 2012-01-05 10:56:23 PST
We don't have all the generic platform pieces in place yet, but here's a way this could work in the future

 - a USB manager application registers itself as a mass storage device using WebUSB
 - the USB manager uses intents to query other applications and build a list of "virtual files" to offer
 - the host requests transfer of files, which the USB manager satisfies through by streaming them over postMessage channels from the hosting application

We don't have all the technology for that yet.  In the meantime, we should ask ourselves whether it's worth implementing an API of unclear utility, vs. relying on cloud photo managers like flickr as stopgaps.  (Or adding in other temporary hacks that are not intended to be on a standards track.)
Comment 6 Ben Francis [:benfrancis] 2012-01-05 11:22:22 PST
Although slightly complicated, that does sound like a good idea to me. Better than trying to standardise something very specialised like an API for browsing photos on a device or enforcing the concept of file systems on the web.

In the meantime I think a gallery app is still useful as long as you can capture an image using the camera and share that image with another app using web intents.

A music player is less useful, unless you can download music from somewhere, but could be used via intents to play audio from another app.
Comment 7 Chris Jones [:cjones] inactive; ni?/f?/r? if you need me 2012-01-05 12:06:00 PST
(In reply to Ben Francis from comment #6)
> In the meantime I think a gallery app is still useful as long as you can
> capture an image using the camera and share that image with another app
> using web intents.
> 

Of course!  The hypothetical USB "fuse" above would talk to a gallery app with intents.  And also flickr, etc.

> A music player is less useful, unless you can download music from somewhere,
> but could be used via intents to play audio from another app.

I don't follow the logic here.
Comment 8 Philipp von Weitershausen [:philikon] 2012-01-05 17:18:01 PST
(In reply to Chris Jones [:cjones] [:warhammer] from comment #5)
> We don't have all the generic platform pieces in place yet, but here's a way
> this could work in the future
> 
>  - a USB manager application registers itself as a mass storage device using
> WebUSB

... or as an MTP-capable device. MTP is the protocol that replaces mass storage capabilities on Honeycomb and later. The gallery, music player, etc. apps could interface with the MTP layer using intents, indeed.
Comment 9 Chris Jones [:cjones] inactive; ni?/f?/r? if you need me 2012-01-05 17:23:25 PST
Yes, without knowing that the USB broker is speaking MTP instead of usb-fuse.
Comment 10 Geoff Flarity 2012-04-20 06:45:39 PDT
The WD was updated recently:

http://www.w3.org/TR/file-system-api/

Just because this isn't the right solution for B2G file sharing, doesn't mean it isn't useful or important. Being able to work in a sandboxed fs is important for the future of the Web Platform. Please, implement this.
Comment 11 Olli Pettay [:smaug] 2012-04-20 08:18:25 PDT
The API is reasonable bad, so I would prefer implementing something better.
Comment 12 Geoff Flarity 2012-04-20 08:40:31 PDT
I'm developing applications against it right now in Chrome. I agree, it could definitely be better. 

However:

"It is intended to be minimal in extent, but sufficiently powerful that easy-to-use libraries may be built on top of it."

So far it is indeed "sufficiently powerful" and I am indeed building easy to use libraries on top of it.  This functionality was needed last year.
Comment 13 Kyle Huey [:khuey] (Exited; not receiving bugmail, email if necessary) 2012-04-20 11:55:44 PDT
What use case isn't covered by support for files in IndexedDB?

A word of caution: at the last TPAC meeting none of the other browser vendors had plans to implement File Systems and Directories.
Comment 14 Geoff Flarity 2012-04-20 12:11:08 PDT
Thanks for the heads up regarding vendor plans. 

Is there a doc describing files support for IndexedDB would look like? The application is encrypting and storing (potentially large) files that are chosen from outside the sandbox into the sandbox. Then being able to upload the sandboxed encrypted file via a XHR. 

So if I can read/write blobs from indexedDB *quickly* that should be enough.
Comment 15 Sonny Piers [:sonny] 2012-04-20 15:01:37 PDT
> Is there a doc describing files support for IndexedDB would look like?
The IndexedDB specification?

> So if I can read/write blobs from indexedDB *quickly* that should be enough.
Depending on what you mean by "quickly", you already can.
Comment 16 Geoff Flarity 2012-04-20 20:08:58 PDT
>The IndexedDB specification?

I did check there. It's not obvious that 'any' value means a Blob will be stored as expected.  In fact, it doesn't work in Chrome, though it's been acknowledge as bug and assigned.

The Mozilla hacks blob was informative:
http://hacks.mozilla.org/2012/02/storing-images-and-files-in-indexeddb/

>Depending on what you mean by "quickly", you already can.
I found the performance of Chrome's implementation to be surprisingly fast. I'll measure it as now I'm curious.
Comment 17 Jan Varga [:janv] 2012-04-20 21:04:55 PDT
(In reply to Geoff Flarity from comment #16)
> I found the performance of Chrome's implementation to be surprisingly fast.
> I'll measure it as now I'm curious.

There's no real Chrome's implementation for storing blobs at the moment, they just store blob urls at the moment.
Comment 18 Geoff Flarity 2012-04-21 05:52:15 PDT
>There's no real Chrome's implementation for storing blobs at the moment, they just store blob urls at the moment.

That may very well be, but I'm able to repeatedly save a 350MB file in about 12 seconds. I can't get a 480MB file to work without an 'ah snap' though. 

Assuming IndexedDB blob storage performs comparably (or better!), it should work fine for me.  Sadly, I won't be able to support FireFox until the FileSaver is implemented, but this is getting off topic enough already as it is. 

Thanks for all the quick responses and helpful feedback.
Comment 19 Olli Pettay [:smaug] 2012-04-24 22:36:06 PDT
Why dev-doc-needed?
Comment 20 Jean-Yves Perrier [:teoli] 2012-04-25 08:30:04 PDT
Olly: to track it. If the bug is R/Fixed in the future, we won't miss it. If it is R/Invalid or other, we will just remove the d-d-n at that point.

D-d-N really means documentation will be need when/if fixed.
Comment 21 cscott 2012-10-16 20:06:51 PDT
Just FWIW: the lawnchair adapter supports both Firefox and Webkit IndexedDB, and the webkit implementation is noticibly faster on the testsuite.  10.637 seconds on Firefox,  2.963 seconds on webkit.

That probably just means that IndexedDB should be improved, not that the Filesystem API should be implemented, though.
Comment 22 cajbir (:cajbir) 2013-01-18 19:38:31 PST
It seems that "Mega", the new incarnation of the hugely popular "Megaupload", will be using the file-system-api. According to an Ars Technica article the site, which goes live in a day or so, is recommending Firefox uses use Chrome: http://arstechnica.com/business/2013/01/mega-arrives-ars-goes-hands-on-with-kim-dotcoms-cloud-storage-site:

"Once you create an account, the first thing the site does is generate that crypto key for you. As a Firefox user, Mega also warned me to switch to Google Chrome, as not doing so would “adversely affect [my] file transfer performance." A browser switch later, and I was faced with a blank “file manager,” which was practically begging me to upload files."

I asked for more information from the developers of Mega and got referred to the file-system-api spec:

https://twitter.com/bramosnl/status/292467486664839168

I think it would be good to implement this spec given that it's getting usage.
Comment 23 Kyle Huey [:khuey] (Exited; not receiving bugmail, email if necessary) 2013-01-18 20:04:02 PST
http://ericbidelman.tumblr.com/post/21649963613/idb-filesystem-js-bringing-the-html5-filesystem-api

I still think we should WONTFIX this.  There was some discussion of combining this spec and B2G's device storage and cutting down the complexity, but we can do that in another bug if/when we decide to.
Comment 24 Olli Pettay [:smaug] 2013-01-19 03:02:11 PST
I wonder for what is Mega using Filesystem API. IndexDB should probably work just fine for them, unless
our IDB implementation is too slow.
Comment 25 Geoff Flarity 2013-01-19 05:47:14 PST
Meanwhile, back on earth...

If Mozilla's indexDB implementation is more than capable of handling gigabyte files efficiently, then I wish someone would just provide a mock around it that provides this interface FileSystem interface. Maybe run some tests that demonstrate it provides a comparable performance result...
Comment 26 Gervase Markham [:gerv] 2013-01-19 07:19:26 PST
(In reply to Olli Pettay [:smaug] from comment #24)
> I wonder for what is Mega using Filesystem API.

Presumably, saving actual files to the user's actual disk?
https://twitter.com/doublec/status/292466262049685504 and comments.

Gerv
Comment 27 mgtemp 2013-01-19 16:44:48 PST
Created attachment 704284 [details]
Mega's a word on browsers

At this point this probably doesn't add anything you don't already know but this is what they internally say (since it's overwhelmed by traffic at the moment).

If I'm understanding this correctly other use cases could include for example a webcam capture app (or any app fetching a lot of data) that wants to sequentially write files that will eventually become too large to hold on RAM.

They also complain about being unable to fetch data over HTTP from HTTPS (a warning on Chrome and Firefox <= 17, an error since Firefox 18), since HTTPS is not needed for their data transfers.
Comment 28 Kyle Huey [:khuey] (Exited; not receiving bugmail, email if necessary) 2013-01-19 16:46:23 PST
Firefox provides ways to store blobs outside of memory (put the blobs in IndexedDB).  AIUI they want to write to a specific file on the disk.
Comment 29 Olli Pettay [:smaug] 2013-01-20 02:33:20 PST
But doesn't file system API use files in a sandbox. That at least used to be the case. Based on attachment 704284 [details] they just want to store the data somewhere. With IE10 they talk about
disk-based Blobs, and we do have them.  And we have FileHandle for writing.
Comment 30 Olli Pettay [:smaug] 2013-01-20 02:42:39 PST
Ah, ok, it is not the kind of sandbox I expected. The files are there.
Comment 31 Olli Pettay [:smaug] 2013-01-20 02:58:44 PST
...but doesn't save the file using its name.
Comment 32 Olli Pettay [:smaug] 2013-01-20 04:38:21 PST
And IndexedDB stores files in local system too, at least on linux.
Comment 33 Jan Varga [:janv] 2013-01-20 05:02:47 PST
"Carefully avoids providing any API that would allow writing files from JavaScript" - this is completely misleading. Firefox supports storing blobs/files in IndexedDB and they are stored as real files (on all platforms) on disk. AFAIK, chrome doesn't support it yet and IE10 stores them in a database. There's also support for (partial) writing to existing stored files using FileHandle API. So we actually have quite solid support for "writing files from JavaScript".

I know, the current situation across browsers is not optimal, but stating that Firefox avoids to support it on purpose is a bit unprofessional.
Comment 34 Wesley Johnston (:wesj) 2013-01-20 09:25:47 PST
I've seen Google demo a file system API polyfill that they themselves wrote:

https://github.com/ebidel/idb.filesystem.js
Comment 35 Robert O'Callahan (:roc) (email my personal email if necessary) 2013-01-20 13:07:31 PST
Who's responsible for following this up with Mega?
Comment 36 Robert O'Callahan (:roc) (email my personal email if necessary) 2013-01-20 13:09:52 PST
http://lists.w3.org/Archives/Public/public-webapps/2012OctDec/0491.html is good ammunition for why the FileSaver spec shouldn't be implemented or used in its current form.
Comment 37 Robert O'Callahan (:roc) (email my personal email if necessary) 2013-01-20 13:12:29 PST
But, do we have an API that lets you upload entire directory trees? Or is that not covered by any spec yet?
Comment 38 Kyle Huey [:khuey] (Exited; not receiving bugmail, email if necessary) 2013-01-20 13:36:50 PST
(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #36)
> http://lists.w3.org/Archives/Public/public-webapps/2012OctDec/0491.html is
> good ammunition for why the FileSaver spec shouldn't be implemented or used
> in its current form.

FileSaver is something different from FileSystems.

(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #37)
> But, do we have an API that lets you upload entire directory trees? Or is
> that not covered by any spec yet?

No but Jonas has a plan for that.
Comment 39 cajbir (:cajbir) 2013-01-20 15:29:23 PST
(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #35)
> Who's responsible for following this up with Mega?

I've followed up with the Mega devs and will be getting more information on their API usage and needs.
Comment 40 Matthias Versen [:Matti] 2013-02-21 15:32:39 PST
*** Bug 843439 has been marked as a duplicate of this bug. ***
Comment 41 Matthew Phillips 2013-03-14 10:53:59 PDT
I'm a developer working on a programmer text editor that I would like to deploy as a packaged app. The user of a text editor expects to be able to open and save files in a directory of their choosing, not import files into a sandboxed directory that they need to export again to get back out.  Chrome provides this functionality currently: https://developer.chrome.com/beta/apps/fileSystem.html

Firefox's packaged apps a bit "nicer" in my opinion, but I can't use it without allowing the user access to the file system.
Comment 42 Ben Francis [:benfrancis] 2013-03-14 12:30:08 PDT
Matthew, do you know about the Device Storage API https://wiki.mozilla.org/WebAPI/DeviceStorageAPI ? Perhaps it could be used or extended for your use cases?

The Device Storage API was intentionally designed without exposing the complex hierarchical structure of the file system to the web, but allows for the storage of files in a shared location.

There's also the option of storing the files as blobs in IndexedDB inside your app.
Comment 43 Matthew Phillips 2013-03-14 12:35:51 PDT
Ben, my understanding of the Device Storage API (from reading the mailing list several months back, the documentation on MDN is lacking currently) was that it was a way to ask for storage space by media type (images, video, music, etc). This won't work for my purposes for the same reason IndexedDB won't work; it hides the actual location of the files from the user.

This is good for some scenarios but a programmer's text editor needs to share the files with other programs (git, vim).
Comment 44 Olli Pettay [:smaug] 2013-03-14 13:16:22 PDT
File API: Directories and System hides the actual location of the file too.
https://developer.chrome.com/beta/apps/fileSystem.html seems to be some Google only API.
Comment 45 Matthew Phillips 2013-03-15 06:42:38 PDT
Olli, yes I'm not suggesting you implement a Chrome API, just pointing out a use case that Mozilla is not covering. 

I think you could extend the normal File API and that would be a fairly elegant solution. Possibly add a requestWriteAccess function to either File and/or FileList that prompts the user. Just one idea off the top of my head.
Comment 46 Jonas Sicking (:sicking) No longer reading bugmail consistently 2013-03-16 05:18:39 PDT
You should be able to use the DeviceStorage API to do the same thing in Firefox OS. It's not yet implemented on other platforms but we are working on it.
Comment 47 Shachar 2013-04-07 11:16:59 PDT
Is there any verdict here? where does IDB falls short? is the polyfill by Eric has bad performance?
Comment 48 Jan Tošovský 2013-10-02 15:35:45 PDT
Reading these comments I suppose there are no plans to support in the near future something like this:
As a document writer I can select my XML file and upload this file together with all the parent folder content (containing resources - images or XIncluded XML parts).
I want to select not individual files, not parent folder, just a single file (main XML) and the browser would copy the content of the parent folder somewhere to the server. The original hierarchy would be preserved. Then I could invoke XSLT transformation on the server and generate various outputs.
Now I use Java client application that copies all the stuff into the shared folder on the server. Every app update needs to be applied on every user computer. Web app would be more handy here.
Comment 49 Florian Bender 2013-10-17 01:56:22 PDT
(In reply to Chris Double (:doublec) from comment #39)
> I've followed up with the Mega devs and will be getting more information on
> their API usage and needs.

What was the outcome? 


I guess this bug should be WONTFIXed in favor of http://w3c.github.io/filesystem-api/Overview.html
Comment 50 Florian Bender 2013-10-17 02:05:15 PDT
Bugzilla eating my ni?. :( 

Chris, please see comment 49.
Comment 51 cajbir (:cajbir) 2013-10-17 19:51:25 PDT
(In reply to Florian Bender from comment #50)
> Bugzilla eating my ni?. :( 
> 
> Chris, please see comment 49.

We discussed what could be done with existing HTML APIs and what APIs they needed to be on par with Chrome. The result was they were able to do what they wanted without needing this API and Firefox support improved: https://mega.co.nz/#blog_12
Comment 52 Florian Bender 2013-10-18 02:35:16 PDT
Thanks, that's good news! 

I think the correct bug to track is bug 910387, please mark this bug as WONTFIX or DUPE.

Note You need to log in before you can comment on or make changes to this bug.