Open Bug 704128 Opened 13 years ago Updated 2 years ago

Maybe implement File API: Directories and System

Categories

(Core :: DOM: Core & HTML, defect, P5)

Product:
Core
Component:
DOM: Core & HTML
Type:
defect

Tracking

()

People

(Reporter: benfrancis, Unassigned)

References

Details

(Keywords: dev-doc-needed)

Attachments

(1 file)

Mega's a word on browsers
348.69 KB, image/png
Details
Mail sent to dev-webapi... While working on gallery & camera apps for B2G I've come across the use case where the user wants to copy photos to another device. Most digital cameras and camera phones currently follow the DCF standard for the storage of photographs, which specifies a particular directory structure, file names and file formats for storage in a filesystem. When the user wants to copy their photos to another device they can usually just plug it in to a USB host and mount the file system via USB mass storage in order to copy or move files using a desktop file manager. Currently the only option available in B2G for local storage of photographs is storing the base64 encoded in IndexedDB, although storing files in IndexedDB is coming. However, this does not make it possible to mount the files as a filesystem via USB so does not satisfy this use case in the usual way. The W3C's File API: Writer specification is currently being implemented, but the use case described here is outside the scope of this spec. The W3C has a working draft of another specification called File API: Directories and System, which would enable use cases more similar to the one given here. Should File API: Directories and System be implemented, and implemented in such a way that files written via this API could be mounted as a filesystem over USB? Alternatively, should B2G provide some other mechanism for moving media to another device in a more web-like fashion over an IP network rather than USB?
Last time I looked at File API:D&S it wasn't a very good API, but I didn't care too much about it since it wasn't clear at all whether we would ever implement it. If we now decide to implement it, better to make sure the API is reasonable.
I really don't want to implement D&S, and we should explore alternative solutions for B2G.
Status: UNCONFIRMED → NEW
Ever confirmed: true
Summary: Implement File API: Directories and System → Maybe implement File API: Directories and System
Jonas, This is the thread discussing a potential "Gallery API" we talked about in the B2G meeting https://groups.google.com/d/topic/mozilla.dev.webapi/ODGDpW8Rs6M/discussion but I'm still not sure that's the right direction, it might be better to have something lower level for general file access.
If we don't want to use something like the filesystem (that could be a good idea, to get rid of this old way of managing files), we should investigate a way to let two devices share files. For example give the possibility to (in some way) "mount" IndexedDB as a filesystem? I think we shouldn't create an API for each single application, but something more generally usable.
We don't have all the generic platform pieces in place yet, but here's a way this could work in the future - a USB manager application registers itself as a mass storage device using WebUSB - the USB manager uses intents to query other applications and build a list of "virtual files" to offer - the host requests transfer of files, which the USB manager satisfies through by streaming them over postMessage channels from the hosting application We don't have all the technology for that yet. In the meantime, we should ask ourselves whether it's worth implementing an API of unclear utility, vs. relying on cloud photo managers like flickr as stopgaps. (Or adding in other temporary hacks that are not intended to be on a standards track.)
Although slightly complicated, that does sound like a good idea to me. Better than trying to standardise something very specialised like an API for browsing photos on a device or enforcing the concept of file systems on the web. In the meantime I think a gallery app is still useful as long as you can capture an image using the camera and share that image with another app using web intents. A music player is less useful, unless you can download music from somewhere, but could be used via intents to play audio from another app.
(In reply to Ben Francis from comment #6) > In the meantime I think a gallery app is still useful as long as you can > capture an image using the camera and share that image with another app > using web intents. > Of course! The hypothetical USB "fuse" above would talk to a gallery app with intents. And also flickr, etc. > A music player is less useful, unless you can download music from somewhere, > but could be used via intents to play audio from another app. I don't follow the logic here.
(In reply to Chris Jones [:cjones] [:warhammer] from comment #5) > We don't have all the generic platform pieces in place yet, but here's a way > this could work in the future > > - a USB manager application registers itself as a mass storage device using > WebUSB ... or as an MTP-capable device. MTP is the protocol that replaces mass storage capabilities on Honeycomb and later. The gallery, music player, etc. apps could interface with the MTP layer using intents, indeed.
Yes, without knowing that the USB broker is speaking MTP instead of usb-fuse.
The WD was updated recently: http://www.w3.org/TR/file-system-api/ Just because this isn't the right solution for B2G file sharing, doesn't mean it isn't useful or important. Being able to work in a sandboxed fs is important for the future of the Web Platform. Please, implement this.
The API is reasonable bad, so I would prefer implementing something better.
I'm developing applications against it right now in Chrome. I agree, it could definitely be better. However: "It is intended to be minimal in extent, but sufficiently powerful that easy-to-use libraries may be built on top of it." So far it is indeed "sufficiently powerful" and I am indeed building easy to use libraries on top of it. This functionality was needed last year.
What use case isn't covered by support for files in IndexedDB? A word of caution: at the last TPAC meeting none of the other browser vendors had plans to implement File Systems and Directories.
Thanks for the heads up regarding vendor plans. Is there a doc describing files support for IndexedDB would look like? The application is encrypting and storing (potentially large) files that are chosen from outside the sandbox into the sandbox. Then being able to upload the sandboxed encrypted file via a XHR. So if I can read/write blobs from indexedDB *quickly* that should be enough.
> Is there a doc describing files support for IndexedDB would look like? The IndexedDB specification? > So if I can read/write blobs from indexedDB *quickly* that should be enough. Depending on what you mean by "quickly", you already can.
>The IndexedDB specification? I did check there. It's not obvious that 'any' value means a Blob will be stored as expected. In fact, it doesn't work in Chrome, though it's been acknowledge as bug and assigned. The Mozilla hacks blob was informative: http://hacks.mozilla.org/2012/02/storing-images-and-files-in-indexeddb/ >Depending on what you mean by "quickly", you already can. I found the performance of Chrome's implementation to be surprisingly fast. I'll measure it as now I'm curious.
(In reply to Geoff Flarity from comment #16) > I found the performance of Chrome's implementation to be surprisingly fast. > I'll measure it as now I'm curious. There's no real Chrome's implementation for storing blobs at the moment, they just store blob urls at the moment.
>There's no real Chrome's implementation for storing blobs at the moment, they just store blob urls at the moment. That may very well be, but I'm able to repeatedly save a 350MB file in about 12 seconds. I can't get a 480MB file to work without an 'ah snap' though. Assuming IndexedDB blob storage performs comparably (or better!), it should work fine for me. Sadly, I won't be able to support FireFox until the FileSaver is implemented, but this is getting off topic enough already as it is. Thanks for all the quick responses and helpful feedback.
Why dev-doc-needed?
Olly: to track it. If the bug is R/Fixed in the future, we won't miss it. If it is R/Invalid or other, we will just remove the d-d-n at that point. D-d-N really means documentation will be need when/if fixed.
Just FWIW: the lawnchair adapter supports both Firefox and Webkit IndexedDB, and the webkit implementation is noticibly faster on the testsuite. 10.637 seconds on Firefox, 2.963 seconds on webkit. That probably just means that IndexedDB should be improved, not that the Filesystem API should be implemented, though.
It seems that "Mega", the new incarnation of the hugely popular "Megaupload", will be using the file-system-api. According to an Ars Technica article the site, which goes live in a day or so, is recommending Firefox uses use Chrome: http://arstechnica.com/business/2013/01/mega-arrives-ars-goes-hands-on-with-kim-dotcoms-cloud-storage-site: "Once you create an account, the first thing the site does is generate that crypto key for you. As a Firefox user, Mega also warned me to switch to Google Chrome, as not doing so would “adversely affect [my] file transfer performance." A browser switch later, and I was faced with a blank “file manager,” which was practically begging me to upload files." I asked for more information from the developers of Mega and got referred to the file-system-api spec: https://twitter.com/bramosnl/status/292467486664839168 I think it would be good to implement this spec given that it's getting usage.
http://ericbidelman.tumblr.com/post/21649963613/idb-filesystem-js-bringing-the-html5-filesystem-api I still think we should WONTFIX this. There was some discussion of combining this spec and B2G's device storage and cutting down the complexity, but we can do that in another bug if/when we decide to.
I wonder for what is Mega using Filesystem API. IndexDB should probably work just fine for them, unless our IDB implementation is too slow.
Meanwhile, back on earth... If Mozilla's indexDB implementation is more than capable of handling gigabyte files efficiently, then I wish someone would just provide a mock around it that provides this interface FileSystem interface. Maybe run some tests that demonstrate it provides a comparable performance result...
(In reply to Olli Pettay [:smaug] from comment #24) > I wonder for what is Mega using Filesystem API. Presumably, saving actual files to the user's actual disk? https://twitter.com/doublec/status/292466262049685504 and comments. Gerv
At this point this probably doesn't add anything you don't already know but this is what they internally say (since it's overwhelmed by traffic at the moment). If I'm understanding this correctly other use cases could include for example a webcam capture app (or any app fetching a lot of data) that wants to sequentially write files that will eventually become too large to hold on RAM. They also complain about being unable to fetch data over HTTP from HTTPS (a warning on Chrome and Firefox <= 17, an error since Firefox 18), since HTTPS is not needed for their data transfers.
Firefox provides ways to store blobs outside of memory (put the blobs in IndexedDB). AIUI they want to write to a specific file on the disk.
But doesn't file system API use files in a sandbox. That at least used to be the case. Based on attachment 704284 [details] they just want to store the data somewhere. With IE10 they talk about disk-based Blobs, and we do have them. And we have FileHandle for writing.
Ah, ok, it is not the kind of sandbox I expected. The files are there.
...but doesn't save the file using its name.
Blocks: 832732
And IndexedDB stores files in local system too, at least on linux.
"Carefully avoids providing any API that would allow writing files from JavaScript" - this is completely misleading. Firefox supports storing blobs/files in IndexedDB and they are stored as real files (on all platforms) on disk. AFAIK, chrome doesn't support it yet and IE10 stores them in a database. There's also support for (partial) writing to existing stored files using FileHandle API. So we actually have quite solid support for "writing files from JavaScript". I know, the current situation across browsers is not optimal, but stating that Firefox avoids to support it on purpose is a bit unprofessional.
I've seen Google demo a file system API polyfill that they themselves wrote: https://github.com/ebidel/idb.filesystem.js
Who's responsible for following this up with Mega?
http://lists.w3.org/Archives/Public/public-webapps/2012OctDec/0491.html is good ammunition for why the FileSaver spec shouldn't be implemented or used in its current form.
But, do we have an API that lets you upload entire directory trees? Or is that not covered by any spec yet?
(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #36) > http://lists.w3.org/Archives/Public/public-webapps/2012OctDec/0491.html is > good ammunition for why the FileSaver spec shouldn't be implemented or used > in its current form. FileSaver is something different from FileSystems. (In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #37) > But, do we have an API that lets you upload entire directory trees? Or is > that not covered by any spec yet? No but Jonas has a plan for that.
(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #35) > Who's responsible for following this up with Mega? I've followed up with the Mega devs and will be getting more information on their API usage and needs.
I'm a developer working on a programmer text editor that I would like to deploy as a packaged app. The user of a text editor expects to be able to open and save files in a directory of their choosing, not import files into a sandboxed directory that they need to export again to get back out. Chrome provides this functionality currently: https://developer.chrome.com/beta/apps/fileSystem.html Firefox's packaged apps a bit "nicer" in my opinion, but I can't use it without allowing the user access to the file system.
Matthew, do you know about the Device Storage API https://wiki.mozilla.org/WebAPI/DeviceStorageAPI ? Perhaps it could be used or extended for your use cases? The Device Storage API was intentionally designed without exposing the complex hierarchical structure of the file system to the web, but allows for the storage of files in a shared location. There's also the option of storing the files as blobs in IndexedDB inside your app.
Ben, my understanding of the Device Storage API (from reading the mailing list several months back, the documentation on MDN is lacking currently) was that it was a way to ask for storage space by media type (images, video, music, etc). This won't work for my purposes for the same reason IndexedDB won't work; it hides the actual location of the files from the user. This is good for some scenarios but a programmer's text editor needs to share the files with other programs (git, vim).
File API: Directories and System hides the actual location of the file too. https://developer.chrome.com/beta/apps/fileSystem.html seems to be some Google only API.
Olli, yes I'm not suggesting you implement a Chrome API, just pointing out a use case that Mozilla is not covering. I think you could extend the normal File API and that would be a fairly elegant solution. Possibly add a requestWriteAccess function to either File and/or FileList that prompts the user. Just one idea off the top of my head.
You should be able to use the DeviceStorage API to do the same thing in Firefox OS. It's not yet implemented on other platforms but we are working on it.
Is there any verdict here? where does IDB falls short? is the polyfill by Eric has bad performance?
Reading these comments I suppose there are no plans to support in the near future something like this: As a document writer I can select my XML file and upload this file together with all the parent folder content (containing resources - images or XIncluded XML parts). I want to select not individual files, not parent folder, just a single file (main XML) and the browser would copy the content of the parent folder somewhere to the server. The original hierarchy would be preserved. Then I could invoke XSLT transformation on the server and generate various outputs. Now I use Java client application that copies all the stuff into the shared folder on the server. Every app update needs to be applied on every user computer. Web app would be more handy here.
(In reply to Chris Double (:doublec) from comment #39) > I've followed up with the Mega devs and will be getting more information on > their API usage and needs. What was the outcome? I guess this bug should be WONTFIXed in favor of http://w3c.github.io/filesystem-api/Overview.html
Bugzilla eating my ni?. :( Chris, please see comment 49.
Flags: needinfo?(chris.double)
(In reply to Florian Bender from comment #50) > Bugzilla eating my ni?. :( > > Chris, please see comment 49. We discussed what could be done with existing HTML APIs and what APIs they needed to be on par with Chrome. The result was they were able to do what they wanted without needing this API and Firefox support improved: https://mega.co.nz/#blog_12
Flags: needinfo?(chris.double)
Thanks, that's good news! I think the correct bug to track is bug 910387, please mark this bug as WONTFIX or DUPE.
https://bugzilla.mozilla.org/show_bug.cgi?id=1472046 Move all DOM bugs that haven’t been updated in more than 3 years and has no one currently assigned to P5. If you have questions, please contact :mdaly.
Priority: -- → P5
Component: DOM → DOM: Core & HTML
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: