846931 - Improve <input type=file multiple> to allow picking a directory

Status: RESOLVED DUPLICATE of bug 1164310

(Core :: DOM: Core & HTML, defect)

Description

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

For a while we've been meaning to expand the functionality of
<input type=file multiple> to allow the user to pick a directory which would attach all files in that directory tree to the page.

We talked about this at the recent work week in London and came to the conclusion that we need multiple parts:

* Allow the UI to have 2 buttons, one for "select file" and one for "select
  directory". This is needed since windows and linux has completely different
  file-pickers for picking a file compared to picking a directory. So we can't
  have a single button which allows picking one or the other.
* Once one or more files are selected, the buttons are replaced with UI
  displaying an indication of how many files are selected, as well as an X to
  clear the control and go back to the two buttons. This is needed in order
  to easily select another file if you selected the wrong one.
* Display a dropdown (when the picker is hovered or focused) or some other type
  of UI which allows more finegrained control over adding and removing
  individual files. This dropdown can contain two buttons for adding more
  files or directories.
* Add the equivalent of a .click() function but which brings up the directory
  picker rather than the file picker. This is needed for cases when pages want
  to completely replace our UI with one of their own.

So the UI would look something like:

+--------------------------------+
|[Choose files][Choose directory]|
+--------------------------------+

Once something is picked

+--------------------------------+
|3 files attached             [x]|
+--------------------------------+

If the X is clicked the UI goes back to the initial one. If the control is hovered/clicked we display

+--------------------------------+
|3 files attached             [x]|
+--------------------------------+
 |file_name.html            [x]|
 |somePic.jpg               [x]|
 |vacation_pic.jpg          [x]|
 |                             |
 |[Add files][Add directory]   |
 +-----------------------------+


Once a directory is picked, we'd add all the files in the directory to the .files list. If the user picks a directory, the .name of the file still only contains the leaf name of the file, but we add a .longName which contains the partial path name under the directory that the user picked (including the name of the directory that the user picked).

Google was worried that this approach would mean that there might be an extended period of time between when the user picks a directory, and when we have traversed all the files under the directory so that we could expose them through the .files property.

We should test if this is a valid concern though before we worry about it.

But if it turns out to be an issue there are fairly easy solutions to it. We could fire an event to the page once we start scanning the directory. And then fire events for every N files scanned indicating which filename we are currently scanning. When the scanning is done we fire a "change" event as usual. That way the page can display whatever UI it wants.

We could also automatically display a built-in UI. But I suspect that's harder.


I think the best first step here is to get the most low-hanging fruit and implement the API for bringing up a directory picker. That way we can test the performance implications and websites that want to use it can do so while we are working on implementing the UI. This might be best done in a separate bug blocking this one.

Comment 1

[Jet Villegas (inactive)]

I'd love to see this replace the use cases that authors use Flash Player for:
http://help.adobe.com/en_US/FlashPlatform/reference/actionscript/3/flash/net/FileReference.html

Comment 2

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

Sooo... jwatt. Was this something you were interested in taking?

Comment 3

[Jonathan Watt [:jwatt]]

Yes, I'm definitely interested. I'll get back to you after I've had my 1:1 with Jet on Thursday.

Comment 4

[silverwind]

Just a quick info:

It seems that some people believe that

<input type="file" directory mozdirectory webkitdirectory>

works in Firefox, while I'm pretty sure only Chrome has implemented their prefixed webkitdirectory attribute so far. Whatever attribute you choose (prefixed or unprefixed), it might be wise to name it "directory".

Comment 5

[Jonathan Watt [:jwatt]]

I updated Jonas on IRC, but forgot to report back here; I plan on picking this up in July after I get back from vacation and finish of a couple of SVG bugs.

Comment 6

[Jonathan Watt [:jwatt]]

shorlander is going to help out with the UX.

shorlander: in addition to your thoughts on the UI and interaction described above, I'd like to know what I should do with the tooltip. Currently only files from the same directory can be picked so it just shows a list of file names. Once a directory can be picked and the files of all subdirectories are attached, should the file names in the tooltip have the file's path relative to the picked directory prepended? (Given that files with the same name may exist in various subdirectories (e.g. .DS_Store).)

Comment 7

[Jonathan Watt [:jwatt]]

Now that we seem to have a clear route ahead for the spec side of things firmed up I sent an email to the whatwg list to get feedback from others:

http://lists.whatwg.org/pipermail/whatwg-whatwg.org/2013-August/040340.html

Comment 8

[Stephen Horlander [:shorlander]]

Attachment: Multiple File Input Wireframe - i01

Attaching a wireframe based on Comment 0. 

Something I was wondering is if the distinction between Files and Directories is a hard technical limit? Ideally we wouldn't want to present the user with two buttons to choose multiple files. Anyway this wireframe is built on the assumption that they have to be separate interfaces.

Some notes to go with the wireframe:

• The UI and wording between single and multiple file uploads should be consistent. For example it would be nice to change the wording from "Browse…" to "Choose a file…" for the single case and "Choose files…" or "Choose a directory…" for the multiple case.

• Should visually distinguish the buttons with icons as well as text

• After choosing files the user is presented with some kind of progress indicator (bar or indeterminate)

• After the files are added and the user is presented with an expanded list of 4–5 visible results that can scroll to see the rest

• The list can be collapsed or expanded. We should keep this list inline and as a toggle and not in any kind of hover interface.

• Hovering the X should show a label that you will clear the file

I am not sure how much control we have over the filepicker dialog but it is worth considering an option to not add directories recursively.


> shorlander: in addition to your thoughts on the UI and interaction described
> above, I'd like to know what I should do with the tooltip. Currently only
> files from the same directory can be picked so it just shows a list of file
> names. Once a directory can be picked and the files of all subdirectories
> are attached, should the file names in the tooltip have the file's path
> relative to the picked directory prepended? (Given that files with the same
> name may exist in various subdirectories (e.g. .DS_Store).)

I would go with absolute paths and not relative paths in the tooltip since it's possible to add directories and files from different locations.

Comment 9

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

We can't really do a progress-bar since we don't know the total number of files to process. We could show total number of files processed as well as the name+path of the last processed file.

Comment 10

[silverwind]

One question that's been pondering me regarding the issue regarding this double-button dialog: Why not implement your own file picking dialog that can select either single files, multiple files or folders? Any reason to limit yourself to what the OS provides?

Comment 11

[Josh Aas]

Is this why Firefox is not supported for folder upload in Google Drive?

Comment 12

[silverwind]

(In reply to Josh Aas (Mozilla Corporation) from comment #11)
> Is this why Firefox is not supported for folder upload in Google Drive?

Most likely. I've investigated folder uploads quite a bit and there are currently two methods to get folders uploaded in Webkit Browsers:

- Through <input type="file" webkitdirectory>, which this bug is about.
- Through the Drag & Drop API, which bug 876480 is about.

Comment 13

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

Here's my latest thinking regarding this feature. Mostly this has evolved due to the fact that there are now drafts for a filesystem API which more browser vendors are expressing interest in.


For the general use-case of "upload a directory", here's what I think our options are:

* Use existing <input type=file multiple> API

We can relatively easily simply enable the user to select a directory,
in addition to selecting files, through the <input type=file multiple>
UI. This can even be done without changing any APIs.

The API returns a list of files, so if the user chooses a directory,
we can simply traverse that directory recursively and return the list
of files inside that directory. While we are traversing, we can show
some progress UI inside the <input type=file multiple> widget.

One piece of relevant complexity here is that Windows and Linux do not
have a native widget which allows the user to pick a file or a
directory. The only way to enable users to pick a directory on those
platforms is to bring up a widget specifically for picking a
directory. So at least on those platforms we would need to render two
buttons in the <input type=file multiple> widget. One for "pick file"
and one for "pick directory".

The problem is that many websites feel that the <input type=file
multiple> widget is really ugly. And so they hide it and render their
own UI. When the user clicks the webpage UI, the page calls
inputElement.click(). This brings up the normal file picker just as if
the user had clicked directly on the <input type=file multiple>
widget.

There are two problems with this. First of all, when
inputElement.click() is called on Windows or Linux, do we display the
file picker or the directory picker? Or do we display some UI which
allows the user to select if they want to select a file or a
directory? None of the options here are really good.

The other problem is that since the page is in charge of the UI, we
can't render any progress UI while we are traversing the directory.

We could make some minor additions to the API in order to solve these
problems. We could add a inputElement.clickDirectoryPicker() function
which enables the website to show separate "pick file" and "pick
directory" buttons, and then call inputElement.click() or
inputElement.clickDirectoryPicker() as appropriate.

We could also add events that fire when we start doing the directory
traversal, as we progress through the directory traversal, and as we
end directory traversal. This way the page could display their own UI.

However obviously no existing content is using any of these new APIs,
so we'd lose a lot of the benefit us using the existing <input
type=file multiple> widget.

* Improve <input type=file multiple> API

A lot of the complexity in the previous proposed solution comes from
the fact that <input type=file multiple> has an API which promises to
only return a list of files. If we instead extend <input type=file
multiple> to allow it to return directories as well, we could allow
the page to traverse the directory and display its own UI as needed.

The page would of course have to opt in to this since suddenly
exposing directory objects rather than lists of files would break
existing content. Hence we would need to add a new 'directory'
attribute, or some other form of opt-in, to the <input> element.

The problem with this is that it requires building an API which allows
a webpage to traverse a directory tree. I.e. we would need a
filesystem API. One is being developed at [1], but it's a fair amount
of work to implement.

But once we have that, this would be a pretty good solution. Basically
for a <input type=file multiple directory> we would render the same UI
as for a <input type=file multiple>, i.e. two separate buttons for
"pick file" and "pick directory". But if the user chose a directory,
we'd just add that directory to inputElement.files rather than expand
it into individual files.

We would still have to add a inputElement.clickDirectoryPicker()
function, though, in order to support webpages rendering their own UI
for bringing up file/directory pickers.

* Use drag'n'drop API

Currently we support dropping a file, or a list of files, onto a
webpage. But we don't support dropping a directory. This API is based
on exposing a list of files through the drag'n'drop events.

We could technically add support for dragging directories in a
backwards compatible way by traversing the directory contents and
exposing the list of files. However the UX would be pretty bad since
we don't have a good place to show any progress UI since drag'n'drop
UI is generally provided by the webpage.

Instead we could do the same thing as above and expose a directory
object to the webpage. Again this would however require implementing a
filesystem API which is a considerable amount of work.



Conclusions:

My recommended solution would be to go with a hybrid.

Let's for now add support for a .clickDirectoryPicker() function.
Additionally we should in the <input type=file multiple> widget
display both a "pick directory" button and a "pick file" button. If
the user picks a directory we would traverse the directory and display
progress UI inside the <input type=file multiple> widget. However for
now I wouldn't worry about exposing additional progress events to the
webpage.

I think some parts of this has already been implemented.

Once we get here, I think we can ship.

At some point in the future we should implement the filesystem API at
[1]. When that happens we could add support for a 'directory'
attribute on <input type=file multiple> as well as support for
drag'n'drop of directories.


[1] http://w3c.github.io/filesystem-api/Overview.html

Comment 14

[silverwind]

I like your suggestion of providing flat FileList from a picked directory, as in the end what most authors probably do with the files is FormData.append(File) to prepare for an XHR2 upload, and a flat list (which includes the relative path) is easiest to handle in this case.

Chrome implementation of <input type="file" directory> provides a FileList with a "webkitRelativePath" property on each file which provides path relative to the selected folder. This information can then be encoded into the FormData.

What I currently do for the directory/file picker problem is to have two buttons and a hidden <input type="file" multiple> where I dynamically add/remove a "webkitdirectory" attribute, depending on which button the user pressed. I'd suggest something like a "mozdirectory" attribute instead of the proposed "clickDirectoryPicker()", just for the sake of parity with Chrome.

For Drag and Drop folder uploads I'd also prefer a flat FileList. Chrome's implementation of folder drops doesn't provide a relative path like on the <input> element, so one needs to utilize their version of the FileSystem API to parse the dropped folder recursively, which requires a pretty arcane recursive function on the author's side.

Comment 15

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

Whatever we come up with with regards to a directory attribute vs. a clickDirectoryPicker() function it needs to be standardized in discussion with Chrome and other browsers.

However I personally prefer to not have a directory attribute which changes the meaning of the click() function since that doesn't work well with platforms that doesn't need separate widgets for directory picking and file picking. Neither OSX nor most mobile platforms needs this distinction.

Comment 16

[brandon]

> Google was worried that this approach would mean that there might be an extended period of time between when the user picks a directory, and when we have traversed all the files under the directory so that we could expose them through the .files property.
> 
> We should test if this is a valid concern though before we worry about it.

This is definitely a valid concern, at least for our use case. We use Chrome's Filesystem API to allow users to drag-and-drop one or more folders that may contain millions of files. We read it incrementally, uploading some files, ignoring others, and queuing up directories for recursive traversal.

We use this for uploading a very large number of documents to our server via WebSockets. I could just as easily imagine this being used for an image editing/sharing/storage service, a backup service, or other applications that may involve uploading a very large folder hierarchy.

When that folder is on a networked filesystem, visiting all of the files before starting is pretty useless, especially if it freezes up the browser for several minutes or even hours in the process.

Comment 17

[Florian Bender]

Should this bug be closed in favor of Bug 1164310?

Comment 18

[Andrew Overholt [:overholt]]

(In reply to Florian Bender from comment #17)
> Should this bug be closed in favor of Bug 1164310?

I would say yes.