1221562 - Investigate asynchronous DataTransfer API

Status: NEW

(Core :: DOM: Copy & Paste and Drag & Drop, defect, P5)

Description

[Neil Deakin]

We should investigate adding an asynchronous API to DataTransfer for drag and drop and clipboard operations. This has various advantages:

- works better on Linux where retrieving data is asynchronous and we currently block on the data.
- works better for multi-process use.
- would allow sources to generate the data lazily which is useful for large data or data that needs to be converted

Using promises, this might look like:

callback interface DataTransferDataProvider {
  Promise<DOMString> getData(DataTransfer dt, DOMString format);
};

interface DataTransfer {
 Promise<DOMString> promiseData(DOMString format);
 DOMString setData(DOMString format, DataTransferDataProvider provider);
};

And also incorporate into .items structure and a variation for mozGetDataAt.

Comment 1

[Hallvord R. M. Steen [:hallvors]]

Neil, this is a proposal: https://github.com/garykac/clipboard/blob/master/clipboard.md

When would be a good time to start coding something? :)

Comment 2

[Neil Deakin]

That's a fairly decent api for general clipboard reading and writing, but doesn't address the issue that I filed this bug on, namely being able to retrieve the individual data asynchronously, as for example, Windows and GTK actually does.

For example, when one opens a large image and then copies it to the clipboard and then pastes it into a textbox, we currently have the issue in e10s where large amounts of data are copied around several times between child and parent processes. (This sometimes crashes by running out of memory).

Ideally though no large memory copies would happen as one cannot paste an image into a textbox. The proposed api here doesn't address this as it assumes that after read() is called, the resulting DataTransfer will contain all of the data even if the image data won't end up being needed.

I proposed a variation of getData that would return a promise and return the data only when asked for.

The other part proposed above for setData perhaps doesn't apply as much to web sites, but is an important usecase for drag and drop of data to the desktop.

However, I do think the proposed api is a good solution for general cut/paste where an event doesn't suffice. I am still a bit concerned about the author of the proposal not seeing to understand why the cut/copy/paste events are needed.

Comment 3

[Hallvord R. M. Steen [:hallvors]]

I guess doubling down on the promising would be too much?

navigator.clipboard.read().then(function(data){
  // code iterating, looking at data.items[n].type
  // finds type we want at, say, n = 5
  data.items[n].getAsPromise().then(function(realdata){
    // yay, we have real data!
  });
});

So the proposal here fits better with the existing event-based model (IMHO) - but it would sure be nicer to just do navigator.clipboard.write(myDataTransferObject) than the current hackish add-copy-event-listener-and-do-execCommand trick. So the other proposal is sweet too.. If we're going to suggest changes we had better give feedback now.

Comment 4

[Neil Deakin]

So I thought about this some more and I definitely think the proposal (the Gary proposal) has some shortcomings that would be addressed by my proposal:

- it allows asynchronous use for both direct reading and modification of the clipboard using read/write and through the cut/copy/paste events, whereas the Gary proposal only handles the former.
- the proposal gives the example of sanitizing large image data before pasting it. However the Gary proposal only supports this for clipboard paste and not drag and drop of images. It would seem only logical that any filtering would need to happen in both cases.
- it handles the case I gave above in comment 2 where pasting an image into a textbox would not need to perform any large memory copies since a get type call would only be made for text data and a request for image data would never be made.

To reduce the need to use multiple promise oriented calls, I would suggest making clipboard.read() just return a DataTransfer directly instead of a promise. An implementation could then present a 'give permission to paste' UI to the user when calling dataTransfer.getAsPromise() and possibly cache such a decision within the DataTransfer.

Shorthand methods such clipboard.readText could exist that return promises.

If you wanted to enforce the use of asynchronous apis, you could also make clipboard.read return a base class of DataTransfer that only contained promise oriented methods, then make the clipboardevent's dataTransfer property a subclass of this that contained the existing methods.

I haven't though about how this might work with the .items oriented api so some adjustments may be needed. I think dataTransfer.items itself can remain synchronous though.

I can write this up more formally if desired.

We can also ask Ehsan if he has any thoughts since he has also looked at what people want to do with the clipboard.

Comment 5

[poiru]

See also: https://github.com/w3c/clipboard-apis/issues/41

Comment 6

[Jens Stutte [:jstutte]]

Bulk-downgrade of unassigned, >=3 years untouched DOM/Storage bug's priority.

If you have reason to believe this is wrong, please write a comment and ni :jstutte.