Open Bug 1515879 Opened 7 years ago Updated 1 month ago

HTML5 drag and drop doesn't work

Categories

(Remote Protocol :: Marionette, enhancement, P2)

Product:
Remote Protocol
Component:
Marionette
Type:
enhancement

Tracking

(Not tracked)

People

(Reporter: whimboo, Unassigned)

References

(Blocks 2 open bugs,
URL
)

Details

(Whiteboard: [webdriver:backlog])

Originally reported as: https://github.com/mozilla/geckodriver/issues/1450 On the following page the drag and drop action doesn't work: https://mdn.github.io/dom-examples/drag-and-drop/copy-move-DataTransfer.html I can see that with the code snippet we correctly position the mouse over the destination `div`, but no element is dragged, and that even with a `pointer_down` action performed.

Maybe this happens because no drag and drop specific events like dragstart are synthesized? I'm not familiar with D&D so if you have some feedback James, I would appreciate. The spec also doesn't mention any of those events, so I assume it's an implementation detail?

Flags: needinfo?(james)

I assumed this would happen as a side effect of the other events, but perhaps not. It's definitely intended per-spec that this kind of event happens. I notice that EventUtils has a specific way of creating drag events [1]; I don't know how exactly we should know when to invoke that. Maybe smaug has some insight into how this stuff is implemented in gecko? In particular what's needed to ensure that these events are emitted at the same time they would be for real pointer inputs.

[1] https://searchfox.org/mozilla-central/source/testing/mochitest/tests/SimpleTest/EventUtils.js#2587

Flags: needinfo?(james) → needinfo?(bugs)

So what does Marionette do when drag operation is supposed to start?
Does it do something like mousedown, then some moves ?
https://searchfox.org/mozilla-central/rev/014fe72eaba26dcf6082fb9bbaf208f97a38594e/dom/events/EventStateManager.cpp#1900-1919

Flags: needinfo?(bugs)

All the event related code can be found in https://searchfox.org/mozilla-central/source/testing/marionette/event.js.

Most of the methods are copied from EventUtils.js, but the last sync happened a long time ago. Maybe meanwhile new code has been added to (better) support D&D?

Anyway, lets see how it should work... The entry point for D&D activity via Marionette are clearly the actions, which are a sequence of mouse and keyboard events to synthesize. The code is here:

https://searchfox.org/mozilla-central/source/testing/marionette/action.js

The important methods here are:

Each of the methods call into event.synthesizeMouseAtPoint().

Here how the equivalent method in EventUtils.js looks like:
https://searchfox.org/mozilla-central/rev/014fe72eaba26dcf6082fb9bbaf208f97a38594e/testing/mochitest/tests/SimpleTest/EventUtils.js#552

Olli, if you have an idea what's missing here I would appreciate. Otherwise what's the best way to investigate these missing DOM events? Is there a log option to turn DOM logging on?

Flags: needinfo?(bugs)

I wonder if that might be related to what Andrew just found out on bug 1686361.

But which events exactly? And where? Does that pointermove end up triggering drag threshold?
The events can be logged simply by adding event listeners for those.

Flags: needinfo?(bugs)
Severity: normal → S3

From the Selenium user perspective, this bug is one of the most filed issues, especially since it doesn't work on any browser right now.

Being able to point to a fix or an explanation of why this isn't working would be greatly appreciated. Thanks.

Product: Testing → Remote Protocol
Whiteboard: [webdriver:backlog]

We should check again with the fixes as landed on bug 1838221 for the drag and drop test.

Depends on: 1838221

So I had a look with the updated tests and we are still failing. Reason is that we do not really send drag and drop events. Instead we assume that a "pointerMove" after a "pointerDown" is still just a mouse event. Instead we should keep the pressed state of the pointer input device and emit drag events in case a movement happens and the pointer is down.

Maybe it's something we can have a look at during our M9 milestone.

There are currently issues with DnD on the platform side and with the ongoing work on bug 1844723 those should be fixed and we hopefully can immediately support it. So lets wait until it got landed.

Depends on: 1844723

The remaining issues are:

  • When mousemove is synthesized, dragover should be dispatched instead
  • When mouseup is synthesized, drop should be dispatched (then, I guess dragend will be fired by nsIDragService.endDragSession automatically

The main issue to implement that is, how to treat the dataTransfer and new drop effect.

I think that for emulating everything, we should handle it in the C++ world, i.e., I think we should add nsIDOMWindowUtils.sendDragOverEvent and nsIDOMWindowUtils.sendDropEvent APIs.

Masayuki, would these additional drag events be dispatched by the browser when we would dispatch async events via the parent process? Or is it really something EventUtils definitely would have to do?

Flags: needinfo?(masayuki)

Although I don't have concrete idea. Ideally, they should be synthesized by browser automatically if there is a drag session because the dispatcher needs to adjust drag session state (dataTransfer management, dropEffect, etc) if we make EventUtils.js do it.

Flags: needinfo?(masayuki)

The issue can also be observed with BiDi by looking at the events reported by this test:

  • with a duration < 23 there's only a mousemove and a dragstart event
  • with larger durations, there are also dragenter, dragover and dragleave events (this has also been reported as a workaround here)

I also tried with async mouse events and in that case only mousemove and dragstart are dispatched even with large durations.

This issue also causes Playwright's drag-and-drop tests to fail with BiDi because it doesn't set a duration for the pointerMove action.

You need to log in before you can comment on or make changes to this bug.