1513658 - Implement DocumentOrShadowRoot.node(s)FromPoint.

Status: RESOLVED FIXED

(Core :: DOM: CSS Object Model, enhancement)

Description

[Yura Zenevich [:yzen] (please use "needinfo")]

Usecase:

Developer tools (specifically accessibility panel) has a highlighter that calculates colour contrast for text nodes. Right now it is fairly cumbersome as it has to use accessibility API to get an accessible object to more reliably get the text node and calculate its bounds. The tool also does not work with elements that have pointer-events CSS property set to none.

After talking to Markus, he suggested opening a bug for adding a new API similar to DocumentOrShadowRoot.elementsFromPoint but instead it would return a list of nodes e.g. nodesFromPoint. Additionally it would also allow for ignoring and including nodes that have pointer-events set to none.

This will help with removing a number of workarounds that are currently in place for the colour contrast analyzer functionality to work.

Comment 1

[Emilio Cobos Álvarez (:emilio)]

See also https://github.com/w3c/csswg-drafts/issues/556#issuecomment-433411568.

Comment 2

[Emilio Cobos Álvarez (:emilio)]

This should be very trivial to implement, I'll take a look when I have the time, though feel free to steal.

Comment 3

[Markus Stange [:mstange]]

Any ideas about how to expose the "include nodes even if they have pointer-events:none" functionality on an API level? It sounds like this is an important piece for devtools.

Comment 4

[Emilio Cobos Álvarez (:emilio)]

(In reply to Markus Stange [:mstange] from comment #3)
> Any ideas about how to expose the "include nodes even if they have
> pointer-events:none" functionality on an API level? It sounds like this is
> an important piece for devtools.

We could add a separate [ChromeOnly] method or something, right? We could add a fancier API, but it wouldn't be exposed to the web so I'd rather keep it simple.

Comment 5

[Markus Stange [:mstange]]

Sounds good!

Comment 6

[Emilio Cobos Álvarez (:emilio)]

Why doesn't DOMWindowUtils.nodesFromRect suffice here?

https://searchfox.org/mozilla-central/rev/3160ddc1f0ab55d230c595366662c62950e5c785/dom/interfaces/base/nsIDOMWindowUtils.idl#756

Comment 7

[Emilio Cobos Álvarez (:emilio)]

First WIP is at https://treeherder.mozilla.org/#/jobs?repo=try&revision=a5f144bad24de4c9a3e58cf9307f56ff5a62c4cf.

Though if somebody volunteered to write / adapt the elementsFromPoint tests it'd be amazing. Most of it is cleanup, so I'll try to land that separately.

Comment 8

[Emilio Cobos Álvarez (:emilio)]

Attachment: Bug 1513658 - Implement DocumentOrShadowRoot.node(s)FromPoint. r=smaug,mats

ChromeOnly for now, needs tests that I'll make sure to land with.

Comment 9

[Yura Zenevich [:yzen] (please use "needinfo")]

(In reply to Emilio Cobos Álvarez (:emilio) from comment #6)
> Why doesn't DOMWindowUtils.nodesFromRect suffice here?
> 
> https://searchfox.org/mozilla-central/rev/
> 3160ddc1f0ab55d230c595366662c62950e5c785/dom/interfaces/base/
> nsIDOMWindowUtils.idl#756

I was not aware about this API, it actually might work (at least when I tested locally), granted it can support ignoring pointer-events none.

Comment 10

[Yura Zenevich [:yzen] (please use "needinfo")]

(In reply to Yura Zenevich [:yzen] from comment #9)
> (In reply to Emilio Cobos Álvarez (:emilio) from comment #6)
> > Why doesn't DOMWindowUtils.nodesFromRect suffice here?
> > 
> > https://searchfox.org/mozilla-central/rev/
> > 3160ddc1f0ab55d230c595366662c62950e5c785/dom/interfaces/base/
> > nsIDOMWindowUtils.idl#756
> 
> I was not aware about this API, it actually might work (at least when I
> tested locally), granted it can support ignoring pointer-events none.

After doing more testing , there are some funny instances where I would get a list back with nodesFromRect that looks like this:

[nodeC, nodeA, nodeB, nodeC, ...]

STR: about:sessionrestore "View Previous Tabs" button has that.

Comment 11

[Yura Zenevich [:yzen] (please use "needinfo")]

From what I can tell, get a list of nodes that includes an iframe is a limitation too, accessibility API currently allows for stepping into iframes (It will likely change with fission however). As it is right now though, nodesFromRect do not do that.. Any suggestions on how to go about that ?

Comment 12

[Emilio Cobos Álvarez (:emilio)]

Sorry for the lag here, I somehow missed those comments.

(In reply to Yura Zenevich [:yzen] from comment #10)
> After doing more testing , there are some funny instances where I would get
> a list back with nodesFromRect that looks like this:
> 
> [nodeC, nodeA, nodeB, nodeC, ...]
> 
> STR: about:sessionrestore "View Previous Tabs" button has that.

That also happens with document.elementsFromPoint. I took a look and I think that happens because there's a ::after pseudo-element overlaying the whole thing, so the order hit-testing sees this is:

  [button::after, span, button, ...]

Which means that we return the button twice. For a chromeonly API we could expose the pseudo-element, though that's a bit dangerous. I'm not sure we can do much better there, thoughts?

(In reply to Yura Zenevich [:yzen] from comment #11)
> From what I can tell, get a list of nodes that includes an iframe is a
> limitation too, accessibility API currently allows for stepping into iframes
> (It will likely change with fission however). As it is right now though,
> nodesFromRect do not do that.. Any suggestions on how to go about that ?

You could just recursively call into the iframe's contentWindow, right?

Ok, so for now I'll land this API, which should eventually be exposed to content, and file a new bug where we can tailor it for accessibility's use-case, since it seems a bit more complicated than just ignoring pointer-events.

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/50303cb35e84
Implement DocumentOrShadowRoot.node(s)FromPoint. r=smaug

Comment 14

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/50303cb35e84

Comment 15

[Yura Zenevich [:yzen] (please use "needinfo")]

(In reply to Emilio Cobos Álvarez (:emilio) from comment #12)
> Sorry for the lag here, I somehow missed those comments.
> 
> (In reply to Yura Zenevich [:yzen] from comment #10)
> > After doing more testing , there are some funny instances where I would get
> > a list back with nodesFromRect that looks like this:
> > 
> > [nodeC, nodeA, nodeB, nodeC, ...]
> > 
> > STR: about:sessionrestore "View Previous Tabs" button has that.
> 
> That also happens with document.elementsFromPoint. I took a look and I think
> that happens because there's a ::after pseudo-element overlaying the whole
> thing, so the order hit-testing sees this is:
> 
>   [button::after, span, button, ...]
> 
> Which means that we return the button twice. For a chromeonly API we could
> expose the pseudo-element, though that's a bit dangerous. I'm not sure we
> can do much better there, thoughts?

I think this is unnecessary , I can probably just check and ignore repeating nodes, assuming the reverse order (from the top)  is correct.

> 
> (In reply to Yura Zenevich [:yzen] from comment #11)
> > From what I can tell, get a list of nodes that includes an iframe is a
> > limitation too, accessibility API currently allows for stepping into iframes
> > (It will likely change with fission however). As it is right now though,
> > nodesFromRect do not do that.. Any suggestions on how to go about that ?
> 
> You could just recursively call into the iframe's contentWindow, right?
> 
> Ok, so for now I'll land this API, which should eventually be exposed to
> content, and file a new bug where we can tailor it for accessibility's
> use-case, since it seems a bit more complicated than just ignoring
> pointer-events.

Thanks

Comment 16

[Chris Mills [:cmills]]

Note for MDN team — I've aded a note about these additions to the 66 rel notes: https://developer.mozilla.org/en-US/docs/Mozilla/Firefox/Releases/66#APIs

In terms of doing the actual docs work, we'll need to add pages for these two new methods, link them from appropriate places, and add BCD.

Comment 17

[Chris Mills [:cmills]]

I've documented these new methods:
https://developer.mozilla.org/en-US/docs/Web/API/DocumentOrShadowRoot
https://developer.mozilla.org/en-US/docs/Web/API/DocumentOrShadowRoot/nodeFromPoint
https://developer.mozilla.org/en-US/docs/Web/API/DocumentOrShadowRoot/nodesFromPoint

Added an entry to our compat data, so people know what the situation is:
https://github.com/mdn/browser-compat-data/pull/3580

And removed the Fx66 rel notes entry, as it is non-standard and chrome-only, so we don't need to annouce it there.

Let me know if that is all OK. Thanks!