1759204 - Have <img> shadow root ready for span elements for text recognition

Status: RESOLVED FIXED

(Core :: DOM: Core & HTML, task, P2)

Description

[Greg Tatum [:gregtatum]]

As a follow-up to Bug 1759201, we would like to begin experimenting with adding span elements to the shadow root of the <img> element. There are potential issues with this around text selection and img dragging.

It would be nice to begin experimenting with this on a real img element.

Feel free to add more details and clarifications here Emilio.

Comment 1

[Emilio Cobos Álvarez (:emilio)]

Attachment: Bug 1759204 - Introduce ShmemImage to easily pass images via IPC. r=aosmond

This shares code with the transferable / cursor stuff which are the
other two non-rendering callers that deal with images over IPC.

We're going to introduce a consumer of this new type in a second, but
this was worth getting reviewed separately.

Comment 2

[Emilio Cobos Álvarez (:emilio)]

Attachment: Bug 1759204 - Minor drive-by nsImageLoadingContent cleanups. r=edgar,aosmond

There's no need to QI etc to go from nsImageLoadingContent to
nsINode/nsIContent.

Depends on D141110

Comment 3

[Emilio Cobos Álvarez (:emilio)]

Attachment: Bug 1759204 - Make text recognition interface be just C++, and deal with images and processes. r=dminor,aosmond

This is all still unused, but we'll add an JS API to HTMLImageElement
for testing in a bit.

The reason for not having the JS-exposed interface is that dealing with
all the relevant edge cases is pretty hard from JS (think of e.g., the
image source changing mid-recognition, and so on).

The frontend will probably want an "is image recognition available"
boolean, but that doesn't need to be on its own XPCOM component, IMO.

If we need a JS API for unit-testing or something we can always add it
later.

Depends on D141111

Comment 4

[Emilio Cobos Álvarez (:emilio)]

Attachment: Bug 1759204 - Add an HTMLImageElement.recognizeCurrentImageText ChromeOnly API to test text recognition and create the shadow root. r=dminor,aosmond

We could expose the text recognition values to JS if need be, but
there's no reason to do that for now. We expose them via the UA
widget DOM for now.

We can trivially do that if we want to build the Shadow DOM in JS
or what not. I decided to still return a promise so that we can
expose errors though.

Depends on D141112

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a4e1475f2a32
Introduce ShmemImage to easily pass images via IPC. r=aosmond
https://hg.mozilla.org/integration/autoland/rev/ba558395ee2f
Minor drive-by nsImageLoadingContent cleanups. r=aosmond
https://hg.mozilla.org/integration/autoland/rev/0a65a83cf944
Make text recognition interface be just C++, and deal with images and processes. r=aosmond
https://hg.mozilla.org/integration/autoland/rev/466314d367bb
Add an HTMLImageElement.recognizeCurrentImageText ChromeOnly API to test text recognition and create the shadow root. r=aosmond

Comment 6

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/a4e1475f2a32
https://hg.mozilla.org/mozilla-central/rev/ba558395ee2f
https://hg.mozilla.org/mozilla-central/rev/0a65a83cf944
https://hg.mozilla.org/mozilla-central/rev/466314d367bb