Allow rendering shadow roots for <img> elements for text recognition
Categories
(Core :: DOM: Core & HTML, task, P2)
Tracking
()
Tracking | Status | |
---|---|---|
firefox100 | --- | fixed |
People
(Reporter: gregtatum, Assigned: emilio)
References
(Blocks 1 open bug)
Details
Attachments
(1 file, 1 obsolete file)
Similar to Bug 1431255, and to the video element, we're investigating adding a shadow root element to the <img> tag where we can add on OCR results, without opening it up to the content-accessible DOM.
For this root, we can then begin experimenting with attaching DOM elements, such as hidden spans for text selection within the image.
Per the spec, the img element doesn't allow shadow roots for user-facing code.
I'm not assigning this bug yet, but I'm adding a needinfo for :emilio to provide some initial direction.
Comment 1•3 years ago
|
||
Would you be doing that for the SVG image element too? It's pretty similar to HTML's img element.
Reporter | ||
Comment 2•3 years ago
|
||
Right now this is just exploration work, but I don't see why we wouldn't have it there as well. I doubt it will be in the MVP though.
Comment 3•3 years ago
|
||
When would the shadow DOM be created? Or rather, how often. I'm always just a bit worried about performance.
And what is the use case in mind when it comes to the selection handling? Currently selection can't cross shadow boundary.
Assignee | ||
Comment 4•3 years ago
|
||
This doesn't change behavior just yet, but seems worth doing separately.
With this patch, we can render content overlaid on top of an image
frame, by attaching a shadow root to it. The idea is to use this for
text recognition (OCR) of text inside images.
I chose to implement this using the DynamicLeaf stuff that I removed in
bug 1746310 (because nsMenuPopupFrame was its only user).
That seems simpler than either a new frame class, or more special cases
in the frame constructor, etc. But let me know if you disagree.
There are further changes that we want to do. For example, trying to
select the overlaid text with the mouse right now will start dragging
the image. For that, we might need to tweak our selection / mouse
dragging code. But those seem all changes that can go on top of this.
Updated•3 years ago
|
Assignee | ||
Comment 5•3 years ago
|
||
Depends on D140638
Assignee | ||
Comment 6•3 years ago
|
||
(In reply to Olli Pettay [:smaug] from comment #3)
When would the shadow DOM be created? Or rather, how often. I'm always just a bit worried about performance.
The idea is that it would be done once the image is sent for OCR recognition and the results come back. For now, that will be user-triggered (I think in Greg's prototype is via a context menu item). There should be no behavior change for regular images.
Also, it might not need to be JS code what drives these (in case that's your concern). We might want to create the shadow root in C++, it should be pretty straight-forward and it should be immutable once created.
If we were to do text recognition on images automatically, we'd probably want to use some heuristics like "is the image big enough", etc. And even so we'd only create the shadow root iff the image contains text.
And what is the use case in mind when it comes to the selection handling? Currently selection can't cross shadow boundary.
The idea is to allow copying / selecting text inside the image, but we might or might not want the contained text to be there when you select across the image. That's mostly TBD, as I understand it.
Comment 8•3 years ago
|
||
bugherder |
Assignee | ||
Comment 9•3 years ago
|
||
(Retitling to be precise)
Updated•3 years ago
|
Description
•