Fission: Implement frontend API(s) for taking a screenshot of OOP iframes
Categories
(Core :: Graphics, enhancement, P3)
Tracking
()
Fission Milestone | M4 |
People
(Reporter: rhunt, Assigned: mattwoodrow)
References
(Depends on 1 open bug, Blocks 2 open bugs)
Details
(Whiteboard: [gfx-noted], m4-browsing)
Reporter | ||
Comment 1•7 years ago
|
||
Comment 2•7 years ago
|
||
Comment 3•7 years ago
|
||
Comment 4•7 years ago
|
||
Reporter | ||
Comment 5•7 years ago
|
||
Comment 6•7 years ago
|
||
Comment 7•7 years ago
|
||
Hi Ryan,
I just saw that you worked on this concurrently to this bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1475139
To answer your question - the devtools panel runs in the parent process.
This api that you worked on looks really promising: would we be able to use it?
Reporter | ||
Comment 8•7 years ago
|
||
Hi Yulia, I haven't gotten any time to work on it since this proposal. It should be fairly simple to implement though.
I'll try and take a look at it in the next week.
Reporter | ||
Updated•6 years ago
|
Reporter | ||
Updated•6 years ago
|
Reporter | ||
Updated•6 years ago
|
Reporter | ||
Updated•6 years ago
|
Updated•6 years ago
|
Updated•6 years ago
|
![]() |
||
Updated•6 years ago
|
Assignee | ||
Comment 10•6 years ago
|
||
Ryan and I chatted about this today, and we're concerned that it will be hard for devtools to come up with a unique selector for an arbitrary DOM element that you want to snapshot.
We discussed the possibility of chaining nth-child selectors to identify an element w.r.t the nearest uniquely identifiable ancestor, but it seems like that could change in undesired ways between when it's computed in devtools code, to when it's asynchronously processed after a roundtrip through the parent process.
Are there other ways to identify an element that devtools could pass to the parent process, and then the snapshot API can pass down to the child?
We discussed another alternative, where there would be a captureSnapshot API in the content process that returns an opaque identifier, and then a separate API in the parent which retrieves the ImageBitmap result for that identifier asynchronously. As mentioned earlier, we can't return the ImageBitmap to the content process, since it may include pixels from a different origin.
Yulia, do any of those sound workable from a devtools perspective?
Comment 11•6 years ago
|
||
Are there other ways to identify an element that devtools could pass to the parent process, and then the snapshot API can pass down to the child?
I think I am out of my depth to answer that. Patrick, I think you have more knowledge about how the inspector handles selections and what we could do. Could you chime in?
Comment 12•6 years ago
|
||
Right now, DevTools captures screenshots of DOM nodes in the content process where those DOM nodes live using drawWindow
.
When you right-click on a node in the inspector and select the "screenshot node" option, then that sends a request to the DevTools server running in the content process (which has access to chrome privileges) with some DevTools unique ID for this node.
Upon receiving that request, the DevTools server part locates the DOM node, grabs the screenshot with drawWindow
and then sends back a data-uri of it to the DevTools front-end so it can take it from here and save the file for the user.
With Fission, DevTools will have a server running in the content page process, as well as one server per remote iframe.
So, if DevTools sends the request to the right server, in the right process (the one where the DOM node lives), then I assume it can still continue using drawWindow
. Except the content of nested remote iframes would appear blank.
So, if we could replace drawWindow
with the proposed captureSelector
API and still pass it a DOM node reference, then I guess it would work fine for DevTools.
We could however change this and always call captureSelector
from the browser parent process instead, but in that case we'd have to pass a unique selector that identifies the element.
We do this in a number of places already in DevTools. So yes, we could chain these selectors as required to traverse all iframes (i.e. an array of selectors where each item targets a DOM node in a given document).
But you are right in saying that the time between calculating this selector and taking the screenshot will be enough for the DOM to mutate.
This is a problem we already are facing when doing, e.g., right-click on a page and selecting "inspect element".
DevTools uses unique "actor IDs" for each DOM nodes that are displayed in the inspector.
Given this ID and code running in the process where the corresponding DOM node exists, it's possible to retrieve the reference to that node directly.
Assignee | ||
Comment 13•6 years ago
|
||
Hi Andreas,
It looks like GeckoDriver/marionette also currently uses drawWindow (from the content process) so will also suffer from the issue of not being able to capture content within OOP iframes.
Could you please have a look at the proposals above and see if they seem feasible to support within marionette?
Thanks!
Assignee | ||
Comment 14•6 years ago
|
||
To clarify, there are two possible APIs that we're considering at this point.
- The parent process-only captureSelector detailed in comment 0.
This requires a unique selector for a content process Element within the parent process (though there is prior art for doing this in toolkit code), and is potentially racy (though devtools has this same issue for inspect element, and it appears to be workable in practice).
- A two part API that captures in content, but returns the rasterized snapshot to the parent process.
In the content process (including OOP <iframe> processes):
SnapshotIdentifier captureSnapshot(Rect)
In the parent process
Promise<ImageBitmap> retrieveSnapshot(SnapshotIdentifier)
The calling code would be responsible for getting the SnapshotIdentifier (an opaque integer ident) to the parent process. This avoids most of the races, since you'd be able to get the bounding rect of an element and have it drawn synchronously. There is still a potential race if the requested area includes OOP content (which is why we can't return the rasterized result), which may still change during the snapshot process.
![]() |
||
Comment 15•6 years ago
•
|
||
For WebDriver, even the API for taking an element screenshot specifically says it is to "draw a bounding box from the framebuffer". That links into the Screen capture section that makes clear that there is only one frame buffer, the "initial viewport’s framebuffer" (i.e. the framebuffer of the top-level window).
Looking at Google's Puppet API - which is generally more flexible I believe - taking a screenshot is only available for Page and Element. "Page" is the top level webpage, which is separate from Frame (which does not provide access to a child "Page", or screenshot API of its own). The Element.screenshot API says that it "uses page.screenshot to take a screenshot" (after scrolling the element into view).
So a parent process based API would seem to be adequate both for WebDriver and for Puppeteer.
![]() |
||
Comment 16•6 years ago
|
||
To be clear, Puppeteer is a "competitor" to WebDriver, not an implementation of it. Puppeteer I believe is basically aimed at exposing Chrome's devtools API to give a lot more flexibility that WebDriver.
Reporter | ||
Comment 17•6 years ago
|
||
Mike mentioned this in the fission meeting today, but there's a JS module for passing cross-process friendly references to DOM elements [1] that we could use here.
Assignee | ||
Updated•6 years ago
|
Comment 18•6 years ago
|
||
Sorry for the delay to my reply, but I’ve been on holiday.
jwatt summarises the needs of WebDriver well, but it bears mentioning
that WebDriver’s presumed “inflexibility” here is dictated by what
can be achieved cross-browser. For Gecko this means a simulated
limitation where we don’t include the rendered content that is
out-of-view for “full document screenshots”. Other browsers have
different limitations, but it is possible that the disappearance
of Edge from the browser landscape means these restrictions can be
lifted in the future.
The relevance of Puppeteer in this context is that we have an ongoing
project to support it for Firefox (tracked in puppeteer and
puppeteer-mvp), so we should make certain a new platform screen
capture API also can support its requirements.
To my understanding, a parent process API like the one outlined in
comment #0 is sufficient for Marionette (and the CDP remote
agent/Puppeteer) as well, as long as it composes in the painted
content from any nested or child browsing contexts with it, i.e.
produces a final screenshot that includes the content of any OOP
<iframe>
s.
Updated•6 years ago
|
Updated•6 years ago
|
Updated•6 years ago
|
Matt, what's left for this bug given that the new drawSnapshot
Gecko API exists now?
Assignee | ||
Comment 20•6 years ago
|
||
Nothing! I think we're all done here.
Description
•