Last Comment Bug 329693 - Make <canvas> drawWindow Web-accessible
: Make <canvas> drawWindow Web-accessible
Status: NEW
:
Product: Core
Classification: Components
Component: Canvas: 2D (show other bugs)
: Trunk
: All All
: P4 normal with 10 votes (vote)
: ---
Assigned To: Nobody; OK to take it and work on it
:
: Milan Sreckovic [:milan]
Mentors:
Depends on: 343513 448186 313462 445474
Blocks:
  Show dependency treegraph
 
Reported: 2006-03-07 17:25 PST by Robert O'Callahan (:roc) (email my personal email if necessary)
Modified: 2013-05-31 11:14 PDT (History)
21 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments
testcase (1.43 KB, text/html)
2006-03-07 17:51 PST, Robert O'Callahan (:roc) (email my personal email if necessary)
no flags Details
patch (9.61 KB, patch)
2006-03-07 19:14 PST, Robert O'Callahan (:roc) (email my personal email if necessary)
no flags Details | Diff | Splinter Review

Description Robert O'Callahan (:roc) (email my personal email if necessary) 2006-03-07 17:25:38 PST
It would be really useful for Web authors if drawWindow was accessible to Web content. The only problem with this is security: we can't allow Web authors to use a combination of drawWindow and toDataURL to read the contents of foreign-origin images or IFRAMEs. But adding the necessary checks isn't that hard.
Comment 1 Robert O'Callahan (:roc) (email my personal email if necessary) 2006-03-07 17:48:05 PST
Here are the basic rules. This is slightly edited from some spec text I sent Hixie (Ian, the only change is that I allow drawWindow to take a foreign IFRAME parameter; it makes the canvas write-only following the text below):

drawWindow requires significant security considerations. If drawWindow renders content from a domain different to the canvas element's domain (including foreign IFRAMEs), subsequent calls to toDataURL must fail (see the security note for toDataURL in section 6.1). If any of the rendered content has visual properties which would not normally be exposed to the canvas element's domain, including but not limited to:
* full file paths in file input controls, if applicable
* images with a different domain to the canvas
* canvas elements which themselves decline read access via toDataURL
then subsequent toDataURLs should fail.

[An alternative policy would be to "censor" the troublesome content and allow future toDataURL()s. I prefer this policy because it's easier to implement, and also I think it gives more power to application developers. It confines its effects to Web app developers who want to use both drawWindow() and toDataURL() on the same canvas; they just have to be careful about what they render with drawWindow().]

For the sake of not polluting the Web in advance of any standardization, I'm calling this Web-accessible version MozDrawWindow for now. drawWindow itself remains accessible only by chrome. Later we can make drawWindow a synonym for MozDrawWindow.
Comment 2 Hixie (not reading bugmail) 2006-03-07 17:50:55 PST
As per my e-mail, I'm not convinced a blacklist is the way to go here.
Comment 3 Robert O'Callahan (:roc) (email my personal email if necessary) 2006-03-07 17:51:54 PST
Created attachment 214383 [details]
testcase
Comment 4 Jesse Ruderman 2006-03-07 17:58:53 PST
There are many ways for content to reference images on different domains: <img>, background images, <svg:image xlink:href>.

Another thing to worry about is :visited / :link state.  If web sites are allowed to see the colors of links as they are displayed, it will be harder to fix bug 147777, which is a privacy hole.
Comment 5 Robert O'Callahan (:roc) (email my personal email if necessary) 2006-03-07 18:01:44 PST
(In reply to comment #2)
> As per my e-mail, I'm not convinced a blacklist is the way to go here.

The alternative of just always making the canvas write-only would block potentially useful applications, such as using drawWindow to stuff complex content into a graphical editor whose results can be saved. I can't currently think of any other alternatives.
Comment 6 Robert O'Callahan (:roc) (email my personal email if necessary) 2006-03-07 18:04:16 PST
(In reply to comment #4)
> There are many ways for content to reference images on different domains:
> <img>, background images, <svg:image xlink:href>.

My patch traps all of those.

> Another thing to worry about is :visited / :link state.  If web sites are
> allowed to see the colors of links as they are displayed, it will be harder
> to fix bug 147777, which is a privacy hole.

That is true. However bug 147777 is a hole in all browsers, that has existed in public for many years, and no-one seems to care.
Comment 7 Hixie (not reading bugmail) 2006-03-07 18:04:27 PST
To be honest, the phishing implications alone (being able to take a site in its current state, render it to a canvas, then modify the rendering a bit and overlay an input box, e.g.) scare me, irrespective of toDataURL().

What's the use case? Did you have any specific examples in mind?
Comment 8 Robert O'Callahan (:roc) (email my personal email if necessary) 2006-03-07 18:09:18 PST
(In reply to comment #6)
> My patch traps all of those.

... and XUL images, and images in XUL trees, and CSS list-style-image, for the record. It's just a matter of auditing the code for image loading calls.

I guess I can make it force write-only for now, and we can open the door wider later if it matters. 
Comment 9 Robert O'Callahan (:roc) (email my personal email if necessary) 2006-03-07 18:10:32 PST
(In reply to comment #7)
> To be honest, the phishing implications alone (being able to take a site in its
> current state, render it to a canvas, then modify the rendering a bit and
> overlay an input box, e.g.) scare me, irrespective of toDataURL().

As far as I can tell, there's nothing here that you can't already do using IFRAME and CSS positioning.
Comment 10 Hixie (not reading bugmail) 2006-03-07 18:19:17 PST
"It's just a matter of auditing the code for image loading calls."

...and, as you pointed out, <input type=file>. How about images inside XBL binding shadow trees, are you checking those? How do we know we didn't forget something? My point is just that security by blacklist is fundamentally a bad security design. We're bound to have forgotten _something_. It's not a design I'm willing to put my name to.

Regarding the phishing thing, you're probably right, being able to drawWindow() doesn't seem to open any huge holes of that nature.

BTW I couldn't work out what the parameter you mention in comment 1 does.
Comment 11 Jesse Ruderman 2006-03-07 18:21:18 PST
> To be honest, the phishing implications alone (being able to take a site in 
> its current state, render it to a canvas, then modify the rendering a bit 
> and overlay an input box, e.g.) scare me, irrespective of toDataURL().

They scare me too.  But then again, some of the things you can do with iframes alone scare me:
* Put things over (parts of) them.
* Put things under them, if they don't specify a background color (bug 154957).
* Make them invisible to the eye, and yet accept clicks and keys, using opacity.

I think we should consider making cross-site iframes display "all or none": if you overlap part of it, or screw with its opacity ,it doesn't display at all.  But I suppose that if we did that, it would be fairly easy to add <iframe> to the drawWindow blacklist at the same time.v
Comment 12 Jesse Ruderman 2006-03-07 18:25:59 PST
I think I agree with Hixie in that we should be sure there's a big benefit to exposing drawWindow, or the combination of drawWindow and toDataURL, to web content.  They would add a lot to our attack surface and that's something we want to avoid doing without a good reason.
Comment 13 Robert O'Callahan (:roc) (email my personal email if necessary) 2006-03-07 18:39:58 PST
(In reply to comment #10)
> ...and, as you pointed out, <input type=file>. How about images inside XBL
> binding shadow trees, are you checking those?

Yes.

> How do we know we didn't forget something? My point is just that security by
> blacklist is fundamentally a bad security design. We're bound to have
> forgotten _something_. It's not a design I'm willing to put my name to.

Okay. I'm not really clear on the distinction between "security by blacklist" and "enforcing a security policy by putting checks in the right places", but that doesn't matter because I'm agreeing to just block toDataURL completely.

> BTW I couldn't work out what the parameter you mention in comment 1 does.

I allow a foreign IFRAME to be passed to drawWindow, and it gets rendered. It makes the canvas write-only.
Comment 14 Ted Mielczarek [:ted.mielczarek] 2006-03-07 18:46:52 PST
I agree that a drawWindow that disables toDataURL would still be relatively useful.  I can imagine some uses in greasemonkey scripts and the like.
Comment 15 Robert O'Callahan (:roc) (email my personal email if necessary) 2006-03-07 18:49:23 PST
(In reply to comment #12)
> I think I agree with Hixie in that we should be sure there's a big benefit to
> exposing drawWindow, or the combination of drawWindow and toDataURL, to web
> content.  They would add a lot to our attack surface and that's something we
> want to avoid doing without a good reason.

Forget drawWindow+toDataURL. With just drawWindow, what is actually being added to the attack surface?
Comment 16 Robert O'Callahan (:roc) (email my personal email if necessary) 2006-03-07 18:59:36 PST
I think the main use will be that it lets you draw anything the browser can render into a canvas.
-- paragraphs of CSS-styled text
-- tables
-- SVG
-- even windowless Flash plugins...
It's a new avenue of integration between <canvas> and declarative markup.

A secondary use is that it lets people snapshot, zoom and thumbnail existing Web pages from within Web applications.
Comment 17 Robert O'Callahan (:roc) (email my personal email if necessary) 2006-03-07 19:14:30 PST
Created attachment 214392 [details] [diff] [review]
patch

If we decide we want it, here it is.
Comment 18 Hixie (not reading bugmail) 2006-03-07 22:36:09 PST
(In reply to comment #16)
> I think the main use will be that it lets you draw anything the browser can
> render into a canvas.
> -- paragraphs of CSS-styled text
> -- tables
> -- SVG
> -- even windowless Flash plugins...
> It's a new avenue of integration between <canvas> and declarative markup.

I disagree that this is the way to provide this. If we wanted to offer such features (and it isn't clear to me that we do -- from an accessibility perspective alone is seems to lose quite significantly), then I would want to provide a much more targetted API to do them.


> A secondary use is that it lets people snapshot, zoom and thumbnail existing
> Web pages from within Web applications.

So you mean like Google's search showing thumbnails of the target pages... I could see that, I guess. Not sure that it would be much better than actual IFRAMEs, or IFRAMEs in SVG, but I could see a use case somewhat...
Comment 19 Robert O'Callahan (:roc) (email my personal email if necessary) 2006-03-08 00:44:09 PST
(In reply to comment #18)
> (In reply to comment #16)
> > I think the main use will be that it lets you draw anything the browser can
> > render into a canvas.
> > -- paragraphs of CSS-styled text
> > -- tables
> > -- SVG
> > -- even windowless Flash plugins...
> > It's a new avenue of integration between <canvas> and declarative markup.
> 
> I disagree that this is the way to provide this. If we wanted to offer such
> features (and it isn't clear to me that we do -- from an accessibility
> perspective alone is seems to lose quite significantly), then I would want to
> provide a much more targetted API to do them.

There's two discussions there...

I think the case for snapshotting SVG into canvas is quite strong. It lets you use tools like Inkscape to create art that you can then manipulate procedurally where canvas is strong ... e.g. drawing 1000 SVG lemmings. For text and tables and MathML and whatnot, I think the situation is similar to what vlad said on his blog to motivate simple text APIs. Yes, simple text APIs will address that immediate demand, but we know that authors are insatiable. I like the idea that we can say to canvas authors "look, with this you can do whatever you need to do".

As for the API, at the outset I thought about doing a drawElement method. The problem is that the layout and rendering of an element depends on so much context --- the viewport, ancestor elements, the CSS cascade, and other random state --- that it didn't seem to make sense to offer an API to render an element independent of the Window to which it belongs. I'm more than happy to entertain ideas for an alternative API though.

> > A secondary use is that it lets people snapshot, zoom and thumbnail existing
> > Web pages from within Web applications.
> 
> So you mean like Google's search showing thumbnails of the target pages... I
> could see that, I guess. Not sure that it would be much better than actual
> IFRAMEs, or IFRAMEs in SVG, but I could see a use case somewhat...

I don't think that one is actually a good use case for this. I was thinking more like doing slide transitions in a Web-based slideshow, or fancier "preview" modes for various Web-based authoring applications, or thumbnaily navigation bars.
Comment 20 Hixie (not reading bugmail) 2006-03-08 15:57:57 PST
Have you looked at Opera's "declarative <canvas>" work? That may be useful.

What would the markup/script for using this API look like? Maybe that would give us an insight into how simple/complicated the API actually is, and how it might be improved. To take a page from the microformats.org people, it seems we need to do some research before committing to an API here. :-)
Comment 21 Robert O'Callahan (:roc) (email my personal email if necessary) 2006-03-08 16:38:21 PST
(In reply to comment #20)
> Have you looked at Opera's "declarative <canvas>" work? That may be useful.

Where's that? Google fails me.

> What would the markup/script for using this API look like?

You mean the API implemented here? Something like this:

<iframe id="lemming" src="lemming.svg" style="visibility:hidden"></iframe>
<!-- let's say we know that lemming.svg is 100x100 CSS pixels -->
<canvas id="canvas" width="500" height="500"></canvas>
<script>
  var ctx = document.getElementById('canvas').getContext('2d');
  var lemming = document.getElementById('lemming').contentWindow;
  for (i = 0; i < 300; i += 10) {
    ctx.save();
    ctx.translate(i, i);
    ctx.drawWindow(lemming, 0, 0, 100, 100, "transparent");
    ctx.restore();
  }
</script>

Here's what text could look like.

<iframe id="text" src="data:text/html,<body style='margin:0; font-size:12px'></body>" width="50" style="visibility:hidden"></iframe>
<canvas id="canvas" width="500" height="500"></canvas>
<script>
  var ctx = document.getElementById('canvas').getContext('2d');
  var text = document.getElementById('text').contentWindow;
  var rootElem = text.document.documentElement;
  text.document.body.innerHTML = "<em>I</em> am a <span style='font-size:200%'>MORON</span>";
  ctx.save();
  ctx.scale(10, 10);
  ctx.drawWindow(text, 0, 0, rootElem.offsetWidth, rootElem.offsetHeight, "transparent");
  ctx.restore();
</script>

(these examples don't quite work with the patch; you need to change drawWindow to MozDrawWindow, transparent to rgba(0,0,0,0), and draw your own SVG lemming)
Comment 22 Hixie (not reading bugmail) 2006-03-08 17:40:52 PST
Hm, it may not have been published yet. Howcome was showing it to loads of people at the W3C plenary last week.

For the SVG case, I'd much rather we did:

   <canvas id="canvas" width="500" height="500"></canvas>
   <script>
     var ctx = document.getElementById('canvas').getContext('2d');
     var lemming = new Image();
     lemming.src = 'lemming.svg';
     for (i = 0; i < 300; i += 10)
       ctx.drawImage(lemming, i, i, 100, 100);
   </script>

The text case shows exactly why I don't think this is the right API. The code you show doesn't convey "I'm trying to render fancy text". It looks really quite complicated for what it does. Using semantic markup for rendering to a canvas seems weird. Having bogus markup (no <title>, no block-level element, etc) be what you use to render also seems weird, and encouraging non-conformant markup seems like a bad idea. There's also the problem that your iframe would take up space on the page (150px's worth of height), and that display:none would fail (as mentioned earlier).

How about:

   <canvas id="canvas" width="500" height="500"></canvas>
   <script type="text/javascript;e4x=1">
     var ctx = document.getElementById('canvas').getContext('2d');
     ctx.save();
     ctx.scale(10, 10);
     ctx.drawHTML(0, 0, 50, 50,
                  <div style="12px">
                    <i>I</i> am a <font style="2em">MORON</font>
                  </div>);
     ctx.restore();
   </script>

...where drawHTML takes x,y,w,h coordinates and an E4X fragment, the fragment of which is rendered as a standalone XML document with the following stylesheet:

   div { display: block; }
   i { font-style: italics; }
   b { font-weight: bold; }
   u { text-decoration: underline; }
   s { text-decoration: line-through; }
   /* etc */

...and with "style" attributes in the null namespace being treated as CSS sheets that apply to those elements. Or something like that.

The x,y,w,h give the location to start painting (x,y) and the height and width of the viewport to paint (w,h), which matches the height and width of where to paint to.


I dunno, this needs work, but IMHO would already be a much nicer API for authors that having to use HTML <body> tags, etc, in IFrames, etc, to get the same effect.
Comment 23 Robert O'Callahan (:roc) (email my personal email if necessary) 2006-03-08 18:48:08 PST
(In reply to comment #22)
> For the SVG case, I'd much rather we did:
> 
>    <canvas id="canvas" width="500" height="500"></canvas>
>    <script>
>      var ctx = document.getElementById('canvas').getContext('2d');
>      var lemming = new Image();
>      lemming.src = 'lemming.svg';
>      for (i = 0; i < 300; i += 10)
>        ctx.drawImage(lemming, i, i, 100, 100);
>    </script>

Okay. This particular case won't work so well, since lemming.svg won't load instantly so we'll need to block the script until it has loaded or something. But you could use <img src="lemming.svg"> and get the right effect, assuming everything runs off an onload handler, I guess.

> The text case shows exactly why I don't think this is the right API. The code
> you show doesn't convey "I'm trying to render fancy text". It looks really
> quite complicated for what it does. Using semantic markup for rendering to a
> canvas seems weird.

Okay, I should have use <i> instead of <em>.

> Having bogus markup (no <title>, no block-level element,
> etc) be what you use to render also seems weird, and encouraging
> non-conformant markup seems like a bad idea.

If there was a shortcut way to create a valid empty HTML document with no UA BODY margin I would have used it.

> There's also the problem that
> your iframe would take up space on the page (150px's worth of height), and
> that display:none would fail (as mentioned earlier).

Hiding IFRAMEs is easy. Throw a position:absolute in there.

> How about:
> 
>    <canvas id="canvas" width="500" height="500"></canvas>
>    <script type="text/javascript;e4x=1">
>      var ctx = document.getElementById('canvas').getContext('2d');
>      ctx.save();
>      ctx.scale(10, 10);
>      ctx.drawHTML(0, 0, 50, 50,
>                   <div style="12px">
>                     <i>I</i> am a <font style="2em">MORON</font>
>                   </div>);
>      ctx.restore();
>    </script>
> 
> ...where drawHTML takes x,y,w,h coordinates and an E4X fragment, the fragment
> of which is rendered as a standalone XML document with the following
> stylesheet:
> 
>    div { display: block; }
>    i { font-style: italics; }
>    b { font-weight: bold; }
>    u { text-decoration: underline; }
>    s { text-decoration: line-through; }
>    /* etc */
> 
> ...and with "style" attributes in the null namespace being treated as CSS
> sheets that apply to those elements. Or something like that.
>
> The x,y,w,h give the location to start painting (x,y) and the height and
> width of the viewport to paint (w,h), which matches the height and width of
> where to paint to.

Using E4X is a good idea. And I agree that it is a nicer API in terms of syntax.

However, calling it drawHTML when the content isn't actually HTML does not seem like a good idea. Doing a half-hearted job of cloning the presentational aspects of the HTML UA style sheet doesn't seem like a good idea either, people will just wonder what's wrong when they stumble over something missing. I really strongly prefer something that lets people write HTML (or at least XHTML) which turns into real HTML.

How does this work with delayed-loading stuff like IMG elements? Do we wait for an onload in the temporary document before rendering?

This is going to be slower than drawWindow can be, if you need to call it repeatedly. There is no way to incrementally update the content to be rendered and re-render.

You can implement drawHTML fairly easily using drawWindow, but not the other way around. The computer scientist in me thinks that favours drawWindow, but maybe that's a red herring.
Comment 24 Hixie (not reading bugmail) 2006-03-08 19:03:56 PST
My concern is mainly over the semantics of the thing.

Pages shouldn't break if you disable CSS, they should just look less nice. I would consider having all the precached images to be a breaking of the page, not just making it look less nice. Thus, having <img> and <iframe> elements in the source just strikes me as wrong for this case. (I know it's common, but we shouldn't design APIs around bad authoring practices.)

I would be fine with having this API support all of HTML elements, e.g. by treating the given E4X stream as having an implied default namespace of the XHTML namespace; I just couldn't find a way to say that in E4X spec terms.

I'm not saying the API I proposed is perfect, I agree we need to do something to make it easier to optimise. But I think it's a lot better than the drawWindow alternative _for this use case_.

I'm not saying drawWindow() itself isn't useful for other things (notably, taking screenshots).

I'll think about this some more.
Comment 25 Robert O'Callahan (:roc) (email my personal email if necessary) 2006-03-08 19:34:39 PST
(In reply to comment #24)
> I would consider having all the precached images to be a breaking of the
> page, not just making it look less nice. Thus, having <img> and <iframe>
> elements in the source just strikes me as wrong for this case. (I know it's
> common, but we shouldn't design APIs around bad authoring practices.)

That's an interesting point. It suggests to me we need a better way to link resources into a page. IMG/IFRAME may "break the page" but they are declarative, searchable, and survive "Save As".
Comment 26 Hixie (not reading bugmail) 2006-03-08 21:39:30 PST
Indeed. We should raise this on the WHATWG list. Since you have a better handle on the use cases you are trying to address, could you send a mail about it? I would quite like to see what other people's opinions on this are.
Comment 27 Chris Nokleberg 2006-05-09 15:48:16 PDT
My usecase is that I am writing an SVG-based presentation editor, which uses heavy DOM manipulation, and I need the slide thumbnails to update in close to real-time. The ability to take a snapshot of the current SVG document into an image or a canvas would be ideal. Right now the only workaround I know of it to have two identical SVG documents with different dimensions and manipulate them both, which is sub-optimal, especially given SVG memory usage at the moment.
Comment 28 Joe Drew (not getting mail) 2008-03-04 15:34:16 PST
roc/hixie, any updates?
Comment 29 Robert O'Callahan (:roc) (email my personal email if necessary) 2008-03-04 19:16:21 PST
I was thinking about this the other day. Opera is supporting <img src="foobar.svg"> elements and <svg> elements as parameters to the canvas drawImage API. That's a back door to this functionality, via foreignObject. Or we could even allow <object> and <iframe> to be passed directly as parameters to drawImage, on the grounds that they're "just like" <img>. Maybe we should just do that.

The same security issues arise with Opera's approach as I mentioned in comment #1. I wonder if Opera's thought them through.
Comment 30 Dão Gottwald [:dao] 2008-08-06 05:18:19 PDT
Also, it might be a good idea to solve some of the drawWindow bugs before exposing it to web authors.
Comment 31 Hans Schmucker 2008-08-12 16:23:16 PDT
Another usecase. I'm currently writing a tiny photo editor. The majority of filters needed are available as SVG filter primitives, but without a way to render the SVG back into an image, I have no way of offering the result for download, short of implementing the filters with getimagedata/putimagedata, which is terribly, terribly slow.

An image renderer for SVG would do the trick as I could dynamically construct the SVG, but drawWindow is much simpler as I can just create the canvas inside an iframe and render that.

As for security. If we confine drawWindow to a same-domain policy, what would the remaining issues be? I mean is there anything that's visible that isn't visible in the DOM of an in included iframe anyway? Wouldn't that be a good first step?
Comment 32 Robert O'Callahan (:roc) (email my personal email if necessary) 2008-08-12 17:15:04 PDT
Opera's canvas supports SVG elements as a source for drawImage. That would give you what you want. Of course, by sticking a foreignObject in the element it also gives you essentially the functionality of drawWindow, and hence all the problems that were raised in this bug. Opera kind of works around the problems by not supporting foreignObject in that case --- hard to tell if that's intentional or not.
Comment 33 Hans Schmucker 2008-08-13 02:11:30 PDT
Is there a way to block off-site foreignObject, object, iframe and so on requests? Restricting the foreignObject and all of its children to the same domain would allow many uses without any major security problems (I think).
Comment 34 Leonard Ritter 2011-05-19 09:54:35 PDT
Bump!

I'm currently working on a WebGL application that does Cocoa-like presentations of photos and text, with fluidly animated buttons, backgrounds and page flips. It would be very helpful to me if I would be able to style and format HTML content for textures. As I see it, canvas.drawWindow() is still the closest functionality to what I want.

My suggestion for improving security of drawWindow is to have the browser check for nested XSS elements like foreign objects, iframes and so on, and simply refuse to draw if any of these elements are being found. This way, drawing simple HTML elements on canvases and textures becomes possible.
Comment 35 Ted Mielczarek [:ted.mielczarek] 2011-05-19 10:06:34 PDT
You should look into -moz-element: https://developer.mozilla.org/en/CSS/-moz-element

It might provide a more straightforward way to do what you want.
Comment 36 Leonard Ritter 2011-05-19 12:47:01 PDT
Yeah, on my journey I found -moz-element, unfortunately I'm too stupid to fill the gap between "is part of a background", "is on my canvas" and "is in my texture". I don't think it will help.
Comment 37 Robert O'Callahan (:roc) (email my personal email if necessary) 2011-05-20 07:02:18 PDT
Scripts using WebGL are able to read pixel data from textures using shader-based timing attacks. It isn't reasonable to allow scripts to extract the pixel data of drawWindow results --- we don't even want scripts to be able to extract the pixel contents of the user's native theme for form controls --- so we could never allow drawWindow results to be used by WebGL.

Leonard, could CSS 3D transforms be used instead of WebGL for your application? Or at least for the parts that need to manipulate HTML content?
Comment 38 Benoit Jacob [:bjacob] (mostly away) 2011-05-20 07:09:51 PDT
(In reply to comment #37)
> Scripts using WebGL are able to read pixel data from textures using
> shader-based timing attacks. It isn't reasonable to allow scripts to extract
> the pixel data of drawWindow results --- we don't even want scripts to be
> able to extract the pixel contents of the user's native theme for form
> controls --- so we could never allow drawWindow results to be used by WebGL.

Indeed; that could be prevented by marking the canvas as write-only when drawWindow is called. In bug 656277 we're preventing WebGL from using write-only canvases as textures.
Comment 39 Leonard Ritter 2011-05-21 00:03:09 PDT
Which, of course, kills my use case.

What about my suggestion to only draw safe content?

Also, could drawWindow perhaps be expanded by drawElement, a function rendering any hand-crafted HTMLElement, much like -moz-element? I find it cumbersome to first create an IFrame just to get some text flow and CSS styling in a texture.

Robert: CSS just doesn't provide the performance, especially on mobile devices. And there's more I want to do to that HTML, like depth of field blurring and other shader tricks.
Comment 40 Robert O'Callahan (:roc) (email my personal email if necessary) 2011-05-21 06:24:59 PDT
(In reply to comment #39)
> What about my suggestion to only draw safe content?

It's hard to be sure which content is safe. Even native-themed controls could be considered unsafe in that they would allow Web pages to figure out what OS theme you're using, and that assists user fingerprinting and spoofing attacks.

> Also, could drawWindow perhaps be expanded by drawElement, a function
> rendering any hand-crafted HTMLElement, much like -moz-element? I find it
> cumbersome to first create an IFrame just to get some text flow and CSS
> styling in a texture.

Yes it could...

> Robert: CSS just doesn't provide the performance, especially on mobile
> devices.

Then browser developers should probably work on performance of existing features instead of adding new features :-).

> And there's more I want to do to that HTML, like depth of field
> blurring and other shader tricks.

Perhaps we could handle those cases by appying SVG filters to the HTML content, extending SVG filters if necessary.
Comment 41 Robert O'Callahan (:roc) (email my personal email if necessary) 2011-05-21 06:33:29 PDT
(In reply to comment #40)
> (In reply to comment #39)
> > What about my suggestion to only draw safe content?
> 
> It's hard to be sure which content is safe. Even native-themed controls
> could be considered unsafe in that they would allow Web pages to figure out
> what OS theme you're using, and that assists user fingerprinting and
> spoofing attacks.

And note that even if we figured out what content was "safe" (e.g. contains only same-domain images, contains no <input type="file"> controls, contains no non-origin-clean canvases, etc) we'd end up with a feature that was very hard to use reliably because you'd have to be very careful to follow complex rules.
Comment 42 Leonard Ritter 2011-05-21 14:40:33 PDT
Sounds like this WebGL stuff is red herring bollocks then. All that accelerated power for nothing. There I was, thinking I could actually do something decent with the browser.
Comment 43 Robert O'Callahan (:roc) (email my personal email if necessary) 2011-11-07 17:29:52 PST
http://robert.ocallahan.org/2011/11/drawing-dom-content-to-canvas.html
SVG images + foreignObject handle some use-cases for drawing HTML to a canvas.
Comment 44 christoph.burgmer 2012-10-15 00:21:15 PDT
For those on this bug report I'd like to point to the simple JS implementation on top of the concepts Robert pointed out in his blog: http://cburgmer.github.com/rasterizeHTML.js

Isn't the bug somewhat done?

Note You need to log in before you can comment on or make changes to this bug.