Open Bug 677989 Opened 13 years ago Updated 2 years ago

Allow web content to render without Gecko resampling it (render at native resolution)

Categories

(Core :: General, defect)

Product:
Core
Component:
General
Type:
defect

Tracking

()

People

(Reporter: cjones, Unassigned)

References

(Blocks 1 open bug)

Details

(Whiteboard: [tech-p1]) 

Use case: https://pdfreader.com wants a window-sized <canvas> with pixels that map 1:1 to output device pixels.  Since pdfreader.com is doing its own image resampling and blending, an additional resampling step will hurt quality.  Additionally, pdfreader.com might want to nudge the position and sizes of primitives (like lines) to improve rendering quality (e.g. sharper).  If the pixel coordinate system used by pdfreader.com doesn't map 1:1 to output pixels, then the nudging step is useless, and could even hurt quality.

Use case: https://videoplayer.com chooses videos to play based on the screen resolution reported by the UA.  videoplayer.com wants to control whether video frames are resampled: its primary goal is to be able to play a full-window (or full-screen) video without resampling, to conserve battery.  Secondarily, it may choose to play a lower-resolution video with intentional upsampling.

(These concerns cross desktop and mobile, but it's more pressing on mobile IMHO.)

Last I knew, it wasn't possible to do either across all screen resolutions.  On fennec, setting a <viewport width=device-width zoom=1.0> doesn't get 1:1 CSS-pixel:device-pixel because the fennec frontend maps CSS pixels 1:1.5 device pixels by default (for iPhone compat).  A manual viewport setting has to account for that factor, but the factor isn't visible to web content.

Additionally, on very high dpi (or was it medium-low dpi?) screens, we have another fudge factor for CSS-pixel:device-pixel ratio.  This means that accounting for the 1:1.5 doesn't work on all screens, even if sites knew to hack it in.

I understand and agree with the motivation for trying to hide dpi from web content as much as possible.  Additionally, vector content doesn't need to care much about dpi anyway.  So to be clear, I'm not arguing over the default settings.  What I'm arguing for are "power user" mechanisms to escape the mollycoddling.

I think there are two classes of escape needed
 (1) "Reflow me to window width": served by viewport width=device-width, and less hackily by CSS Device Adaptation.

 (2) "Don't lie to me about screen resolution": <viewport width=device-width zoom=0.66666> isn't a solution.  Do people have ideas on how to solve this?  (I don't, really.)  I wonder if some extension to CSS Device Adaptation that understood min-/max-resolution as specified by media queries would be on the right track.  Personally, I would love to see this not need too much goop to enable, so being able to set a CSS property like |native-window| on (something) to get full-window native-resolution would be great.

Thoughts?
Another issue I neglected to mention for the pdfreader-built-on-canvas is that it would like to know the "zoom" being applied by the frontend so that it could rerasterize at the right resolution.  The pdfreader would like to "allow" zooming since that makes sense, but instead of creating its own incompatible zooming UX, it would be great if it could integrate better into the browser and re-use native zooming.  A separate UX is a valid fallback, though.
One way to get that would be to make canvas use an internal buffer size that's more than one pixel per CSS pixel, automatically chosen to match the zoom factor. Clearing the canvas would reset the internal buffer size. Then you would just need to clear the canvas and rerender in response to a resize() event or something like that.
That won't work for the pdfreader case because raster images drawn by the pdfreader will look like crap when blitted to canvas.  Resolution will be lost (or rather, won't be possible to attain).  That also won't allow the pdfreader do its own device-pixel-accurate nudging/snapping, so spacing etc. will suffer.
(In reply to Chris Jones [:cjones] [:warhammer] from comment #3)
> That won't work for the pdfreader case because raster images drawn by the
> pdfreader will look like crap when blitted to canvas.  Resolution will be
> lost (or rather, won't be possible to attain).

I don't understand this issue.
What you're proposing would almost work for pdfreader.  pdfreader could figure out that it's being lied to if it creates say a 1x1 offscreen canvas every time before drawing, and then counts how many bytes come back in getImageData() (assuming you're proposing that offscreen canvases get a larger backing store too).  Then the pdfreader could do all future computations taking that lie factor into account, drawing objects to fractional "canvas device pixels" and hope that they pass through to the 2d graphics library as integral "real device pixels".  When the pdfreader needed to rasterize objects manually, it would need to be careful to rasterize taking the lie factor into account.

This proposal breaks down when the user zooms in with pinch-zoom, because a subset of the document is visible and there's no way for web content to tell what that is.  We would need to expose something like our displayport setting to web content, or otherwise something like pdfreader would try to allocate a canvas not just the size of the window (much of which would not be visible), but also times zoom-factor^2 (the fudged backing store).  That's a memory-usage nightmare.  But let's put that problem aside for a moment.

First question: imagine implementing libgklayout on top of canvas.  Is the "fudged canvas backing store" the way you would want to deal with rendering to high-DPI devices?  Honest question, I'm not trying to make any points.  It strikes me as quite kludgy for "power-users" like libgklayout and pdfreader, but the expertise of people CC'd here is much higher than mine.

Second question: do you think we can do this automatically for The Web without breaking canvas-using pages left and right due to violated assumption of 1:1 canvas-device-pixel to real-device-pixel?  My intuition is that we can't (yes, I'm aware it's in the spec already).  If we can't do it automatically, then pages need to opt-in using new API.  If we're going to add new API, is the fudged backing store what you would prefer, instead of keeping 1:1 canvas:real pixel and exposing DPI and zoom-factor to pages?  The latter seems like a smaller surface area, and puts the burden on power-user pages instead of forcing the user agent to be smart.

But what you suggest is workable, modulo the pinch-zoom problem.
If you really need to snap to native pixel boundaries then trying to reverse-engineer those boundaries by cobbling together information from getImageData and other sources is horrible. I wouldn't want people to be doing that.

Unfortunately exposing APIs "just for power users" makes me queasy. No doubt all kinds of Web developers will jump on those APIs, copy-paste some example and get things horribly wrong :-(. In this case they'll make assumptions about device characteristics and what kinds of zooming happen and their apps will break when those assumptions are violated.

We do already support -moz-device-pixel-ratio though:
https://developer.mozilla.org/en/CSS/media_queries#-moz-device-pixel-ratio
Seems like that, plus matchMedium, could handle comment #1?
+1 on easy access to actual device pixels. It's a nightmare. I devoted weeks to writing a plugin for Zynga that handles resolutions across lots of iOS, Android and WebOS devices, and I've got sore muscles from endlessly reloading and rotating devices.

-moz-device-pixel-ratio might almost work, if it is similar to the Android setting. It doesn't, however, limit the size of the viewport correctly. By default, I don't see how an empty page should have a scrollbar. Yet it does, because the device height on the meta viewport is set to the absolute screen height, not account for the chrome. Different, but adjacent issue.
-moz-device-pixel-ratio is just about what I want, modulo zooming which I don't believe it will account for.  That's probably enough for a separate bug.

> Unfortunately exposing APIs "just for power users" makes me queasy. No doubt all kinds of Web developers will jump on those APIs, copy-paste some example and get things horribly wrong :-(.

WebGL must have left you sick for weeks :).
(In reply to Chris Jones [:cjones] [:warhammer] from comment #8)
> -moz-device-pixel-ratio is just about what I want, modulo zooming which I
> don't believe it will account for.  That's probably enough for a separate
> bug.

I think it does account for zooming actually.

> > Unfortunately exposing APIs "just for power users" makes me queasy. No doubt all kinds of Web developers will jump on those APIs, copy-paste some example and get things horribly wrong :-(.
> 
> WebGL must have left you sick for weeks :).

In principle, authors copy-pasting and mangling WebGL code can't create lasting compatibility hazards.
> I think it does account for zooming actually.

I meant "resolution" zooming (pinch zoom), not fullZoom zooming.  Zoom.  It should account for the latter but I doubt it accounts for the former.
(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #6)
> We do already support -moz-device-pixel-ratio though:
> https://developer.mozilla.org/en/CSS/media_queries#-moz-device-pixel-ratio
> Seems like that, plus matchMedium, could handle comment #1?

I think we need two more ingredients.  First we need to account for "non-reflow scaling", windowUtils.resolution in gecko, somehow.  We could multiply that into -moz-device-pixel-ratio but that would change the "spec" slightly, although IMHO it's a good change.  Then using that, we could bisect over matchMedium() queries to determine the resolution to arbitrary precision (sigh).

Second, we would need some kind of "onresolutionchange"/"onzoom" event or something along those lines.  That would fire on both fullZoom windowUtils.resolution changes.

(We would continue to assume that canvas.width/height correspond to device pixels.  Otherwise lots of other things would break too.)

With those two additions, plus a grab-bag of fixed-positioned canvas, CSS transform for the inverse of the computed resolution (to prevent the canvas from being rescaled), listening for onscroll and checking the canvas's getBoundingClientRect(), we could compute the region of the PDF covered by the canvas and at what scale, and redraw appropriately.  The PDF page itself would be represented by a large empty HTML document defined in CSS twips, so the UA could choose whatever scale it wanted.

The next problem would creating a larger canvas than the window size, in order to prerender content for faster scrolling, but I think that's possible with an additional level of indirection on top of the fixed-pos canvas.  The next next problem would be building a multi-page view, but that could be built with yet another level of indirection.

Is there a simpler/better solution?
It seems like here you're really trying to bypass the entire browser displayport handling, zooming and scrolling system.

How about going full-screen, or just forcing your viewport to fill the content window, and implementing your own touch-event-handling UI for pinch zooming and touch scrolling?
(In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #12)
> It seems like here you're really trying to bypass the entire browser
> displayport handling, zooming and scrolling system.
> 

*Quite* the contrary: I'm bending over backwards trying to *reuse* as much of it as possible, so that the pdf reader "feels native"; doesn't have to define its own scrolling behavior.

> How about going full-screen, or just forcing your viewport to fill the
> content window, and implementing your own touch-event-handling UI for pinch
> zooming and touch scrolling?

That's always an option.  Also a recipe for lots of hand-rolled subtly incompatible UIs.
(In reply to Chris Jones [:cjones] [:warhammer] from comment #13)
> (In reply to Robert O'Callahan (:roc) (Mozilla Corporation) from comment #12)
> > It seems like here you're really trying to bypass the entire browser
> > displayport handling, zooming and scrolling system.
> 
> *Quite* the contrary: I'm bending over backwards trying to *reuse* as much
> of it as possible, so that the pdf reader "feels native"; doesn't have to
> define its own scrolling behavior.

Sure, you'd like to reuse the UI, but you want to replace the browser's implementation of zooming and scrolling with your own.
I just want the browser to tell my code what rect to repaint at what scale factor.  I want the browser to handle everything else.

felipc and I toyed around with the idea of a "virtual scroll area" kind of widget: a box in which content could set virtual bounds, and the browser would manage a virtual pan offset, scale, and visible region within the box.  Content would repaint in response to scroll/zoomchanged events.  Not sure how to specify this though; a new CSS display mode?
Something like that would make sense. I'll have to think about what the right API would be.

Hmm ... SVG documents have some API support for zooming and panning. See SVGZoomEvent, http://mxr.mozilla.org/mozilla-central/source/dom/interfaces/svg/nsIDOMSVGZoomEvent.idl. Maybe if we hooked that up properly to mobile zoom and pan UI, that would work?
Should we spin the scrollbox off into another bug?  We're getting pretty far off the goal for this one, which was to allow content to reliably disable auto-rescaling by the frontend.
I'll cave, although I'll regret it later.
Here's what I think what content can do to turn this off, e.g. to create an unscaled canvas the width/height of the window
 - gecko makes -moz-device-pixel-ratio account for "resolution scaling", viewport zoom
 - content uses @viewport to set width=device-width and height=device-height.  Can set zoom=1.0, but it doesn't matter.
 - onload, content
   * computes -moz-device-pixel-ratio with binary search.  Call this value "scale".
   * allocates a canvas that's width/height * scale
   * sets up a CSS scale transform of the body by 1 / scale.  Could also just scale the canvas.
 - gecko fires an event whenever -moz-device-pixel-ratio changes
 - on that event, content recomputes the anti-fit scale.  (It should always be true that the originally computed canvas size is the window size in device pixels, so the canvas never has to be reallocated.)
 (- alternately, the page could set user-zoom=fixed, but I kind of hope we don't implement that property.)
This is the default "style" for apps, but we may diverge from that with high-DPI devices.  If so, we need to be able to bring this back.

roc, if there's a better bug on file for this issue, feel free to dup.
Blocks: b2g-v-next
Needed for competitive parity, as other platforms let content have this.
Whiteboard: [tech-p1]
Blocks: 845690
One way to make this easier would be the non-standard "target-densityDpi" viewport property (bug 845690).  Another way would be full support of CSS Device Adaptation, so that authors could write CSS styles like "@viewport { width: 200vw; }".

(In reply to Chris Jones [:cjones] [:warhammer] from comment #21)
> Needed for competitive parity, as other platforms let content have this.

In what way do other platforms do this?  As far as I know, only certain version of Android WebKit ever supported target-densityDpi.  Chrome and Safari and IE do not.
Depends on: 737090, @viewport
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.