1547137 - Potential 4K video quality lost in GeckoView when playing on 4K display

Status: RESOLVED INCOMPLETE

(Core :: Graphics, defect, P3)

Description

[John Lin [:jhlin][:jolin]]

adb shell dumpsys SurfaceFlinger dumps display/layer/surface info [1].

When playing 4K video in Firefox for Fire TV, the info shows:

Display 0 HWC layers:
-------------------------------------------------------------------------------
 Layer name
           Z |  Comp Type |   Disp Frame (LTRB) |          Source Crop (LTRB)
-------------------------------------------------------------------------------
 SurfaceView - org.mozilla.tv.firefox[...]/org.mozilla.tv.firefox.MainActivity
       21005 |     Device |    0    0 3840 2160 |    0.0    0.0 3840.0 2160.0
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
 org.mozilla.tv.firefox.debug/org.mozilla.tv.firefox.MainActivity
       21010 |     Device |    0    0 3840 2160 |    0.0    0.0 1920.0 1080.0
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

h/w composer state:
  h/w composer enabled
Display(0)
 visible_layer:2 commit_layer:2 gles:[-1,-1] mir:-1
+------------------------------------------------------------------------------------------------------------------------+
| layer id |       handle |    z order | blend |           comp      |         source crop   |         display frame | tr|
+----------+--------------+------------+-------+---------------------+-----------------------+-----------------------+---+
|      920 |   0xa5dcf500 |      21005 |   NON | DEV(  MM,DEV,  318) | (   0,   0,3840,2160) | (   0,   0,3840,2160) |  0|
+----------+--------------+------------+-------+---------------------+-----------------------+-----------------------+---+
|      914 |   0xa5dcfbe0 |      21010 |   PRE | DEV(  UI,DEV,  301) | (   0,   0,1920,1080) | (   0,   0,3840,2160) |  0|
+----------+--------------+------------+-------+---------------------+-----------------------+-----------------------+---+

. With the 3840x2160 SurfaceView layer, 4K contents will be displayed w/o scaling.

The layer of GVTV, on the other hand, is only 1920x1080:

Display 0 HWC layers:
-------------------------------------------------------------------------------
 Layer name
           Z |  Comp Type |   Disp Frame (LTRB) |          Source Crop (LTRB)
-------------------------------------------------------------------------------
 SurfaceView - org.mozilla.tv.firefox[...]/org.mozilla.tv.firefox.MainActivity
       21010 |     Device |    0    0 3840 2160 |    0.0    0.0 1920.0 1080.0
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
 org.mozilla.tv.firefox.gecko.debug/org.mozilla.tv.firefox.MainActivity
       21015 |     Device |    0    0 3840 2160 |    0.0    0.0 1920.0 1080.0
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

h/w composer state:
  h/w composer enabled
Display(0)
 visible_layer:2 commit_layer:2 gles:[-1,-1] mir:-1
+------------------------------------------------------------------------------------------------------------------------+
| layer id |       handle |    z order | blend |           comp      |         source crop   |         display frame | tr|
+----------+--------------+------------+-------+---------------------+-----------------------+-----------------------+---+
|      935 |   0xa6b8c640 |      21010 |   NON | DEV(  UI,DEV,  301) | (   0,   0,1920,1080) | (   0,   0,3840,2160) |  0|
+----------+--------------+------------+-------+---------------------+-----------------------+-----------------------+---+
|      934 |   0xa6b8c3c0 |      21015 |   PRE | DEV(  UI,DEV,  301) | (   0,   0,1920,1080) | (   0,   0,3840,2160) |  0|
+----------+--------------+------------+-------+---------------------+-----------------------+-----------------------+---+

, and Gecko/GL compositor will need to scale down 4K video textures for its FHD layer, and rely on h/w composer to scale up back to 4K.

[1] https://source.android.com/devices/graphics/arch-sv-glsv#composition

Comment 1

[John Lin [:jhlin][:jolin]]

https://developer.android.com/about/versions/marshmallow/android-6.0#4K-display has some info about 4K display support since Anddroid 6.0. But it looks like WebView don't need that for video contents. My guess is that WebView manages the sizes of the dedicate SurfaceView for displaying video itself.

Also, detecting 4K display and using 4K surface in GeckoView will probably increase the workload of Gecko compositor and impact the performance and power consumption.

Comment 2

[John Lin [:jhlin][:jolin]]

Currently, I cannot think of any good solution for this. Moving to SurfaceView for video could fix this problem (plus bug 1543625), but it will break the WebGL video texture support in GV. A flow that :snorp suggested in bug 1486659 comment 19 or similar to [1] might work for WebGL, but cannot address bug 1543625.

Out of curiosity, I tried the demo on MDN[2]. It is broken in Fire OS WV, but works fine in the latest WV on Pixel 2/Pie. Will check what they did there.

[1] https://source.android.com/devices/graphics/arch-st#continuous_capture
[2] https://mdn.github.io/webgl-examples/tutorial/sample8/

Comment 3

[Randell Jesup [:jesup] (needinfo me)]

Would what chrome was doing in the CL linked from bug 1529629 help?
Per cpearce and you in IRC, WV is using a dedicated SurfaceView for fullscreen video playback on the Fire; how tough would that be to do?

Comment 4

[John Lin [:jhlin][:jolin]]

Sorry for the late reply.

(In reply to Randell Jesup [:jesup] (needinfo me) from comment #3)

Would what chrome was doing in the CL linked from bug 1529629 help?

IIUC, the CL uses android.app.Dialog to create a surface for video output. That is similar to SurfaceView, but with the ability to layout the surface outside the app view hierarchy.

Per cpearce and you in IRC, WV is using a dedicated SurfaceView for fullscreen video playback on the Fire; how tough would that be to do?

Actually, WV uses SurfaceView for non-fullscreen video too.

For GeckoView to move from SurfaceTexture to SurfaceView, I guess most of the changes will be in gfx/layers/ (something similar to bug 1002823?) and GeckoView gfx/. We might also want to listen to video element fullscreen state changes for decoder or surface recreation.

Bug 1529629 should be able to solve this problem so I will set this one to depend on it.

Comment 5

[Chris Peterson [:cpeterson]]

Dropping priority from P1 to P3 because playing 4K video on non-TV Android devices is an unusual use case.

Comment 6

[Chris Peterson [:cpeterson]]

Moving this bug from the GeckoView Bugzilla component to Graphics.

Comment 7

[u597032]

https://support.mozilla.org/en-US/kb/end-support-firefox-amazon-devices