1731980 - Youtube video playback only fills 1/4 of the expected display box

Status: RESOLVED FIXED

(Core :: Graphics: WebRender, defect)

Description

[Vengatesh Alagarsamy]

Attachment: IMG_20210922_140119_1.jpg

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:88.0) Gecko/20100101 Firefox/88.0

Steps to reproduce:

Build app Using geckoview in android TV(Redmi)
playing SD video content in app.

Actual results:

SD video content is playing on top left corner not in full screen(all other area are black screen).
But playing HD video its playing on full screen.

Expected results:

SD video content should play in full screen in TV by using geckoview 94.

Comment 1

[Vengatesh Alagarsamy]

Changing severity to S? because of <rationale>.

Comment 2

[Kevin Brosnan [:kbrosnan]]

Priority and severity are set by the triage owner when they review the bugs. Mentor assumes that you know how to fix the bug and want to help a community member fix it at some point in the future.

Ideally a sample app or code snippets plus the exact video you are using for testing would be useful. Does the Geckoview Example browser reproduce the same problem?

Comment 3

[[ex-Mozilla] Agi Sferro | :agi]

Hi Vengatesh, could you attach a sample page/app to reproduce this?

Comment 5

[Vengatesh Alagarsamy]

(In reply to Kevin Brosnan [:kbrosnan] from comment #2)

Priority and severity are set by the triage owner when they review the bugs. Mentor assumes that you know how to fix the bug and want to help a community member fix it at some point in the future.

Ideally a sample app or code snippets plus the exact video you are using for testing would be useful. Does the Geckoview Example browser reproduce the same problem?

I have just install the geckoview_example.apk in android TV(Redmi) from the given link.
Playing youtube videos ,same issue is comming playing on top-left corner.

file:///home/arunjos/BK/device-2021-09-24-182353.png

Comment 6

[Vengatesh Alagarsamy]

Attachment: device-2021-09-24-182353.png

I have just install the geckoview_example.apk in android TV(Redmi) from the given link.
Playing youtube videos ,same issue is comming playing on top-left corner.

Comment 7

[Vengatesh Alagarsamy]

I build app with implementation "org.mozilla.components:browser-engine-gecko:92.0.20210720182827 in android TV,its playing all SD and HD video in full screen.
But in this version i am getting video freeze in both mobile and TV(audio is playing but video stopped).
So, i moved to org.mozilla.components:browser-engine-gecko:94.0.20210907190136 version.
From this version i am not getting video freeze in both mobile and TV ,But in TV video is playing on top-left corner.(SD video with ABR).

Comment 8

[Kevin Brosnan [:kbrosnan]]

I can reproduce this on Firefox 92 but not 91 using the x86 build on emulator.

Still reproduces on a 94 x86 nightly.
94.0a1 (Build #2015836685)
AC: 94.0.20210927143215, 3777ac76a7
GV: 94.0a1-20210927093458
AS: 84.0.0
Mozilla/5.0 (Android 11; Mobile; rv:94.0) Gecko/94.0 Firefox/94.0

Comment 9

[Kevin Brosnan [:kbrosnan]]

2021-09-28T16:35:15.676000: INFO : Narrowed integration regression window from [2c6a5828, ccc6e271] (4 builds) to [2c6a5828, 19948f58] (2 builds) (~1 steps left)
2021-09-28T16:35:16.247000: DEBUG : Found commit message:
Bug 1721190 - Allow fallback from WR to SW-WR on Android. r=jrmuizel
Differential Revision: https://phabricator.services.mozilla.com/D120247
2021-09-28T16:35:16.247000: INFO : The bisection is done.

Comment 10

[Jamie Nicol [:jnicol]]

It sounds like we have 2 bugs - first something is going wrong causing us to fall back to SWGL from hardware webrender, and secondly the video only gets rendered in the top left corner in SWGL.

Could you please go to about:support, click "copy text to clipboard", and attach the contents to this bug? You mention both a mobile and android TV device, could you do that for both please?

Kevin, is there anything in the logcat when you reproduce this on the emulator?

Comment 11

[Kevin Brosnan [:kbrosnan]]

Attachment: 1731980-logcat-startup.txt

Comment 12

[Kevin Brosnan [:kbrosnan]]

Attachment: 1731980-logcat-video-playback.txt

Comment 13

[Vengatesh Alagarsamy]

(In reply to Kevin Brosnan [:kbrosnan] from comment #9)

2021-09-28T16:35:15.676000: INFO : Narrowed integration regression window from [2c6a5828, ccc6e271] (4 builds) to [2c6a5828, 19948f58] (2 builds) (~1 steps left)
2021-09-28T16:35:16.247000: DEBUG : Found commit message:
Bug 1721190 - Allow fallback from WR to SW-WR on Android. r=jrmuizel
Differential Revision: https://phabricator.services.mozilla.com/D120247
2021-09-28T16:35:16.247000: INFO : The bisection is done.

what is the solution for this issue?
Is there any possibilities get revert this changes and to avoid this issue when build geckoview from source?.

Comment 14

[Jamie Nicol [:jnicol]]

I'm afraid you cannot simply revert that change, as the code from the old fallback rendering path has been removed.

In Kevin's logcat I can see that we are using SWGL fallback because the emulator has no GLES 3 support. If he were to configure the emulator to allow GLES 3 (on linux: create the file ~/.android/advancedFeatures.ini and add the line GLESDynamicVersion = on) I suspect he would no longer be able to reproduce. However, setting gfx.webrender.software to true in about config would reproduce again (after a restart).

So it definitely seems like the video rendering in the top left corner is a SWGL bug. But I'm equally curious as to why your device is falling back to SWGL in the first place. From your description of it freezing, I suspect it occurs when attempting to play video rather than at startup. That's also an important bug which we should fix.

Vengatesh, could you please attach the about:support information from both your mobile and TV device to this bug. Could you please also capture a logcat when opening firefox and triggering this bug, then attach it here. That information will be invaluable for fixing this issue. Thanks!

Comment 15

[Jamie Nicol [:jnicol]]

Kevin, did you do anything in particular to reproduce on the emulator? What OS, emulator version, and AVD are you using?

Comment 16

[Vengatesh Alagarsamy]

(In reply to Jamie Nicol [:jnicol] from comment #14)

I'm afraid you cannot simply revert that change, as the code from the old fallback rendering path has been removed.

In Kevin's logcat I can see that we are using SWGL fallback because the emulator has no GLES 3 support. If he were to configure the emulator to allow GLES 3 (on linux: create the file ~/.android/advancedFeatures.ini and add the line GLESDynamicVersion = on) I suspect he would no longer be able to reproduce. However, setting gfx.webrender.software to true in about config would reproduce again (after a restart).

So it definitely seems like the video rendering in the top left corner is a SWGL bug. But I'm equally curious as to why your device is falling back to SWGL in the first place. From your description of it freezing, I suspect it occurs when attempting to play video rather than at startup. That's also an important bug which we should fix.

Vengatesh, could you please attach the about:support information from both your mobile and TV device to this bug. Could you please also capture a logcat when opening firefox and triggering this bug, then attach it here. That information will be invaluable for fixing this issue. Thanks!

From my point of view in new version ABR is working fine with web rendering thats why i am not getting any video freeze issue,the same thing for some devices(Redmi Tv) that web rendering is breaking.
can I please get in when the bug will get solved?.

Comment 17

[Jamie Nicol [:jnicol]]

I am trying to fix the bug as quickly as possible. In order to do so it would be very helpful if you could provide me with the about:support information from your device, as well as a logcat taken when the bug occurs

Comment 18

[Vengatesh Alagarsamy]

Attachment: logcat for redmi TV

(In reply to Jamie Nicol [:jnicol] from comment #17)
> I am trying to fix the bug as quickly as possible. In order to do so it would be very helpful if you could provide me with the about:support information from your device, as well as a logcat taken when the bug occurs

Comment 19

[Vengatesh Alagarsamy]

(In reply to Jamie Nicol [:jnicol] from comment #17)

I am trying to fix the bug as quickly as possible. In order to do so it would be very helpful if you could provide me with the about:support information from your device, as well as a logcat taken when the bug occurs

I attached the logcat for Redmi TV (full screen issue)

Comment 20

[Vengatesh Alagarsamy]

Any update on this issue sir?

Comment 21

[Vengatesh Alagarsamy]

have you found any fix for this (In reply to Jamie Nicol [:jnicol] from comment #17)

I am trying to fix the bug as quickly as possible. In order to do so it would be very helpful if you could provide me with the about:support information from your device, as well as a logcat taken when the bug occurs

Have you found any fix for this issue?

Comment 22

[Jamie Nicol [:jnicol]]

I'm afraid I haven't been able to reproduce the issue on any of my devices. Are there any other websites where you see the issue? Or are you able to share the source code or apk of your geckoview app?

Comment 23

[Vengatesh Alagarsamy]

Its only reproducing in Mali-450 processor devices.
just build (In reply to Jamie Nicol [:jnicol] from comment #22)

I'm afraid I haven't been able to reproduce the issue on any of my devices. Are there any other websites where you see the issue? Or are you able to share the source code or apk of your geckoview app?

Its only reproducing in Mali-450 processor devices.
just https://nightly.maven.mozilla.org/?prefix=maven2/org/mozilla/components/browser-engine-gecko/94.0.20210930143139/ try this binary app in Mali-450 processor devices(Redmi TV) it will produce that issue.

Comment 24

[Marco Castelluccio [:marco]]

Set release status flags based on info from the regressing bug 1721190

Comment 25

[Sotaro Ikeda [:sotaro]]

(In reply to Vengatesh Alagarsamy from comment #23)

Its only reproducing in Mali-450 processor devices.

I have Mali-400 android phone, but I could not reproduce the problem with GeckoView example app. And I tried to reproduce the problem with emulator, but I could not reproduce it.

Comment 26

[Sotaro Ikeda [:sotaro]]

I wonder if RenderCompositorOGLSWGL::HandleExternalImage() does not work as expected when the problem happened.

Comment 27

[Vengatesh Alagarsamy]

(In reply to Sotaro Ikeda [:sotaro] from comment #25)

(In reply to Vengatesh Alagarsamy from comment #23)

Its only reproducing in Mali-450 processor devices.

I have Mali-400 android phone, but I could not reproduce the problem with GeckoView example app. And I tried to reproduce the problem with emulator, but I could not reproduce it.

In all mobile its work fine.Only issue with Android TV.
I checked with two TV

1. Redmi (Mali 450) ->its having that full screen issue
   2.TCL(Mali 470) -> its working fine.
   Thats why i am telling its having issue in Mali 450 only.

Comment 28

[Vengatesh Alagarsamy]

(In reply to Sotaro Ikeda [:sotaro] from comment #25)

(In reply to Vengatesh Alagarsamy from comment #23)

Its only reproducing in Mali-450 processor devices.

I have Mali-400 android phone, but I could not reproduce the problem with GeckoView example app. And I tried to reproduce the problem with emulator, but I could not reproduce it.

Are you able to reproduce this issue in Android TV (Mali 450) ?Is any ideo in which module its causing the issue?After the update on July 21 and 22 ,its causing this issue.

Comment 29

[Vengatesh Alagarsamy]

(In reply to Sotaro Ikeda [:sotaro] from comment #25)

(In reply to Vengatesh Alagarsamy from comment #23)

Its only reproducing in Mali-450 processor devices.

I have Mali-400 android phone, but I could not reproduce the problem with GeckoView example app. And I tried to reproduce the problem with emulator, but I could not reproduce it.

Have you found the issues?will you please tell any other way to solve this issue.

Comment 30

[Jamie Nicol [:jnicol]]

Unfortunately neither Soraro or I have been able to reproduce this issue so have been unable to find a solution.

Kevin, are you still able to reproduce in an emulator? If so, what AVD are you using? What android version, screen dimensions etc. What are the GPU acceleration settings? What host OS are you running on and what GPU does the host computer have?

Comment 31

[Jamie Nicol [:jnicol]]

Re-needinfoing for above question

Comment 32

[Kevin Brosnan [:kbrosnan]]

I can't reproduce this with my current VMs. I did some cleanup a week or two ago and may have deleted the VM that reproduced this. Tried all the configs that I had downloaded (Android 10, 6 and 5.1) and could not reproduce. Not sure that the host info will be useful here but in case it is,

Host: Windows 10 21H1
Host GPU: Nvidia 980 driver v466.27 (27.21.14.6627)

Comment 33

[Vengatesh Alagarsamy]

(In reply to Sotaro Ikeda [:sotaro] from comment #26)

I wonder if RenderCompositorOGLSWGL::HandleExternalImage() does not work as expected when the problem happened.

i Found the issue after changing width and height to 1920 and 1080 in this function its working.But for kernel version after 4.6 only the full screen issue occurred so dynamically i will get the kernel version if kernel version is below 4.6 i will not change any height and width ,else its greater than 4.6 i will change the height and width.

I am facing one more issue,i want move my apk to play store (which i build geckoview from source and included in my app).I want to build geckoview with all architecture(arm 32 and 64 ,x86,x86_64).
i build geckoview with this options.

Use the same Java binary that was specified in bootstrap. This way, if the default system

Java is different than what Firefox needs, users should just need to override it (with

$JAVA_HOME) when running bootstrap, rather than when interacting with the build.

ac_add_options --with-java-bin-path=/usr/lib/jvm/java-8-openjdk-amd64/bin

Build GeckoView/Firefox for Android:

ac_add_options --enable-application=mobile/android

Targeting the following architecture.

For regular phones, no --target is needed.

For x86 emulators (and x86 devices, which are uncommon):

#ac_add_options --target=i686

For newer phones.

#ac_add_options --target=aarch64

For x86_64 emulators (and x86_64 devices, which are even less common):

#ac_add_options --target=x86_64
But playstore denied to upload because its not having arm 64 binaries.
Is there any other procedure to build geckoview for production(play store with all architecture).
or how to i can move arm 32 and arm 64 binary to same app.

Comment 34

[Kevin Brosnan [:kbrosnan]]

Bugs are not a great place to ask for help. This is unrelated to your initial problem. Asking on Matrix or the mailing lists are better places to get help if you are still having problems. https://chat.mozilla.org/#/room/#geckoview:mozilla.org or https://groups.google.com/a/mozilla.com/g/firefox-mobile are

Comment 35

[Vincas Dargis]

Can confirm for Lenovo TB3-850M with Android 6. Videos works fine in official Youtube app, but not in Firefox. Youtube, Vimeo, Facebook videos reproduces this issue:

https://i.imgur.com/HQemo8v.png

Comment 36

[Vincas Dargis]

Some more info:

Compositing: WebRender
WebGL 1 Driver Renderer: ARM -- Mali-T720
WebGL 1 Driver Version: OpenGL ES3.1

Comment 37

[Release mgmt bot [:suhaib / :marco/ :calixte]]

The severity field for this bug is relatively low, S3. However, the bug has 6 See Also bugs.
:gw, could you consider increasing the bug severity?

For more information, please visit auto_nag documentation.

Comment 38

[Jamie Nicol [:jnicol]]

I've finally been able to reproduce this, so am looking in to it

Comment 39

[Jamie Nicol [:jnicol]]

Attachment: Bug 1731980 - Ensure SurfaceTextures with transforms get rendered at correct size. r?#gfx-reviewers

On Android we use SurfaceTextures to render content from sources such
as the video decoder. These may have a transform set which is supposed
to be applied to the texture coordinates used to sample the
texture. Webrender (and software webrender), however, do not handle
this correctly, meaning videos may be rendered at the incorrect size
on some devices.

SurfaceTextures should always be rendered with their bottom-left being
their origin, eg vertically flipped. Additionally, the texture
transform returned on most devices seems to be a simple y-flip
transform with no scaling. Webrender currently just ignores the y-flip
due to the texture origin, which cancels out us not handling the
y-flip from the transform, meaning video looks correct on most
devices. Some devices, however, do return a scaling transform which we
must handle.

This patch removes the override of WebRenderTextureHost::NeedsYFlip()
that was causing us to ignore the y-flip due to the texture origin -
since we will now apply the transform we must handle this correctly
too.

It adds a virtual method RenderTextureHost::GetUvCoords(), that
returns the texture coordinates that should be used by webrender to
sample from external textures. In most cases these are simply (0, 0)
and (size.x, size.y), but in RenderAndroidSurfaceTextureHost we
override this function to apply the transformation. This ensures we
use the correct coordinates whenever the texture is rendered by
webrender, eg in both software and hardware webrender when rendering
in the non-compositing-path, and by hardware webrender's draw
compositor. Additionally, the composite.glsl shader requires a fix to
calculate the UV bounds correctly, as the coordinates may now be
inverted.

Lastly, we fix software webrender with the OpenGL
compositor. CompositorOGL already has the required functionality to
apply the texture transformation as it was used back in the layers
days. We must simply ensure that we pass the value of the
mIgnoreTransform flag from the original SurfaceTextureHost, through to
the RenderAndroidSurfaceTextureHost, and finally to the
SurfaceTextureSource which we hand to CompositorOGL.

Comment 40

[Jamie Nicol [:jnicol]]

Attachment: Bug 1731980 - Ensure SurfaceTextures with transforms get rendered at correct size. r?#gfx-reviewers

On Android we use SurfaceTextures to render content from sources such
as the video decoder. These may have a transform set which is supposed
to be applied to the texture coordinates used to sample the
texture. Webrender (and software webrender), however, do not handle
this correctly, meaning videos may be rendered at the incorrect size
on some devices.

SurfaceTextures should always be rendered with their bottom-left being
their origin, eg vertically flipped. Additionally, the texture
transform returned on most devices seems to be a simple y-flip
transform with no scaling. Webrender currently just ignores the y-flip
due to the texture origin, which cancels out us not handling the
y-flip from the transform, meaning video looks correct on most
devices. Some devices, however, do return a scaling transform which we
must handle.

This patch removes the override of WebRenderTextureHost::NeedsYFlip()
that was causing us to ignore the y-flip due to the texture origin -
since we will now apply the transform we must handle this correctly
too.

It adds a virtual method RenderTextureHost::GetUvCoords(), that
returns the texture coordinates that should be used by webrender to
sample from external textures. In most cases these are simply (0, 0)
and (size.x, size.y), but in RenderAndroidSurfaceTextureHost we
override this function to apply the transformation. This ensures we
use the correct coordinates whenever the texture is rendered by
webrender, eg in both software and hardware webrender when rendering
in the non-compositing-path, and by hardware webrender's draw
compositor. Additionally, the composite.glsl shader requires a fix to
calculate the UV bounds correctly, as the coordinates may now be
inverted.

Lastly, we fix software webrender with the OpenGL
compositor. CompositorOGL already has the required functionality to
apply the texture transformation as it was used back in the layers
days. We must simply ensure that we pass the value of the
mIgnoreTransform flag from the original SurfaceTextureHost, through to
the RenderAndroidSurfaceTextureHost, and finally to the
SurfaceTextureSource which we hand to CompositorOGL.

Comment 41

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/26f0e55c45e7
Ensure SurfaceTextures with transforms get rendered at correct size. r=gfx-reviewers,lsalzman

Comment 42

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/26f0e55c45e7