1525230 - Video cropped when resolution changes

Status: VERIFIED FIXED

(Core :: WebRTC: Audio/Video, defect, P2)

Description

[stefan.birrer]

User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.81 Safari/537.36

Steps to reproduce:

1. Publish a WebRTC stream that may change video dimensions mid stream: http://phenixrts.com/examples/ChannelPublisher/index.html

2. Subscribe to the stream in real-time using h264 codec
   http://phenixrts.com/examples/ChannelViewer/index.html

3. At some point video dimension switches and cropping appears on screen.

This bug was introduced in FF 65.0. We are not able to reproduce in FF 64.0.

Actual results:

The video resolution of the stream may change due to publisher/network constraints. When that happen the display port remains at the same resolution and the video is being cropped.

Expected results:

When the video resolution changes, the display port is adjusted so that the video takes the same space on the screen with higher resolution.

Comment 1

[Matthias Versen [:Matti]]

Can you please run mozregresssion to find the change in Firefox that caused this ?

• https://mozilla.github.io/mozregression/

Comment 2

[Nils Ohlmeier [:drno]]

I tried to reproduce this, but without luck so far. Video is fine in both directions. I'm wondering how we could force the resolution change to happen.

Comment 3

[stefan.birrer]

(In reply to Nils Ohlmeier [:drno] from comment #2)

I tried to reproduce this, but without luck so far. Video is fine in both directions. I'm wondering how we could force the resolution change to happen.

1. https://phenixrts.com/examples/ChannelPublisher/index.html
2. Press "Publish"
3. https://phenixrts.com/examples/ChannelViewer/index.html

This is a full end to end example. Step 1 & 2 creates multiple renditions of the video when publishing. Step 3 switches at least once between two renditions with different resolution. You can refresh the browser in step 3. It typically happens within a few seconds of entering the URL or reloading the viewing page.

If your question relates to an internal test. Perhaps limiting the bitrate midstream would be a way to force the encoder to lower the resolution to stay within target bitrate.

Comment 4

[stefan.birrer]

(In reply to Matthias Versen [:Matti] from comment #1)

Can you please run mozregresssion to find the change in Firefox that caused this ?

• https://mozilla.github.io/mozregression/

Our QA run the regression tool for you and found this commit to be the cause of this bug. Please let us know if you need any additional information:

2019-02-06T17:06:08: DEBUG : Found commit message:
Bug 1509305 - Update webrender to commit 3d7a8fa933769b94875f822b6f4a7803da4320ee (WR PR #3335). r=kats

Differential Revision: https://phabricator.services.mozilla.com/D12649

2019-02-06T17:06:08: DEBUG : Did not find a branch, checking all integration branches
2019-02-06T17:06:08: INFO : The bisection is done.
2019-02-06T17:06:08: INFO : Stopped

Comment 5

[John Lin [:jhlin][:jolin]]

The regression window identified in comment 4 suggests that it seems to be a webrender related (clipping?) issue. Change component to get help from gfx.

Comment 6

[Nils Ohlmeier [:drno]]

Stefan thanks a lot for the help on moving this forward!

Do you know if the problem is limited to Windows only? Or do you see the problem on Mac or Linux as well?

Comment 7

[Jeff Muizelaar [:jrmuizel]]

Stefan, can you post the graphics section of about:support from Firefox 65 where you see the problem?

Comment 8

[Jean-Yves Avenard [:jya]]

(In reply to stefan.birrer from comment #3)

(In reply to Nils Ohlmeier [:drno] from comment #2)

I tried to reproduce this, but without luck so far. Video is fine in both directions. I'm wondering how we could force the resolution change to happen.

1. https://phenixrts.com/examples/ChannelPublisher/index.html
2. Press "Publish"
3. https://phenixrts.com/examples/ChannelViewer/index.html

Ok, I'm must be missing something, but this only gives me a black screen using Firefox Nightly.
At no time am I prompted for sharing webcam or microphone.

One the step 3 (on another machine) the message "joinChannel():subscriberCallback(error, response) returned response status=no-stream-playing

That too is just showing a black rectangle

So what are we missing?

Without the ability to reproduce or proper instructions on how to do we won't get far I'm afraid.

Also, AFAIK, WebRender isn't enabled in Firefox 65, so I think the regression range found is incorrect.

Comment 9

[Jean-Yves Avenard [:jya]]

Ok. I've managed to reproduce using Chrome as the publisher. Somehow that page isn't working with Firefox.

Comment 10

[Jean-Yves Avenard [:jya]]

confirmed not webrender

Comment 11

[Jean-Yves Avenard [:jya]]

Attachment: Bug 1525230 - Reset ImageRect when resolution change. r?bryce

VideoInfo.mImageRect is typically only used for VP8/VP9 content, however the Windows decoder use a common code between H264 and VP8/VP9 decoding and use this data to determine the image size. So we end up with conflicting image size and image rect (which contains cropping data).

Comment 12

[stefan.birrer]

(In reply to Jean-Yves Avenard [:jya] from comment #8)

(In reply to stefan.birrer from comment #3)

(In reply to Nils Ohlmeier [:drno] from comment #2)

I tried to reproduce this, but without luck so far. Video is fine in both directions. I'm wondering how we could force the resolution change to happen.

1. https://phenixrts.com/examples/ChannelPublisher/index.html
2. Press "Publish"
3. https://phenixrts.com/examples/ChannelViewer/index.html

Ok, I'm must be missing something, but this only gives me a black screen using Firefox Nightly.
At no time am I prompted for sharing webcam or microphone.

One the step 3 (on another machine) the message "joinChannel():subscriberCallback(error, response) returned response status=no-stream-playing

That too is just showing a black rectangle

So what are we missing?

Without the ability to reproduce or proper instructions on how to do we won't get far I'm afraid.

Also, AFAIK, WebRender isn't enabled in Firefox 65, so I think the regression range found is incorrect.
Agree on the regression range incorrect. I have to dig in to what went wrong with that.

For reproducing. Did you press 'Publish' on the buttom of the page?

Also where you publish from does not matter. It's only the viewing part that is an issue.

Comment 13

[stefan.birrer]

(In reply to Jean-Yves Avenard [:jya] from comment #9)

Ok. I've managed to reproduce using Chrome as the publisher. Somehow that page isn't working with Firefox.

It does work for me in FF65. Investigating about nightly build.

Comment 14

[stefan.birrer]

(In reply to Nils Ohlmeier [:drno] from comment #6)

Stefan thanks a lot for the help on moving this forward!

Do you know if the problem is limited to Windows only? Or do you see the problem on Mac or Linux as well?

I see this problem on Win 10 and Ubuntu 16.04 myself. Everyone that I asked reproduced the problem on their setup. I have had nobody that didn't reproduce the problem so I believe this affect all systems old/new that run FF65.

Comment 15

[Jan-Ivar Bruaroey [:jib] (needinfo? me)]

[Tracking Requested - why for this release]: webrtc h264 video cropping

Flags set based on regression range reported in comment 4, but not marking as blocking bug 1509305 based on comment 10.

Comment 16

[stefan.birrer]

(In reply to Jan-Ivar Bruaroey [:jib] (needinfo? me) from comment #15)

[Tracking Requested - why for this release]: webrtc h264 video cropping

Flags set based on regression range reported in comment 4, but not marking as blocking bug 1509305 based on comment 10.

Two people have independently found this change set as the one that introduced the regression:

17:54.21 INFO: Last good revision: b446016b76f0efd2c9feee0a9c380bc35e4f0697
17:54.21 INFO: First bad revision: 95d5bb21c934f96482a96579102b3dd853542723
17:54.21 INFO: Pushlog:
https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=b446016b76f0efd2c9feee0a9c380bc35e4f0697&tochange=95d5bb21c934f96482a96579102b3dd853542723

Comment 17

[stefan.birrer]

(In reply to stefan.birrer from comment #16)

(In reply to Jan-Ivar Bruaroey [:jib] (needinfo? me) from comment #15)

[Tracking Requested - why for this release]: webrtc h264 video cropping

Flags set based on regression range reported in comment 4, but not marking as blocking bug 1509305 based on comment 10.

Two people have independently found this change set as the one that introduced the regression:

17:54.21 INFO: Last good revision: b446016b76f0efd2c9feee0a9c380bc35e4f0697
17:54.21 INFO: First bad revision: 95d5bb21c934f96482a96579102b3dd853542723
17:54.21 INFO: Pushlog:
https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=b446016b76f0efd2c9feee0a9c380bc35e4f0697&tochange=95d5bb21c934f96482a96579102b3dd853542723

The tricky part is that one needs to go into about:addons to make sure that Plugins are updated and h264 is available.

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c831a2e31841
Reset ImageRect when resolution change. r=bryce

Comment 19

[Nils Ohlmeier [:drno]]

(In reply to stefan.birrer from comment #16)

(In reply to Jan-Ivar Bruaroey [:jib] (needinfo? me) from comment #15)
Two people have independently found this change set as the one that introduced the regression:

17:54.21 INFO: Last good revision: b446016b76f0efd2c9feee0a9c380bc35e4f0697
17:54.21 INFO: First bad revision: 95d5bb21c934f96482a96579102b3dd853542723
17:54.21 INFO: Pushlog:
https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=b446016b76f0efd2c9feee0a9c380bc35e4f0697&tochange=95d5bb21c934f96482a96579102b3dd853542723

Thank you Stefan. Yes that appears to be the bug which is causing the problem.

Comment 21

[Oana Pop-Rus]

https://hg.mozilla.org/mozilla-central/rev/c831a2e31841

Comment 22

[Jean-Yves Avenard [:jya]]

Comment on attachment 9042106 [details]
Bug 1525230 - Reset ImageRect when resolution change. r?bryce

Beta/Release Uplift Approval Request

Feature/Bug causing the regression

Bug 1505284

User impact if declined

People see cropped images (cut-out people face) during video calls

Is this code covered by automated tests?

No

Has the fix been verified in Nightly?

No

Needs manual test from QE?

Yes

If yes, steps to reproduce

Steps provided in first post

List of other uplifts needed

None

Risk to taking this patch

Low

Why is the change risky/not risky? (and alternatives if risky)

We reset the image size. In the worse case scenario, Aspect ratio for invalid videos (one that change content half way) may not be the same as before. Those are extremely rare,

String changes made/needed

None

Comment 23

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9042106 [details]
Bug 1525230 - Reset ImageRect when resolution change. r?bryce

[Triage Comment]
Approved for 66.0b7. For 65, we're going to go the pref-off route in bug 1520200 instead.

Comment 24

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/releases/mozilla-beta/rev/daa824c92e14

Comment 25

[Catalin Sasca, Desktop Test Engineering [:csasca]]

Attachment: Crop.gif

I tried reproducing the issue, but only got the crop while limiting the speed from the publisher side. The setup I tested on is a macOS 10.12 from where the stream was published using Nightly 67.0a1 (2019-02-10) and the Windows 10 side where channel viewer was setup using both Firefox 65.0 and latest Nightly 67.0a1 (2019-02-12).

In normal circumstances, without limiting the net speed, the stream was rendered correctly and never reproduced the crop, but after using Network Link Conditioner from the mac and tweaking with the settings for bandwith and packets dropped, I constantly got the crop that can be seen in the attachment, both on Firefox 65 and latest Nightly 67.0a1 (2019-02-12).

Stefan, is this the behavior you've been experiencing too?

Comment 26

[Jean-Yves Avenard [:jya]]

The fix was only going to do something on Windows.

If you're seeing it on mac, it's a different bug.

Comment 27

[Catalin Sasca, Desktop Test Engineering [:csasca]]

The mac was used only to publish the stream. The issue is happening on Windows 10.

Comment 28

[Jean-Yves Avenard [:jya]]

(In reply to Catalin Sasca, QA [:csasca] from comment #27)

The mac was used only to publish the stream. The issue is happening on Windows 10.

What I'm seeing is that someone else has published that stream and you're seeing that particular content intermittently.

I can reproduce the same effect with Chrome as ChannelViewer.

Otherwise, I can't reproduce the issue anymore.

When changing the network link condition with the mac utility, typically the video freeze, but that too is occurring with chrome as client.

I did see from time to time a frame being incorrectly sized when the network condition change, but this only occurs for a very short amount of time, it always go back to what it should be quickly. While before it would have stayed that way.

I'll open a new bug for that one.

Comment 29

[Catalin Sasca, Desktop Test Engineering [:csasca]]

In the attachment was the stream which I started from mac, and I covered with my hand just to show the behavior (that 1-2 second of zoom). It never got intercalated by other person's stream (at least it didn't happened to me).

So if that's not the actual behavior of the issue, then I couldn't reproduce it. Are there any other clues which could help me in reproducing the issue?

Comment 30

[stefan.birrer]

For your convenience, this is a 24/7 stream that produces H264 resolution change within the first few seconds:

http://stg.phenixrts.com/examples/H264ResolutionChange/index.html

No need to publish yourself. It loops a clip continuously.

Please note that you need to have H264 support, e.g. about:addons > Plugins > Check for Updates. If your instance lacks H264 support (e.g. fresh nightly install) then you will just get a black screen.

Hope this helps identifying all the different cases and test them easily.

Comment 31

[stefan.birrer]

(In reply to Catalin Sasca, QA [:csasca] from comment #25)

Created attachment 9043559 [details]
Crop.gif

I tried reproducing the issue, but only got the crop while limiting the speed from the publisher side. The setup I tested on is a macOS 10.12 from where the stream was published using Nightly 67.0a1 (2019-02-10) and the Windows 10 side where channel viewer was setup using both Firefox 65.0 and latest Nightly 67.0a1 (2019-02-12).

In normal circumstances, without limiting the net speed, the stream was rendered correctly and never reproduced the crop, but after using Network Link Conditioner from the mac and tweaking with the settings for bandwith and packets dropped, I constantly got the crop that can be seen in the attachment, both on Firefox 65 and latest Nightly 67.0a1 (2019-02-12).

Stefan, is this the behavior you've been experiencing too?

This is certainly related. Given network link conditioning AND a higher resolution on the input, the publisher will downgrade resolution to accommodate lower target bitrate. With Network Link Conditioner you can stimulate the first part. The nuance is the resolution that you publish. If that resolution is low enough in comparison to the new target bitrate then it won't be adjusted. Hence you may see the cropping or not depending on other factors than just network condition.

This demo reproduces the resolution change without fail: http://stg.phenixrts.com/examples/H264ResolutionChange/index.html

It uses a looped clip and when you start up you will be receiving a lower rendition and then upgrade to a higher with a higher resolution. That allows you to reproduce the issue repeatedly on any device. Hope this helps.

Comment 32

[stefan.birrer]

(In reply to Catalin Sasca, QA [:csasca] from comment #29)

In the attachment was the stream which I started from mac, and I covered with my hand just to show the behavior (that 1-2 second of zoom). It never got intercalated by other person's stream (at least it didn't happened to me).

So if that's not the actual behavior of the issue, then I couldn't reproduce it. Are there any other clues which could help me in reproducing the issue?

Try this one http://stg.phenixrts.com/examples/H264ResolutionChange/index.html.

Avoids intercalated issues with others publishing. See other posts for more details.

Comment 33

[Jean-Yves Avenard [:jya]]

(In reply to stefan.birrer from comment #32)

(In reply to Catalin Sasca, QA [:csasca] from comment #29)

In the attachment was the stream which I started from mac, and I covered with my hand just to show the behavior (that 1-2 second of zoom). It never got intercalated by other person's stream (at least it didn't happened to me).

So if that's not the actual behavior of the issue, then I couldn't reproduce it. Are there any other clues which could help me in reproducing the issue?

Try this one http://stg.phenixrts.com/examples/H264ResolutionChange/index.html.

Have you tried with the latest firefox nightly? https://www.mozilla.org/en-US/firefox/channel/desktop/

To reproduce do you still need to play with a network conditioning tool?

I can't reproduce the issue with today's nightly.

Avoids intercalated issues with others publishing. See other posts for more details.

Indeed, someone left the previous publisher running, can hear everything said.

Comment 34

[Jean-Yves Avenard [:jya]]

This is the settings I've tried so far,

Windows PC connected via Wifi to a macbookpro , connected itself via gigabit ethernet to 100/40 link.

The mac is sharing its connection, with the Network Link Conditioner running.

On Windows, go to http://stg.phenixrts.com/examples/H264ResolutionChange/index.html , then change the profile on the mac's network link conditioner between 3G, DSL, LTE and Wifi.

On windows the video quality gets adjusted almost in real-time. I never see any cropped images.

Comment 35

[stefan.birrer]

(In reply to Jean-Yves Avenard [:jya] from comment #34)

This is the settings I've tried so far,

Windows PC connected via Wifi to a macbookpro , connected itself via gigabit ethernet to 100/40 link.

The mac is sharing its connection, with the Network Link Conditioner running.

On Windows, go to http://stg.phenixrts.com/examples/H264ResolutionChange/index.html , then change the profile on the mac's network link conditioner between 3G, DSL, LTE and Wifi.

On windows the video quality gets adjusted almost in real-time. I never see any cropped images.

FF65.0 -> Reproduces the issue on Win10 and Ubuntu 16.04
FF65.0.1 -> NOT able to reproduce on Win10
FF67.0a1/Nightly -> NOT able to reproduce on Win10

Network Link Conditioning is cool though you should be able to see the issue right after opening the page.

Comment 36

[Jean-Yves Avenard [:jya]]

Excellent, thank you for confirming.

Comment 38

[Catalin Sasca, Desktop Test Engineering [:csasca]]

The issue is verified as fixed as mentioned in Comment 35 by Stefan.