Video cropped when resolution changes

VERIFIED FIXED in Firefox 66

Status

()

defect
P2
normal
Rank:
15
VERIFIED FIXED
5 months ago
3 months ago

People

(Reporter: stefan.birrer, Assigned: jya)

Tracking

({regression})

65 Branch
mozilla67
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox-esr60 unaffected, firefox65 wontfix, firefox66+ verified, firefox67+ verified)

Details

Attachments

(2 attachments)

User Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.81 Safari/537.36

Steps to reproduce:

  1. Publish a WebRTC stream that may change video dimensions mid stream: http://phenixrts.com/examples/ChannelPublisher/index.html

  2. Subscribe to the stream in real-time using h264 codec
    http://phenixrts.com/examples/ChannelViewer/index.html

  3. At some point video dimension switches and cropping appears on screen.

This bug was introduced in FF 65.0. We are not able to reproduce in FF 64.0.

Actual results:

The video resolution of the stream may change due to publisher/network constraints. When that happen the display port remains at the same resolution and the video is being cropped.

Expected results:

When the video resolution changes, the display port is adjusted so that the video takes the same space on the screen with higher resolution.

Can you please run mozregresssion to find the change in Firefox that caused this ?

Component: Untriaged → WebRTC: Audio/Video
Product: Firefox → Core
See Also: → 1525286

I tried to reproduce this, but without luck so far. Video is fine in both directions. I'm wondering how we could force the resolution change to happen.

(In reply to Nils Ohlmeier [:drno] from comment #2)

I tried to reproduce this, but without luck so far. Video is fine in both directions. I'm wondering how we could force the resolution change to happen.

  1. https://phenixrts.com/examples/ChannelPublisher/index.html
  2. Press "Publish"
  3. https://phenixrts.com/examples/ChannelViewer/index.html

This is a full end to end example. Step 1 & 2 creates multiple renditions of the video when publishing. Step 3 switches at least once between two renditions with different resolution. You can refresh the browser in step 3. It typically happens within a few seconds of entering the URL or reloading the viewing page.

If your question relates to an internal test. Perhaps limiting the bitrate midstream would be a way to force the encoder to lower the resolution to stay within target bitrate.

(In reply to Matthias Versen [:Matti] from comment #1)

Can you please run mozregresssion to find the change in Firefox that caused this ?

Our QA run the regression tool for you and found this commit to be the cause of this bug. Please let us know if you need any additional information:

2019-02-06T17:06:08: DEBUG : Found commit message:
Bug 1509305 - Update webrender to commit 3d7a8fa933769b94875f822b6f4a7803da4320ee (WR PR #3335). r=kats

Differential Revision: https://phabricator.services.mozilla.com/D12649

2019-02-06T17:06:08: DEBUG : Did not find a branch, checking all integration branches
2019-02-06T17:06:08: INFO : The bisection is done.
2019-02-06T17:06:08: INFO : Stopped

The regression window identified in comment 4 suggests that it seems to be a webrender related (clipping?) issue. Change component to get help from gfx.

Component: WebRTC: Audio/Video → Graphics: WebRender

Stefan thanks a lot for the help on moving this forward!

Do you know if the problem is limited to Windows only? Or do you see the problem on Mac or Linux as well?

Flags: needinfo?(stefan.birrer)

Stefan, can you post the graphics section of about:support from Firefox 65 where you see the problem?

(In reply to stefan.birrer from comment #3)

(In reply to Nils Ohlmeier [:drno] from comment #2)

I tried to reproduce this, but without luck so far. Video is fine in both directions. I'm wondering how we could force the resolution change to happen.

  1. https://phenixrts.com/examples/ChannelPublisher/index.html
  2. Press "Publish"
  3. https://phenixrts.com/examples/ChannelViewer/index.html

Ok, I'm must be missing something, but this only gives me a black screen using Firefox Nightly.
At no time am I prompted for sharing webcam or microphone.

One the step 3 (on another machine) the message "joinChannel():subscriberCallback(error, response) returned response status=no-stream-playing

That too is just showing a black rectangle

So what are we missing?

Without the ability to reproduce or proper instructions on how to do we won't get far I'm afraid.

Also, AFAIK, WebRender isn't enabled in Firefox 65, so I think the regression range found is incorrect.

Ok. I've managed to reproduce using Chrome as the publisher. Somehow that page isn't working with Firefox.

confirmed not webrender

Component: Graphics: WebRender → WebRTC: Audio/Video
Flags: needinfo?(stefan.birrer)

VideoInfo.mImageRect is typically only used for VP8/VP9 content, however the Windows decoder use a common code between H264 and VP8/VP9 decoding and use this data to determine the image size. So we end up with conflicting image size and image rect (which contains cropping data).

Assignee: nobody → jyavenard
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true

(In reply to Jean-Yves Avenard [:jya] from comment #8)

(In reply to stefan.birrer from comment #3)

(In reply to Nils Ohlmeier [:drno] from comment #2)

I tried to reproduce this, but without luck so far. Video is fine in both directions. I'm wondering how we could force the resolution change to happen.

  1. https://phenixrts.com/examples/ChannelPublisher/index.html
  2. Press "Publish"
  3. https://phenixrts.com/examples/ChannelViewer/index.html

Ok, I'm must be missing something, but this only gives me a black screen using Firefox Nightly.
At no time am I prompted for sharing webcam or microphone.

One the step 3 (on another machine) the message "joinChannel():subscriberCallback(error, response) returned response status=no-stream-playing

That too is just showing a black rectangle

So what are we missing?

Without the ability to reproduce or proper instructions on how to do we won't get far I'm afraid.

Also, AFAIK, WebRender isn't enabled in Firefox 65, so I think the regression range found is incorrect.
Agree on the regression range incorrect. I have to dig in to what went wrong with that.

For reproducing. Did you press 'Publish' on the buttom of the page?

Also where you publish from does not matter. It's only the viewing part that is an issue.

(In reply to Jean-Yves Avenard [:jya] from comment #9)

Ok. I've managed to reproduce using Chrome as the publisher. Somehow that page isn't working with Firefox.

It does work for me in FF65. Investigating about nightly build.

(In reply to Nils Ohlmeier [:drno] from comment #6)

Stefan thanks a lot for the help on moving this forward!

Do you know if the problem is limited to Windows only? Or do you see the problem on Mac or Linux as well?

I see this problem on Win 10 and Ubuntu 16.04 myself. Everyone that I asked reproduced the problem on their setup. I have had nobody that didn't reproduce the problem so I believe this affect all systems old/new that run FF65.

[Tracking Requested - why for this release]: webrtc h264 video cropping

Flags set based on regression range reported in comment 4, but not marking as blocking bug 1509305 based on comment 10.

Has Regression Range: --- → yes
Rank: 15
Priority: -- → P2

(In reply to Jan-Ivar Bruaroey [:jib] (needinfo? me) from comment #15)

[Tracking Requested - why for this release]: webrtc h264 video cropping

Flags set based on regression range reported in comment 4, but not marking as blocking bug 1509305 based on comment 10.

Two people have independently found this change set as the one that introduced the regression:

17:54.21 INFO: Last good revision: b446016b76f0efd2c9feee0a9c380bc35e4f0697
17:54.21 INFO: First bad revision: 95d5bb21c934f96482a96579102b3dd853542723
17:54.21 INFO: Pushlog:
https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=b446016b76f0efd2c9feee0a9c380bc35e4f0697&tochange=95d5bb21c934f96482a96579102b3dd853542723

(In reply to stefan.birrer from comment #16)

(In reply to Jan-Ivar Bruaroey [:jib] (needinfo? me) from comment #15)

[Tracking Requested - why for this release]: webrtc h264 video cropping

Flags set based on regression range reported in comment 4, but not marking as blocking bug 1509305 based on comment 10.

Two people have independently found this change set as the one that introduced the regression:

17:54.21 INFO: Last good revision: b446016b76f0efd2c9feee0a9c380bc35e4f0697
17:54.21 INFO: First bad revision: 95d5bb21c934f96482a96579102b3dd853542723
17:54.21 INFO: Pushlog:
https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=b446016b76f0efd2c9feee0a9c380bc35e4f0697&tochange=95d5bb21c934f96482a96579102b3dd853542723

The tricky part is that one needs to go into about:addons to make sure that Plugins are updated and h264 is available.

Pushed by jyavenard@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/c831a2e31841
Reset ImageRect when resolution change. r=bryce
Depends on: 1505284
Keywords: regression

(In reply to stefan.birrer from comment #16)

(In reply to Jan-Ivar Bruaroey [:jib] (needinfo? me) from comment #15)
Two people have independently found this change set as the one that introduced the regression:

17:54.21 INFO: Last good revision: b446016b76f0efd2c9feee0a9c380bc35e4f0697
17:54.21 INFO: First bad revision: 95d5bb21c934f96482a96579102b3dd853542723
17:54.21 INFO: Pushlog:
https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=b446016b76f0efd2c9feee0a9c380bc35e4f0697&tochange=95d5bb21c934f96482a96579102b3dd853542723

Thank you Stefan. Yes that appears to be the bug which is causing the problem.

See Also: → 1520200
Duplicate of this bug: 1519860
Status: ASSIGNED → RESOLVED
Closed: 5 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla67

Comment on attachment 9042106 [details]
Bug 1525230 - Reset ImageRect when resolution change. r?bryce

Beta/Release Uplift Approval Request

Feature/Bug causing the regression

Bug 1505284

User impact if declined

People see cropped images (cut-out people face) during video calls

Is this code covered by automated tests?

No

Has the fix been verified in Nightly?

No

Needs manual test from QE?

Yes

If yes, steps to reproduce

Steps provided in first post

List of other uplifts needed

None

Risk to taking this patch

Low

Why is the change risky/not risky? (and alternatives if risky)

We reset the image size. In the worse case scenario, Aspect ratio for invalid videos (one that change content half way) may not be the same as before. Those are extremely rare,

String changes made/needed

None

Attachment #9042106 - Flags: approval-mozilla-release?
Attachment #9042106 - Flags: approval-mozilla-beta?

Comment on attachment 9042106 [details]
Bug 1525230 - Reset ImageRect when resolution change. r?bryce

[Triage Comment]
Approved for 66.0b7. For 65, we're going to go the pref-off route in bug 1520200 instead.

Attachment #9042106 - Flags: approval-mozilla-release?
Attachment #9042106 - Flags: approval-mozilla-release-
Attachment #9042106 - Flags: approval-mozilla-beta?
Attachment #9042106 - Flags: approval-mozilla-beta+
Whiteboard: [qa-triaged]
Posted image Crop.gif

I tried reproducing the issue, but only got the crop while limiting the speed from the publisher side. The setup I tested on is a macOS 10.12 from where the stream was published using Nightly 67.0a1 (2019-02-10) and the Windows 10 side where channel viewer was setup using both Firefox 65.0 and latest Nightly 67.0a1 (2019-02-12).

In normal circumstances, without limiting the net speed, the stream was rendered correctly and never reproduced the crop, but after using Network Link Conditioner from the mac and tweaking with the settings for bandwith and packets dropped, I constantly got the crop that can be seen in the attachment, both on Firefox 65 and latest Nightly 67.0a1 (2019-02-12).

Stefan, is this the behavior you've been experiencing too?

Flags: needinfo?(stefan.birrer)

The fix was only going to do something on Windows.

If you're seeing it on mac, it's a different bug.

The mac was used only to publish the stream. The issue is happening on Windows 10.

(In reply to Catalin Sasca, QA [:csasca] from comment #27)

The mac was used only to publish the stream. The issue is happening on Windows 10.

What I'm seeing is that someone else has published that stream and you're seeing that particular content intermittently.

I can reproduce the same effect with Chrome as ChannelViewer.

Otherwise, I can't reproduce the issue anymore.

When changing the network link condition with the mac utility, typically the video freeze, but that too is occurring with chrome as client.

I did see from time to time a frame being incorrectly sized when the network condition change, but this only occurs for a very short amount of time, it always go back to what it should be quickly. While before it would have stayed that way.

I'll open a new bug for that one.

In the attachment was the stream which I started from mac, and I covered with my hand just to show the behavior (that 1-2 second of zoom). It never got intercalated by other person's stream (at least it didn't happened to me).

So if that's not the actual behavior of the issue, then I couldn't reproduce it. Are there any other clues which could help me in reproducing the issue?

Flags: needinfo?(jyavenard)

For your convenience, this is a 24/7 stream that produces H264 resolution change within the first few seconds:

http://stg.phenixrts.com/examples/H264ResolutionChange/index.html

No need to publish yourself. It loops a clip continuously.

Please note that you need to have H264 support, e.g. about:addons > Plugins > Check for Updates. If your instance lacks H264 support (e.g. fresh nightly install) then you will just get a black screen.

Hope this helps identifying all the different cases and test them easily.

Flags: needinfo?(stefan.birrer)

(In reply to Catalin Sasca, QA [:csasca] from comment #25)

Created attachment 9043559 [details]
Crop.gif

I tried reproducing the issue, but only got the crop while limiting the speed from the publisher side. The setup I tested on is a macOS 10.12 from where the stream was published using Nightly 67.0a1 (2019-02-10) and the Windows 10 side where channel viewer was setup using both Firefox 65.0 and latest Nightly 67.0a1 (2019-02-12).

In normal circumstances, without limiting the net speed, the stream was rendered correctly and never reproduced the crop, but after using Network Link Conditioner from the mac and tweaking with the settings for bandwith and packets dropped, I constantly got the crop that can be seen in the attachment, both on Firefox 65 and latest Nightly 67.0a1 (2019-02-12).

Stefan, is this the behavior you've been experiencing too?

This is certainly related. Given network link conditioning AND a higher resolution on the input, the publisher will downgrade resolution to accommodate lower target bitrate. With Network Link Conditioner you can stimulate the first part. The nuance is the resolution that you publish. If that resolution is low enough in comparison to the new target bitrate then it won't be adjusted. Hence you may see the cropping or not depending on other factors than just network condition.

This demo reproduces the resolution change without fail: http://stg.phenixrts.com/examples/H264ResolutionChange/index.html

It uses a looped clip and when you start up you will be receiving a lower rendition and then upgrade to a higher with a higher resolution. That allows you to reproduce the issue repeatedly on any device. Hope this helps.

(In reply to Catalin Sasca, QA [:csasca] from comment #29)

In the attachment was the stream which I started from mac, and I covered with my hand just to show the behavior (that 1-2 second of zoom). It never got intercalated by other person's stream (at least it didn't happened to me).

So if that's not the actual behavior of the issue, then I couldn't reproduce it. Are there any other clues which could help me in reproducing the issue?

Try this one http://stg.phenixrts.com/examples/H264ResolutionChange/index.html.

Avoids intercalated issues with others publishing. See other posts for more details.

(In reply to stefan.birrer from comment #32)

(In reply to Catalin Sasca, QA [:csasca] from comment #29)

In the attachment was the stream which I started from mac, and I covered with my hand just to show the behavior (that 1-2 second of zoom). It never got intercalated by other person's stream (at least it didn't happened to me).

So if that's not the actual behavior of the issue, then I couldn't reproduce it. Are there any other clues which could help me in reproducing the issue?

Try this one http://stg.phenixrts.com/examples/H264ResolutionChange/index.html.

Have you tried with the latest firefox nightly? https://www.mozilla.org/en-US/firefox/channel/desktop/

To reproduce do you still need to play with a network conditioning tool?

I can't reproduce the issue with today's nightly.

Avoids intercalated issues with others publishing. See other posts for more details.

Indeed, someone left the previous publisher running, can hear everything said.

Flags: needinfo?(jyavenard) → needinfo?(stefan.birrer)

This is the settings I've tried so far,

Windows PC connected via Wifi to a macbookpro , connected itself via gigabit ethernet to 100/40 link.

The mac is sharing its connection, with the Network Link Conditioner running.

On Windows, go to http://stg.phenixrts.com/examples/H264ResolutionChange/index.html , then change the profile on the mac's network link conditioner between 3G, DSL, LTE and Wifi.

On windows the video quality gets adjusted almost in real-time. I never see any cropped images.

(In reply to Jean-Yves Avenard [:jya] from comment #34)

This is the settings I've tried so far,

Windows PC connected via Wifi to a macbookpro , connected itself via gigabit ethernet to 100/40 link.

The mac is sharing its connection, with the Network Link Conditioner running.

On Windows, go to http://stg.phenixrts.com/examples/H264ResolutionChange/index.html , then change the profile on the mac's network link conditioner between 3G, DSL, LTE and Wifi.

On windows the video quality gets adjusted almost in real-time. I never see any cropped images.

FF65.0 -> Reproduces the issue on Win10 and Ubuntu 16.04
FF65.0.1 -> NOT able to reproduce on Win10
FF67.0a1/Nightly -> NOT able to reproduce on Win10

Network Link Conditioning is cool though you should be able to see the issue right after opening the page.

Flags: needinfo?(stefan.birrer)

Excellent, thank you for confirming.

Status: RESOLVED → VERIFIED
Duplicate of this bug: 1525286

The issue is verified as fixed as mentioned in Comment 35 by Stefan.

QA Whiteboard: [qa-triaged]
Whiteboard: [qa-triaged]
Duplicate of this bug: 1519860
You need to log in before you can comment on or make changes to this bug.