STRs: 1. Open https://blog.mozilla.org/webrtc/fiddle-week-downscale-video-peerconnection/ 2. Click the "Result" tab of the embedded fiddle, and share camera. Expected result: - Video elements appear at 640x480 and 320x240 respectively, and stay there. Actual result: - Video elements appear at 640x480 and 320x240 initially, but after a second or two, the latter shrinks to 120x90, and then after a sec or two more, shrinks again to 80x60. Tested on OSX and Win10. Regression range: 4306:00.22 INFO: Narrowed inbound regression window from [23ad1168, 5bbdb7d3] (4 revisions) to [c7428449, 5bbdb7d3] (2 revisions) (~1 steps left) 4306:00.22 INFO: No more inbound revisions, bisection finished. 4306:00.22 INFO: Last good revision: c7428449127566105bdd94b2823b01e0e5e007d5 4306:00.22 INFO: First bad revision: 5bbdb7d36ee3c136a0ed03be9d5b012d05dfd08e 4306:00.22 INFO: Pushlog: https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=c7428449127566105bdd94b2823b01e0e5e007d5&tochange=5bbdb7d36ee3c136a0ed03be9d5b012d05dfd08e
Likely a bug in how scaleDownBy is hooked into the config logic. Upstream doesn't have scaleDownBy support, and this code had to be restructured significantly because they switched to an encoder settings Factory API.
Priority: -- → P1
I'll have a look.
Assignee: nobody → dminor
Interestingly works as expected on Linux. I'll switch over to OSX.
The scaling occurs here . When we start out, max_pixel_count is MAX_INT and we get the expected resolution. After a while, the quality scaler notices the low resolution and asks for it to be increased . This causes max_pixel_count to be set here  which is then scaled down by the scaling factor, resulting in an even smaller resolution, which causes another request from the quality scaler. We should either not scale when resolution_request_max_pixel_count_ is set or ignore it when scaleResolutionDownBy is specified. My inclination is to ignore the quality scaler if scaleResolutionDownBy is specified. :jib, do you know if the standards cover this situation? I couldn't find anything in the webrtc.org spec. Thanks!  http://searchfox.org/mozilla-central/rev/e5b13e6224dbe3182050cf442608c4cb6a8c5c55/media/webrtc/trunk/webrtc/media/base/videoadapter.cc#136  http://searchfox.org/mozilla-central/rev/e5b13e6224dbe3182050cf442608c4cb6a8c5c55/media/webrtc/trunk/webrtc/modules/video_coding/utility/quality_scaler.cc#170  http://searchfox.org/mozilla-central/rev/e5b13e6224dbe3182050cf442608c4cb6a8c5c55/media/webrtc/trunk/webrtc/media/base/videoadapter.cc#125
Status: NEW → ASSIGNED
Conceptually, ScaleResolutionBy should be applied on the input stream (as if the camera was scaled by that amount). Note however that each layer of a simulcast stream can/will have different ScaleResolutionBy values; it simply has to be applied "first". This may be a pain to properly implement in the code. Note that if the actual input resolution changes, all the other dependent values will end up changing (i.e. you can't cache the result of ScaleResoltionBy, you have to apply it each frame). And you don't want to scale twice, you want to calculate the total scale (scaleresolutionby + quality scaling) and then apply it once. The quality scaler must not be ignored; this handles reducing the resolution when CPU use or bandwidth being used is too high. ScaleResolutionBy can be used in non-simulcast cases.
What Randell said. ScaleResolutionBy should be treated as a target size specifier. We should never scale higher than what it specifies.
Created attachment 8898943 [details] [diff] [review] Fix interaction between quality scaler and scaleResolutionBy This takes the minimum of the requested resolution from the quality scaler and scaleResolutionBy. Testing under system load, I do see it go smaller than 320x240 in response to the quality scaler, but it will return to the requested 320x240 when the load decreases. Incidentally, this probably didn't reproduce for me on Linux due to me testing on my desktop and not generating enough system load for the quality scaler to kick in.
Attachment #8898943 - Flags: review?(rjesup)
Comment on attachment 8898943 [details] [diff] [review] Fix interaction between quality scaler and scaleResolutionBy Review of attachment 8898943 [details] [diff] [review]: ----------------------------------------------------------------- land, and we should uplift to beta ::: media/webrtc/trunk/webrtc/media/base/videoadapter.cc @@ +136,3 @@ > // approximates (width/scale_resolution_by_) * (height/scale_resolution_by_) > + int scaled_pixel_count = (in_width*in_height/scale_resolution_by_)/scale_resolution_by_; > + max_pixel_count = std::min(max_pixel_count, scaled_pixel_count); This needs some comment about what this is doing and why. This particular set of lines is largely our code, so we may end up trying to upstream it. Also, upstream code has changed a LOT in 61.
Attachment #8898943 - Flags: review?(rjesup) → review+
Pushed by email@example.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/1427fb9bbe96 Fix interaction between quality scaler and scaleResolutionDownBy; r=jesup
Status: ASSIGNED → RESOLVED
Last Resolved: 10 months ago
status-firefox57: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla57
Comment on attachment 8898943 [details] [diff] [review] Fix interaction between quality scaler and scaleResolutionBy Approval Request Comment [Feature/Bug causing the regression]: Bug 1341285 [User impact if declined]: Incorrect scaling of video streams when under system load. [Is this code covered by automated tests?]: No. I think this would be prone to intermittent failures as the behaviour depends upon the how busy the computer running the tests is, which isn't predictable in automation. [Has the fix been verified in Nightly?]: Yes. [Needs manual test from QE? If yes, steps to reproduce]: No. [List of other uplifts needed for the feature/fix]: None. [Is the change risky?]: No. [Why is the change risky/not risky?]: This change only affects video scaling. [String changes made/needed]: None.
Attachment #8898943 - Flags: approval-mozilla-beta?
status-firefox55: --- → unaffected
status-firefox56: --- → affected
Comment on attachment 8898943 [details] [diff] [review] Fix interaction between quality scaler and scaleResolutionBy Fix for WebRTC regression in 56, let's uplift this for tomorrow's beta 6 build.
Attachment #8898943 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
status-firefox56: affected → fixed
(In reply to Dan Minor [:dminor] from comment #11) > [Is this code covered by automated tests?]: > No. I think this would be prone to intermittent failures as the behaviour > depends upon the how busy the computer running the tests is, which isn't > predictable in automation. > [Has the fix been verified in Nightly?]: > Yes. > [Needs manual test from QE? If yes, steps to reproduce]: > No. Setting qe-verify- based on Dan's assessment on manual testing needs.
You need to log in before you can comment on or make changes to this bug.