Closed Bug 1244913 Opened 10 years ago Closed 9 years ago

Add support for RTCRtpEncodingParameters.scaleResolutionDownBy

Categories

(Core :: WebRTC, defect, P1)

defect

Tracking

()

RESOLVED FIXED
mozilla47
Tracking Status
firefox47 --- fixed

People

(Reporter: jib, Assigned: jib)

References

(Blocks 1 open bug, )

Details

Attachments

(3 files, 1 obsolete file)

This is needed to downscale video in both simulcast and RtpSender in general. As currently spec'ed the knob is a 1/factor rather than explicit width x height. This is perhaps more pressing in Firefox, where getUserMedia returns native camera resolutions - like the spec intends - than in say Chrome which returns whatever you want rescaled.
Hey Jan-Ivar, based on our conversations, I'm assigning this to you and giving it a very high priority. Thanks!
Assignee: nobody → jib
backlog: --- → webrtc/webaudio+
Rank: 10
Priority: -- → P1
Depends on: 1247622
Just got this working with https://jsfiddle.net/kv14z7r6/ I have questions about how maxBitrate and scaleResolutionBy are supposed to interact, as well as the meaning of maxBitrate in unicast vs. simulcast, but what better way to keep discussions concrete than through review requests.
Attachment #8718383 - Flags: review?(docfaraday)
Comment on attachment 8718383 [details] MozReview Request: Bug 1244913 - resolution-based bitrates for each simulcast layer, scaleResolutionDownBy, and working maxBitrate in unicast. https://reviewboard.mozilla.org/r/34539/#review31229 ::: media/webrtc/signaling/src/media-conduit/VideoConduit.cpp:1321 (Diff revision 1) > + vie_codec.maxBitrate = stream.maxBitrate; I think this will break things for the simulcast case, since the sum of the maxBitrates for all simulcast streams will be larger than the value on the vie_codec. You may need to sum the maxBitrates instead. I'm guessing webrtc.org disregards stream.maxBitrate when there's only one simulcast stream, which I would say is a bug.
Comment on attachment 8718384 [details] MozReview Request: Bug 1244913 - add support for RTCRtpEncodingParameters.scaleResolutionDownBy https://reviewboard.mozilla.org/r/34541/#review31235 ::: media/webrtc/signaling/src/media-conduit/VideoConduit.cpp:1288 (Diff revision 1) > + // Recover ratio from SimulcastStream > + double scaleDownBy = double(vie_codec.width ? vie_codec.width : 640) / > + double(stream.width? stream.width : 640); > + if (scaleDownBy > 1) { > + uint32_t max_width = uint32_t(width / scaleDownBy); > + uint32_t max_height = uint32_t(height / scaleDownBy); > + stream.width = width; > + stream.height = height; > + ConstrainPreservingAspectRatio(max_width, max_height, > + &stream.width, &stream.height); I'm pretty concerned about this causing the resolution to go to zero due to the repeated application of floating point imprecision, integer rounding, and the call to ConstrainPreservingAspectRatio. I think we need to explicitly store the scale-down factor somewhere instead of trying to derive it. Also, ConstrainPreservingAspectRatio is only approximate, and webrtc.org refuses to play ball unless the simulcast streams all have _exactly_ the same aspect ratio. You'll need to use ConstrainPreservingAspectRatioExact, probably.
Attachment #8718384 - Flags: review?(docfaraday)
Yeah I wanted to store it in SimulcastStream but was trying to avoid touching upstream code. Seems like we have to.
https://reviewboard.mozilla.org/r/34541/#review31235 > I'm pretty concerned about this causing the resolution to go to zero due to the repeated application of floating point imprecision, integer rounding, and the call to ConstrainPreservingAspectRatio. I think we need to explicitly store the scale-down factor somewhere instead of trying to derive it. > > Also, ConstrainPreservingAspectRatio is only approximate, and webrtc.org refuses to play ball unless the simulcast streams all have _exactly_ the same aspect ratio. You'll need to use ConstrainPreservingAspectRatioExact, probably. Agree; we must store the scale-down factor directly.
Comment on attachment 8718384 [details] MozReview Request: Bug 1244913 - add support for RTCRtpEncodingParameters.scaleResolutionDownBy https://reviewboard.mozilla.org/r/34541/#review31323 ::: media/webrtc/signaling/src/media-conduit/VideoConduit.cpp:1282 (Diff revision 1) > + if (!vie_codec.numberOfSimulcastStreams) { for non-pointer, non-boolean - please use foo != 0 instead of !foo ::: media/webrtc/signaling/src/media-conduit/VideoConduit.cpp:1335 (Diff revision 1) > + mLastFramerateTenths); I think the overall bandwidth-setting algorithm here (interaction between user requests, defaults, and incoming resolution) needs to be a) documented in comments, and b) correspond to the algorithm from byron's and my chat, and I don't think this matches that right now.
Attachment #8718384 - Flags: review?(rjesup)
Comment on attachment 8718383 [details] MozReview Request: Bug 1244913 - resolution-based bitrates for each simulcast layer, scaleResolutionDownBy, and working maxBitrate in unicast. https://reviewboard.mozilla.org/r/34539/#review31495 ::: media/webrtc/signaling/src/media-conduit/VideoConduit.cpp:1321 (Diff revision 1) > + vie_codec.maxBitrate = stream.maxBitrate; please check byron's guess here
Attachment #8718383 - Flags: review?(rjesup)
Attachment #8718383 - Attachment description: MozReview Request: Bug 1244913 - make maxBitrate work in unicast case. → MozReview Request: Bug 1244913 - resolution-based bitrates for each simulcast layer, scaleResolutionDownBy, and working maxBitrate in unicast.
Attachment #8718383 - Flags: review?(rjesup)
Attachment #8718383 - Flags: review?(docfaraday)
Comment on attachment 8718383 [details] MozReview Request: Bug 1244913 - resolution-based bitrates for each simulcast layer, scaleResolutionDownBy, and working maxBitrate in unicast. Review request updated; see interdiff: https://reviewboard.mozilla.org/r/34539/diff/1-2/
Attachment #8718384 - Attachment is obsolete: true
OK this should work a bit better. Note that attachments are listed in the wrong order in bugzilla for some reason. It also looks like moxReview couldn't figure out how things evolved, and has no record of the previous diff. Oh well, this is quite different anyhoo. Take a look.
Note that there's a visible delay in the scaling on the receiving end, which seems like a pre-existing unfotunate reality. I've filed bug 1248154 on needing the wait(1000) to not trip up the viewWidth/viewHeight reported values.
Attachment #8718383 - Flags: review?(docfaraday)
Comment on attachment 8718383 [details] MozReview Request: Bug 1244913 - resolution-based bitrates for each simulcast layer, scaleResolutionDownBy, and working maxBitrate in unicast. https://reviewboard.mozilla.org/r/34539/#review31675 ::: media/webrtc/signaling/src/media-conduit/VideoConduit.cpp:1032 (Diff revision 2) > - out_max = resAndLimits.max_bitrate; > + resAndLimits.resolution_in_mb == 0)) { It is hard to understand that this check is here to ensure that if all else fails, we'll set the outparams once we reach the end of the array. A comment would improve matters. ::: media/webrtc/signaling/src/media-conduit/VideoConduit.cpp:1318 (Diff revision 2) > + SelectBitrates(stream.width, stream.height, stream.jsMaxBitrate, > + mLastFramerateTenths, > + stream.minBitrate, > + stream.targetBitrate, > + stream.maxBitrate); > + > + vie_codec.minBitrate += stream.minBitrate; > + vie_codec.startBitrate += stream.targetBitrate; > + vie_codec.maxBitrate += stream.maxBitrate; > + > + // webrtc.org expects the last, highest fidelity, simulcast stream to > + // always have the same resolution as vie_codec > if (i == vie_codec.numberOfSimulcastStreams) { > vie_codec.width = stream.width; > vie_codec.height = stream.height; > } We seem to have lost the ability to automatically downscale when the max-bitrate is too low to support the full resolution. Is this the intent? ::: media/webrtc/signaling/src/media-conduit/VideoConduit.cpp:1903 (Diff revision 2) > - stream.targetBitrate = MinIgnoreZero(stream.targetBitrate, > + if (stream.jsScaleDownBy > 1.0) { We want to init this stuff if jsScaleDownBy wasn't specified (ie; 1), right? ::: media/webrtc/signaling/src/peerconnection/PeerConnectionImpl.cpp:2410 (Diff revision 2) > + if (encoding.mScaleResolutionDownBy > 1.0) { > + constraint.constraints.maxWidth = 0; > + constraint.constraints.maxHeight = 0; > + constraint.constraints.scaleDownBy = encoding.mScaleResolutionDownBy; > + } IMO, if we have both specified in JS, our behavior probably needs to honor both (scale down as specified, but reduce resolution further to satisfy maxWidth/Height if needed). But I guess we don't have any actual spec language, and it is still unknown whether there will be a maxWidth/Height knob at all.
https://reviewboard.mozilla.org/r/34539/#review31675 > We seem to have lost the ability to automatically downscale when the max-bitrate is too low to support the full resolution. Is this the intent? That was my intent and interpretation of the spec, yes. I'm open to discuss. > IMO, if we have both specified in JS, our behavior probably needs to honor both (scale down as specified, but reduce resolution further to satisfy maxWidth/Height if needed). But I guess we don't have any actual spec language, and it is still unknown whether there will be a maxWidth/Height knob at all. Does it actually fail if we don't do that, or does it just get blocky? Agree the spec isn't clear. Worse, since scaleResolutionDownBy defaults to 1.0 in the webidl in the spec [1], there's no way to distinguish between someone saying: { maxBitrate: 60000 } // = do what you have to do to obey this maxBitrate! and { maxBitrate: 60000, scaleResolutionDownBy: 1 } // bitrate I want. Don't scale! My guess would be to never scale except maybe in the face of utter failure? [1] http://w3c.github.io/webrtc-pc/#widl-RTCRtpEncodingParameters-scaleResolutionDownBy
https://reviewboard.mozilla.org/r/34539/#review31675 > That was my intent and interpretation of the spec, yes. I'm open to discuss. I could be ok with this as long as the scale parameter is available. You might want to ask jesup though. > Does it actually fail if we don't do that, or does it just get blocky? > > Agree the spec isn't clear. Worse, since scaleResolutionDownBy defaults to 1.0 in the webidl in the spec [1], there's no way to distinguish between someone saying: > > { maxBitrate: 60000 } // = do what you have to do to obey this maxBitrate! > > and > > { maxBitrate: 60000, scaleResolutionDownBy: 1 } // bitrate I want. Don't scale! > > My guess would be to never scale except maybe in the face of utter failure? > > [1] http://w3c.github.io/webrtc-pc/#widl-RTCRtpEncodingParameters-scaleResolutionDownBy It gets really blocky/choppy IIRC. That said, restricting bitrate ends up interacting pretty poorly with things like bandwidth estimation, so I'm pretty close to calling maxBitrate a "considered harmful" anyway.
Flags: needinfo?(rjesup)
Comment on attachment 8719176 [details] MozReview Request: Bug 1244913 - change SelectBandwidth to SelectBitrates. https://reviewboard.mozilla.org/r/34899/#review31955
Attachment #8719176 - Flags: review?(rjesup) → review+
Comment on attachment 8718383 [details] MozReview Request: Bug 1244913 - resolution-based bitrates for each simulcast layer, scaleResolutionDownBy, and working maxBitrate in unicast. https://reviewboard.mozilla.org/r/34539/#review31957 ::: media/webrtc/signaling/src/media-conduit/VideoConduit.cpp:997 (Diff revision 2) > -#define MB_OF(w,h) ((unsigned int)((((w)>>4))*((unsigned int)((h)>>4)))) > +#define MB_OF(w,h) ((unsigned int)((((w+15)>>4))*((unsigned int)((h+15)>>4)))) Good update to move this into the macro ::: media/webrtc/signaling/src/media-conduit/VideoConduit.cpp:1082 (Diff revision 2) > + unsigned int fs = mb_width*mb_height; fs = MB_OF(*width, *height) Let the compiler handle de-duplication. ::: media/webrtc/signaling/src/media-conduit/VideoConduit.cpp:1300 (Diff revision 2) > + if (stream.jsScaleDownBy > 1) { I have a concern: Comparisons with "1" (even > and <) in floating-point are in danger of imprecision (i.e. does foo = 1.0 always cause if (foo > 1) to fail? Even if you do calculations that result in "1.0"? One possibility would be apply the scale, and then compare the final size with the original. I.e.: new_width = width / stream.jsScaleDownBy; new_height = height / stream.jsScaleDownBy; if (width != new_width || height != new_height) { ... } And also, this may play into "if the layers are too similar, only allocate bits to one of them". So those comparisons might need to be x% difference (or change the original > 1 test to > 1.1 or 1.3 or whatever we decide). Let's open a new bug for suppressing "too-close" encodings, and reference it here ::: media/webrtc/signaling/src/media-conduit/VideoConduit.cpp:1305 (Diff revision 2) > + // Use less strict scaling in unicast. That way 320x240 / 2 = 160x120. ummm, 320x240/2 == 160x120... I.e. what's the justification for "ConstrainPreservingAspectRatioExact" in the first place? (whole MB scaling) Neither H.264 nor VP8/VP9 require that... Was it something in webrtc.org beyond just keeping the same aspect ratio? I probably reviewed the code, but I don't remember justification (anymore) Byron? ::: media/webrtc/signaling/src/media-conduit/VideoConduit.cpp:1326 (Diff revision 2) > + vie_codec.maxBitrate += stream.maxBitrate; Byron: I don't think vie_codec's meaning is "total of all layers".... especially for minBitrate and startBitrate (maybe). Min should be min of the smallest/slowest encoding. Depending on how it's used internally, max should be either max of the largest encoding, or sum of maxes (as you have here). "Starting" is especially interesting in this case. Whatever the internal definition of the bitrates is, we should document it here. ::: media/webrtc/signaling/src/media-conduit/VideoConduit.cpp:1903 (Diff revision 2) > - stream.targetBitrate = MinIgnoreZero(stream.targetBitrate, > + if (stream.jsScaleDownBy > 1.0) { ditto on 1.0 comparisons ::: media/webrtc/signaling/src/peerconnection/PeerConnectionImpl.cpp:2413 (Diff revision 2) > + constraint.constraints.scaleDownBy = encoding.mScaleResolutionDownBy; Correct me if I'm wrong, but doesn't this imply "scaleDownBy" has precedence over "maxWidth"/etc? Both should be applied: first scaleDownBy, then maxHeight/Width.
Attachment #8718383 - Flags: review?(rjesup)
Flags: needinfo?(rjesup) → needinfo?(docfaraday)
https://reviewboard.mozilla.org/r/34539/#review31711 ::: media/webrtc/signaling/src/media-conduit/VideoConduit.cpp:1903 (Diff revision 2) > - stream.targetBitrate = MinIgnoreZero(stream.targetBitrate, > + if (stream.jsScaleDownBy > 1.0) { They're already init'ed about 20 lines up (to match cinst).
Comment on attachment 8718383 [details] MozReview Request: Bug 1244913 - resolution-based bitrates for each simulcast layer, scaleResolutionDownBy, and working maxBitrate in unicast. Review request updated; see interdiff: https://reviewboard.mozilla.org/r/34539/diff/2-3/
Attachment #8718383 - Flags: review?(rjesup)
Attachment #8718383 - Flags: review?(docfaraday)
Attachment #8720659 - Flags: review?(docfaraday)
Comment on attachment 8718383 [details] MozReview Request: Bug 1244913 - resolution-based bitrates for each simulcast layer, scaleResolutionDownBy, and working maxBitrate in unicast. https://reviewboard.mozilla.org/r/34539/#review32097 ::: media/webrtc/signaling/src/media-conduit/VideoConduit.cpp:1301 (Diff revisions 2 - 3) > - if (stream.jsScaleDownBy > 1) { > + MOZ_ASSERT(stream.jsScaleDownBy >= 1.0); Comparisons with fixed values (like 1/1.0) are tricky/deceptive in FP. If the value is calculated via some math, and is 0.99999999999* due to fp-inexactness, this will assert. Move it into the new != old case, and MOZ_ASSERT new <= old. ::: media/webrtc/signaling/src/media-conduit/VideoConduit.cpp:1304 (Diff revisions 2 - 3) > - > + // TODO: if two layers are too similar, only allocate bits to one of them. Add a bug and note here. ::: media/webrtc/signaling/src/media-conduit/VideoConduit.cpp:1906 (Diff revisions 2 - 3) > + MOZ_ASSERT(stream.jsScaleDownBy >= 1.0); Ditto
Attachment #8718383 - Flags: review?(rjesup) → review+
Comment on attachment 8720659 [details] MozReview Request: Bug 1244913 - Add test_peerConnection_scaleResolution.html https://reviewboard.mozilla.org/r/35415/#review32099
Attachment #8720659 - Flags: review?(rjesup) → review+
https://reviewboard.mozilla.org/r/34539/#review32097 > Comparisons with fixed values (like 1/1.0) are tricky/deceptive in FP. If the value is calculated via some math, and is 0.99999999999* due to fp-inexactness, this will assert. > Move it into the new != old case, and MOZ_ASSERT new <= old. It can't assert because of guarantees in PeerConnection.js, which in turn are dictated by the spec (in comment 24).
https://reviewboard.mozilla.org/r/34539/#review31957 > ummm, 320x240/2 == 160x120... I.e. what's the justification for "ConstrainPreservingAspectRatioExact" in the first place? (whole MB scaling) Neither H.264 nor VP8/VP9 require that... Was it something in webrtc.org beyond just keeping the same aspect ratio? I probably reviewed the code, but I don't remember justification (anymore) > > Byron? Maybe webrtc.org won't care about a non-integral number of macroblocks? Hopefully won't be too hard to check.
Comment on attachment 8718383 [details] MozReview Request: Bug 1244913 - resolution-based bitrates for each simulcast layer, scaleResolutionDownBy, and working maxBitrate in unicast. https://reviewboard.mozilla.org/r/34539/#review32133
Attachment #8718383 - Flags: review?(docfaraday) → review+
Comment on attachment 8720659 [details] MozReview Request: Bug 1244913 - Add test_peerConnection_scaleResolution.html https://reviewboard.mozilla.org/r/35415/#review32135 We either need to have a simulcast test for resolution scaling, or a bug filed for it.
Attachment #8720659 - Flags: review?(docfaraday) → review+
(In reply to Jan-Ivar Bruaroey [:jib] from comment #27) > It can't assert because of guarantees in PeerConnection.js, which in turn > are dictated by the spec (in comment 24). Ok then. Thanks
Flags: needinfo?(docfaraday)
(In reply to Randell Jesup [:jesup] from comment #25) > > + // TODO: if two layers are too similar, only allocate bits to one of them. > > Add a bug and note here. Filed as bug 1249859.
https://reviewboard.mozilla.org/r/34539/#review31957 > Maybe webrtc.org won't care about a non-integral number of macroblocks? Hopefully won't be too hard to check. FWIW test in bug 1249860 passes for me without the ConstrainPreservingAspectRatioExact function, reducing 50x50 to 25x25, so I removed it. > Byron: I don't think vie_codec's meaning is "total of all layers".... especially for minBitrate and startBitrate (maybe). Min should be min of the smallest/slowest encoding. Depending on how it's used internally, max should be either max of the largest encoding, or sum of maxes (as you have here). "Starting" is especially interesting in this case. > > Whatever the internal definition of the bitrates is, we should document it here. Updated bitrate layer totals based on Randell's recommendation.
Comment on attachment 8719176 [details] MozReview Request: Bug 1244913 - change SelectBandwidth to SelectBitrates. Review request updated; see interdiff: https://reviewboard.mozilla.org/r/34899/diff/1-2/
Comment on attachment 8718383 [details] MozReview Request: Bug 1244913 - resolution-based bitrates for each simulcast layer, scaleResolutionDownBy, and working maxBitrate in unicast. Review request updated; see interdiff: https://reviewboard.mozilla.org/r/34539/diff/3-4/
Comment on attachment 8720659 [details] MozReview Request: Bug 1244913 - Add test_peerConnection_scaleResolution.html Review request updated; see interdiff: https://reviewboard.mozilla.org/r/35415/diff/1-2/
Also updated sizing test to check halving without assuming a specific source size, to fix try. Are we good? You both gave me r+'es and all mozReview issues are closed, but feel free to take another look at the latest changes.
https://reviewboard.mozilla.org/r/34539/#review31957 > FWIW test in bug 1249860 passes for me without the ConstrainPreservingAspectRatioExact function, reducing 50x50 to 25x25, so I removed it. I think you're still going to need something like this. 320x240 scaled down by a factor of 3, for example, is going to fail without similar logic. You probably need to teach ConstrainPreservingAspectRatioExact to work on pixels instead of macroblocks, which should be easy.
Comment on attachment 8718383 [details] MozReview Request: Bug 1244913 - resolution-based bitrates for each simulcast layer, scaleResolutionDownBy, and working maxBitrate in unicast. Review request updated; see interdiff: https://reviewboard.mozilla.org/r/34539/diff/4-5/
Comment on attachment 8720659 [details] MozReview Request: Bug 1244913 - Add test_peerConnection_scaleResolution.html Review request updated; see interdiff: https://reviewboard.mozilla.org/r/35415/diff/2-3/
Comment on attachment 8718383 [details] MozReview Request: Bug 1244913 - resolution-based bitrates for each simulcast layer, scaleResolutionDownBy, and working maxBitrate in unicast. Review request updated; see interdiff: https://reviewboard.mozilla.org/r/34539/diff/5-6/
Comment on attachment 8720659 [details] MozReview Request: Bug 1244913 - Add test_peerConnection_scaleResolution.html Review request updated; see interdiff: https://reviewboard.mozilla.org/r/35415/diff/3-4/
Update notes: - Reverted an unintended reordering of SelectBitrates that caused a try failure. - Fixed wrong argument order that caused try failures. - Fixed uninitialized (0) jsScaleDownBy in H.264 case that caused another try failure. - Incorporated feedback from Byron in comment 38, bringing back the ConstrainPreservingAspectRatioExact function at the granularity of pixels rather than macro blocks. Carrying forward r=jesup, bwc.
That followup disables test_peerConnection_scaleResolution.html on Android 4.3 opt, too, since it has about a 30% chance of failure there.
(In reply to Phil Ringnalda (:philor) from comment #47) Thanks!
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: