Simulcast bitrates different than requested

RESOLVED FIXED in Firefox 63

Status

()

enhancement
P2
normal
Rank:
15
RESOLVED FIXED
11 months ago
10 months ago

People

(Reporter: ng, Assigned: bwc)

Tracking

unspecified
mozilla63
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox63 fixed)

Details

Attachments

(3 attachments)

Reporter

Description

11 months ago
A user is requesting a specific set of bitrates[0] for simulcast video but the observed bitrates are lower:
Expected: 150/500/800 kbps 
Observed: 150/300/500 kbps

We should investigate this.

[0] https://fippo.github.io/simulcast-playground/firefox

Comment 2

11 months ago
FWIW, this appears to be machine-dependent. I'm on a mid-2012 Macbook Pro, and I see 90/160/550 when I run that demo. Presuming we rely on third-party code to perform stream decimation, there's a really good chance that it's making decisions based on items like CPU load.

Given that the simulcast rates defined in draft-ietf-mmusic-sdp-simulcast and draft-ietf-mmusic-rid only define *maximums* for each stream (rather than precise bit rates), the behavior that is being described appears to be within spec.

This might simply be an issue where the WebRTC spec needs to be clearer about precisely what promises are made by the bandwidth controls (that is: they're a "not greater than" number rather than a target).
bitrates are now configurable at page load time so can be changed by doing something like
  sessionStorage.bitrates = JSON.stringify([1500000, 800000, 150000])
and then reloading.
abr: the spec says "maxBitrate" so we're good on that. Adapting to available bandwidth and CPU is reasonable but I don't think this happens here, typically webrtc.org's CPU adaption results in reducing the resolution to 960x540.

I can actually get the high-resolution layer to 800kbps by requesting a bitrate of 800kbps for the mid-resolution stream. But the mid-resolution stays at 300kbps -- even though it briefly jumps to 800kbps. This suggests some kind of issue with the allocation of available bandwidth between the streams. Screenshot for
  sessionStorage.bitrates = JSON.stringify([800000, 800000, 150000])
is attached.
i've played around a bit with Chrome 69s (potentially still behind experimental web platform flags) setParameters for simulcast.
This can be done by pasting the following JS  on https://fippo.github.io/simulcast-playground/chrome

var p = pc1.getSenders()[0].getParameters(); p.encodings[1].maxBitrate = 300*1000; p.encodings[2].maxBitrate = 200*1000; pc1.getSenders()[0].setParameters(p);

Chrome is *very* flexible wrt target bitrates. Does Firefox not use the same APIs to control webrtc.org?
I've tracked it to https://cs.chromium.org/chromium/src/third_party/webrtc/pc/rtpsender.cc?sq=package:chromium&dr=CSs&g=0&l=437 (webrtc.org rtpsender) and further down to https://cs.chromium.org/chromium/src/third_party/webrtc/media/engine/webrtcvideoengine.cc?sq=package:chromium&dr=CSs&g=0&l=1801
I went into ./media/webrtc/signaling/src/media-conduit/VideoConduit.cpp
and added some logging in CreateEncoderStreams before the setRid call.

    CSFLogVerbose(LOGTAG, "FIPPO: idx=%ld target br=%d maxBr=%d scaleDown=%f width=%ld", idx, video_stream.target_bitrate_bps, simulcastEncoding.constraints.maxBr, simulcastEncoding.constraints.scaleDownBy, video_stream.width);

which showed this output on the playground:
/tmp/moz.log.child-1:2018-07-27 06:33:34.612924 UTC - [Child 25086: Unnamed thread 0x7f0fd8a5ca60]: V/signaling [|WebrtcVideoSessionConduit] VideoConduit.cpp:670: FIPPO: idx=2 target br=150000 maxBr=0 scaleDown=1,000000 width=320
/tmp/moz.log.child-1:2018-07-27 06:33:34.612982 UTC - [Child 25086: Unnamed thread 0x7f0fd8a5ca60]: V/signaling [|WebrtcVideoSessionConduit] VideoConduit.cpp:670: FIPPO: idx=1 target br=300000 maxBr=500000 scaleDown=2,000000 width=640
/tmp/moz.log.child-1:2018-07-27 06:33:34.612997 UTC - [Child 25086: Unnamed thread 0x7f0fd8a5ca60]: V/signaling [|WebrtcVideoSessionConduit] VideoConduit.cpp:670: FIPPO: idx=0 target br=300000 maxBr=500000 scaleDown=2,000000 width=1280

First, maxBr for the high-resolution stream (idx=0) seems to be capped at 500k even though the JS wants 800k.
Second, tvideo_stream.target_bitrate_bps is 300k for both the mid resolution and the high resolution stream.

and then also configured the mid-resolution stream for 500kbps:
    if (idx == 1) {
      video_stream.target_bitrate_bps = 500000;
    }
that gave me the 500kbps i wanted (similarly for idx=0 and a bitrate of 800kbps).
So it seems that somehow target_bitrate_bps is set the wrong way?
Assignee

Comment 8

11 months ago
There are more factors influencing the per-encoding bandwidth than you can shake a stick at, but something looks very wrong in the logging you've provided in comment 7; the scaleDown params look flat out wrong (I'm assuming the ',' is really a '.' due to localization, but even with that assumption they still look wrong).
we germans like our kommata so yes, this is 2.0. The output resolutions are correct though so maybe this is relative to the "next" resolution unlike in the spec?

I called setParameters with bitrates of [800000, 400000, 123000] and this resulted in the following output:
FIPPO: idx=2 target br=150000 maxBr=0 scaleDown=1,000000 width=320
FIPPO: idx=1 target br=300000 maxBr=400000 scaleDown=2,000000 width=640
FIPPO: idx=0 target br=300000 maxBr=400000 scaleDown=2,000000 width=1280

maxBr of the low resolution spatial layer with idx=2 seems to be 0 always. I don't really care (for now; this might be useful to get only 15fps for that layer at some point)
maxBr for the mid-resolution layer (idx=1) seems to be 400k but the maxBr for idx=0 seems to have the same value.

The target bitrate of 300k comes from kDefaultStartBitrate_bps.

Maybe the error is that maxBr should be cumulative. I tried that but the medium resolution bitrate stayed the same.
Where does the bitrate for the individual streams get updated? Maybe any increments there are only added to the high-resolution layer target bitrate?
after typing a "lo"/"low" typo in the sdp munging pointed out by :bwc:
FIPPO: idx=2 target br=150000 maxBr=150000 scaleDown=4,000000 width=320
FIPPO: idx=1 target br=300000 maxBr=500000 scaleDown=2,000000 width=640
FIPPO: idx=0 target br=300000 maxBr=500000 scaleDown=2,000000 width=1280
scaleDown and maxBr for idx=2 are correct now.

Comment 12

11 months ago
mozreview-review
Comment on attachment 8995631 [details]
Bug 1474661: Fixed bug where we were overwriting the high quality simulcast config with the next one down, and add another entry to the resolutions/bitrates table.

https://reviewboard.mozilla.org/r/260022/#review267084

::: media/webrtc/signaling/src/media-conduit/VideoConduit.cpp
(Diff revision 1)
>    // This means we can't use scaleResolutionBy/scaleDownBy (yet), even if
>    // the user specified it.  The one exception is that we can apply it on
>    // the full-resolution stream (which also happens to handle the
>    // non-simulcast usage case). NOTE: we make an assumption here, not in the
>    // spec, that the first stream is the full-resolution stream.
> -  auto& simulcastEncoding = mConduit->mCurSendCodecConfig->mSimulcastEncodings[0];

i'd just move this to the #if 0 block so the logic of that block still works.
This probably should be nominated for uplift to Beta and ESR60 (after landing and backing/verifying in Nightly).
Assignee: nobody → docfaraday
Rank: 15
Priority: P3 → P2
3>

so this is down to "why does the second layer only give 300kps" which comes from here:
https://searchfox.org/mozilla-central/source/media/webrtc/signaling/src/media-conduit/VideoConduit.cpp#1634

This table is different from Chrome's simulcast table at https://cs.chromium.org/chromium/src/third_party/webrtc/media/engine/simulcast.cc?q=simulcast.cc&sq=package:chromium&g=0&l=52

Its seems webrtc.org doesn't update the target bitrate for the mid-resolution layer. This will probably change if/when the priority is implemented. Until then Chrome actually behaves similar to Firefox:
https://fippo.github.io/simulcast-playground/chrome
pasting
  var p = pc1.getSenders()[0].getParameters(); p.encodings[1].maxBitrate = 800*1000; pc1.getSenders()[0].setParameters(p)
into the JS console does not make the mid-resolution layer go for 800kbps. While i've successfully tried setting the target bitrate to the max bitrate in VideoConduit.cpp this behaviour suggests this is not what Chrome does so its not a good idea.


Can the 300kbps target bitrate in the Firefox table either
1) be changed to 500kbps -- note that this would affect non-simulcast resolutions too.
2) an additional 360p bitrate be added -- this would normally only be used in simulcast as 360p resolution is uncommon enough for macbook pro cameras to barf when requesting it
additionally, it seems that not setting maxBitrate produces a high-resolution layer with ~2mbps -- which is probably the reason this went unnoticed.

There is another issue here which I noticed testing with my SFU. It sometimes got stuck on the low-resolution layer and the REMB estimate never went above 261472. I know some workaround server-side but changing the minimum bitrate for the 640x360 to 150kbps (similar to what is in the Chrome table) seems better as it eliminates the gap.
Comment on attachment 8995645 [details]
Bug 1474661: Fixed bug where we were overwriting the high quality simulcast config with the next one down, and add another entry to the resolutions/bitrates table.

Nico Grunbaum [:ng] has approved the revision.

https://phabricator.services.mozilla.com/D2480
Attachment #8995645 - Flags: review+

Comment 18

11 months ago
Pushed by bcampen@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/42aff55fb6d5
Fixed bug where we were overwriting the high quality simulcast config with the next one down, and add another entry to the resolutions/bitrates table. r=ng

Comment 19

11 months ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/42aff55fb6d5
Status: NEW → RESOLVED
Closed: 11 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla63

Comment 20

11 months ago
@bwc I think Philipp is right here.  Having a minimum bitrate of 300kbps for the medium resolution will make very hard if not impossible the switch from the low resolution to the high resolution.

Why not reusing the same thresholds as Chrome here?  People have been successfully using simulcast there for a while.
Flags: needinfo?(docfaraday)
chrome sends 640x360 quite well at a bitrate of 150kbps. To test paste
  var p = pc1.getSenders()[0].getParameters();
  p.encodings[1].maxBitrate = 150*1000;
  p.encodings[2].active = false;
  pc1.getSenders()[0].setParameters(p)
    .catch(e => console.error(e))
on https://fippo.github.io/simulcast-playground/chrome in Chrome 69+

or modify the demo to add b=AS:300. Unfortunately, b=TIAS seems to be ignored in Firefox when simulcast is used (but that is hardly relevant)
Assignee

Comment 22

11 months ago
Go ahead and open a separate bug for re-evaluating the bandwidth thresholds.
Flags: needinfo?(docfaraday)

Comment 23

11 months ago
Also have a look at Bug 1390215, I think it might make sense to switch over to using the webrtc.org code to configure simulcast layers.
Blocks: 1481725
You need to log in before you can comment on or make changes to this bug.