972639 - Investigate CPU impact of WebRTC content analysis code

Status: NEW

(Core :: WebRTC: Audio/Video, defect)

Description

[Gian-Carlo Pascutto [:gcp]]

The current content analysis code in WebRTC does some analysis of the picture and picture motion. On low end devices this may end up being significant in itself. Investigate the real impact.

Comment 1

[Andreas Gal :gal]

Do we have a profile showing this?

Comment 2

[Randell Jesup [:jesup] (needinfo me)]

(In reply to Andreas Gal :gal from comment #1)
> Do we have a profile showing this?

Not yet, that will be one of the first actions here (on desktop, on high-end mobile, and on low-end mobile).  A primary concern may be extra frame copies made into CPU space and the amount of memory traffic required to analyze them.  Possible mitigations discussed include disabling subsets of the analysis, scaling in the GPU before the copy and analyzing the scaled copy for complexity/motion, or simply disabling complexity analysis entirely and sticking to simpler heuristics.  Note that this requires some hacking to disable (not a lot), as this API is also where we're processing (CPU) load information.

The last (disable complexity/motion, use simpler heuristics) may be the best in the end (at least for mobil), as this code block is considered experimental by upstream (we have a ping into them on the status/plans).  However, this may require some tuning of the heuristics.  And complexity analysis may be useful/win on Desktop.

The first steps are  to find out upstream plans (underway), get some profiles, and see if the results of complexity analysis do what we want/need (which is to keep good user experience across a range of bandwidth, load and motion/background situations).

Comment 3

[Jay]

Randell,

Is there any update on this?

Comment 4

[Jay]

[Blocking Requested - why for this release]:

Actually, I see that the ContentMetrics is only used by VP8. Please, see the attached code below.

http://git.mozilla.org/?p=releases/gecko.git;a=blob;f=media/webrtc/trunk/webrtc/video_engine/vie_encoder.cc;h=2c877f9ff4ff372dd1149acdfb06867981fdaed5;hb=HEAD#l696

 681 #ifdef VIDEOCODEC_VP8
 682   if (vcm_.SendCodec() == webrtc::kVideoCodecVP8) {
 683     webrtc::CodecSpecificInfo codec_specific_info;
 684     codec_specific_info.codecType = webrtc::kVideoCodecVP8;
 685     codec_specific_info.codecSpecific.VP8.hasReceivedRPSI =
 686         has_received_rpsi_;
 687     codec_specific_info.codecSpecific.VP8.hasReceivedSLI =
 688         has_received_sli_;
 689     codec_specific_info.codecSpecific.VP8.pictureIdRPSI =
 690         picture_id_rpsi_;
 691     codec_specific_info.codecSpecific.VP8.pictureIdSLI  =
 692         picture_id_sli_;
 693     has_received_sli_ = false;
 694     has_received_rpsi_ = false;
 695 
 696     if (vcm_.AddVideoFrame(*decimated_frame,
 697                            vpm_.ContentMetrics(),
 698                            &codec_specific_info) != VCM_OK) {
 699       WEBRTC_TRACE(webrtc::kTraceError,
 700                    webrtc::kTraceVideo,
 701                    ViEId(engine_id_, channel_id_),
 702                    "%s: Error encoding frame %u", __FUNCTION__,
 703                    video_frame->timestamp());
 704     }
 705     return;
 706   }
 707 #endif
 708   if (vcm_.AddVideoFrame(*decimated_frame) != VCM_OK) {
 709     WEBRTC_TRACE(webrtc::kTraceError,
 710                  webrtc::kTraceVideo,
 711                  ViEId(engine_id_, channel_id_),
 712                  "%s: Error encoding frame %u", __FUNCTION__,
 713                  video_frame->timestamp());
 714   }
 715 }

As the content metrics is not used, it should be safe to disable the load manger complete for 2.0 or 2.1 with H.264 codec. This reduce CPU utilization by 2-3%. Can we disable the load manager through pref("media.navigator.load_adapt", false)?

Comment 5

[Gian-Carlo Pascutto [:gcp]]

>As the content metrics is not used, it should be safe to disable the load manger complete

That's quite a leap without further explanation.

But you are right:

AddVideoFrame will use NULL defaults for the missing arguments:
http://dxr.mozilla.org/mozilla-central/source/media/webrtc/trunk/webrtc/modules/video_coding/main/interface/video_coding.h#282

VideoSender will call MediaOpt::UpdateContentData with a null argument:
http://dxr.mozilla.org/mozilla-central/source/media/webrtc/trunk/webrtc/modules/video_coding/main/source/video_sender.cc#372

Which ends up disabling qm_select, which is what controls frame rate and resolution adaption:
http://dxr.mozilla.org/mozilla-central/source/media/webrtc/trunk/webrtc/modules/video_coding/main/source/media_optimization.cc#437

Now, it's possible that B2G doesn't have to care about CPU load because it uses a hardware H264 encoder, but the resolution/framerate adaption for the *available network bandwidth* is controlled from the same module.

So, there's at least 3 bugs here:
1) (New bug) Disable load_manager on B2G stuff that's about to ship and can't benefit from it.
2) (New bug) Fix B2G so it can use load_manager or at the very least qm_select to reduce the resolution on slow connections.
3) (What this bug is about) Optimize the content analysis code so it takes less CPU time itself.

Comment 6

[bhavana bajaj [:bajaj]]

jesup, this sounds like a meta bug to me where we are still investigating, not sure we should block here or any work is going to happen here. Can you help with some input.

Comment 7

[Randell Jesup [:jesup] (needinfo me)]

I think there are two simple, safe changes we can make that should block, so long as they pan out when tested:

1) the bug above (bug 1059765) - let the qm_select code process bitrate/load/etc adaptation
 * note: any OpenH264-specific issues from that bug would not block, and I'd spin them off to a separate bug.
2) (bug to be filed) disable the motion/spatial analysis on b2g (simply ifdef out the call), and hard-code the results to be "normal" for both (in a range of "low/normal/high").  (perhaps "high" for motion to create a preference for framerate over resolution).  This will let the qm_select code operate normally.

We can look to improve these and/or reenable content analysis (especially for vp8) in future revs.

Comment 8

[Jay]

(In reply to Gian-Carlo Pascutto [:gcp] from comment #5)
> >As the content metrics is not used, it should be safe to disable the load manger complete
> 
> That's quite a leap without further explanation.
> 
> But you are right:
> 
> AddVideoFrame will use NULL defaults for the missing arguments:
> http://dxr.mozilla.org/mozilla-central/source/media/webrtc/trunk/webrtc/
> modules/video_coding/main/interface/video_coding.h#282
> 
> VideoSender will call MediaOpt::UpdateContentData with a null argument:
> http://dxr.mozilla.org/mozilla-central/source/media/webrtc/trunk/webrtc/
> modules/video_coding/main/source/video_sender.cc#372
> 
> Which ends up disabling qm_select, which is what controls frame rate and
> resolution adaption:
> http://dxr.mozilla.org/mozilla-central/source/media/webrtc/trunk/webrtc/
> modules/video_coding/main/source/media_optimization.cc#437
> 
> Now, it's possible that B2G doesn't have to care about CPU load because it
> uses a hardware H264 encoder, but the resolution/framerate adaption for the
> *available network bandwidth* is controlled from the same module.
> 
> So, there's at least 3 bugs here:
> 1) (New bug) Disable load_manager on B2G stuff that's about to ship and
> can't benefit from it.
> 2) (New bug) Fix B2G so it can use load_manager or at the very least
> qm_select to reduce the resolution on slow connections.
> 3) (What this bug is about) Optimize the content analysis code so it takes
> less CPU time itself.

Thanks for the detail analysis.
Regarding #2, I thought the bitrate (instead of resolution) is adjusted to network bandwidth. Isn't it?

Comment 9

[Gian-Carlo Pascutto [:gcp]]

When the load manager is active, and content analysis data is present, the qm_select code will reduce the framerate or size (which of the two is dependent on the content analysis data), if the measured bitrate drops below a specific threshold.

Said differently, if you initialize the camera at 1080p60, and have a 200kbps upstream, the code will drop the resolution and framerate to get a better quality picture.

Comment 10

[:shell escalante]

Maire talked with Bhavana about moving this bug to 2.1+ to continue looking at ways to improve.  This is a meta bug for the issue and there are already 2 actionable breakdowns that are already landed (both 2.0+ already): bug 1059765 and bug 1060249.  There's a problem with one of the fixes which Randell is investigating/debuggin 9/3.

Comment 11

[bhavana bajaj [:bajaj]]

[Blocking Requested - why for this release]:

Comment 12

[Maire Reavy [:mreavy]]

> So, there's at least 3 bugs here:
> 1) (New bug) Disable load_manager on B2G stuff that's about to ship and
> can't benefit from it.
> 2) (New bug) Fix B2G so it can use load_manager or at the very least
> qm_select to reduce the resolution on slow connections.
> 3) (What this bug is about) Optimize the content analysis code so it takes
> less CPU time itself.

Bugs #1 and #2 have been addressed (Bug 1059765, Bug 1060249, and Bug 1063883) and will be uplifted into B2G v2.0.  With those bugs addressed, the remaining investigation for this bug is a longer term effort primarily focused on software codecs.