1097524 - h264 codec can't work when setting mode 0

Status: RESOLVED WORKSFORME

(Core :: WebRTC: Audio/Video, defect, P2)

Description

[karina]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:33.0) Gecko/20100101 Firefox/33.0
Build ID: 20141030112145

Steps to reproduce:

testing on http://mozilla.github.io/webrtc-landing/pc_test.html
select "Required H.264 Video" and "Prefer H.264 Mode 0"


Actual results:

receiver can't see the video 


Expected results:

video can be seen correctly

Comment 1

[karina]

Attachment: multi-slice-support.txt

Comment 2

[karina]

Root cause: For mode 0, the first VCL NAL is considered as the last NAL in one frame, so other NALs can’t be sent to decoder.
solution: use one bit to indicate whether the current NAL is the last NAL in one frame.
in rtp_format_h264.cc line 96
      Old code:
       if (type == kSps || type == kPps ||
        type == kSei || type == kPrefix) {
      *last_packet   = false;
    } else {
      *last_packet   = true;
    }

   Updated code: using forbiddenzerobit to judge whether current NAL is the last NAL. 
      if (type == kSps || type == kPps ||
        type == kSei || type == kPrefix) {
      *last_packet   = false;
    } else {
      if(payload_data_[0]&kFBit)
         *last_packet   = true;
      else
         *last_packet   = false;
    }

Comment 3

[karina]

1. This is a sender side issue, the sender side doesn’t set the correct last packet flag.  One frame has several VCL NAL for Mode 0.  But the sender side sets the last packet flag in  each VCL NAL.
       2. When the receiver side receives the first VCL NAL, it thinks that the AU is complete and sends it to decoder, so the decoder can’t decode correctly.  The other VCL packets are discarded (but I don’t read which code do this,only from the trace )
  
    updated code on the sender side.
     1. In WebrtcGmpVideCodec.cpp, set last NAL flag in WebrtcGmpVideoEncoder::Encoded(),because only here can get this flag.  If it is last NAL, set forbidden bit as 1. Otherwise not set.
     2. In rtp_format_h264.cc, parser this bit to judge whether it is the last NAL and set *last_packet. And reset forbidden bit to keep the same data with encoder output.

Comment 4

[Maire Reavy [:mreavy]]

Randell -- Can you look at this?

Comment 5

[Ethan Hugg [:ehugg]]

Currently we negotiate Mode 0, but we do not send it.  A patch to the OpenH264 wrapper to fix this is here:
https://rbcommons.com/s/OpenH264/r/584/
You can also build it quickly from Karina's repo: 
https://github.com/ruil2/openh264/tree/max_nal_size

This change respects the maxPayloadSize in the GMP wrapper.  If you try it out, Mode 1 still works but Mode 0 will not because of the issues Karina has outlined above.  We're trying to figure out what we need to Fix in Firefox so we can start properly sending Mode 0, then we will need to coordinate a release of FF with an update to the OpenH264 GMP plugin.

Comment 6

[Ethan Hugg [:ehugg]]

Attachment: WebRTC Support for H264 Mode 0

Comment 7

[Ethan Hugg [:ehugg]]

Comment on attachment 8521681 [details] [diff] [review]
WebRTC Support for H264 Mode 0

Review of attachment 8521681 [details] [diff] [review]:
-----------------------------------------------------------------

Re-uploaded Karina's fix as a patch with couple format/style changes.

This makes Mode 0 work for loopback, but it's unclear to me that this improves our compatibility with other endpoints using Mode 0.

Comment 8

[Randell Jesup [:jesup] (needinfo me)]

Ethan - this patch is totally bit-rotted due to the webrtc_40 packetization changes.  Likely there's some small bug in the mode 0 support; from my reading the code currently *should* be setting the Marker bit correctly since we're feeding the data up using the FragmentationHeader structure.  I suspect the entire (or almost) current patch is moot.

Can you work with Jon Peter Davies (who posted on dev.media) to investigate this?  Thanks.

Comment 9

[Ethan Hugg [:ehugg]]

Yes, I will take another look.

Comment 10

[Ethan Hugg [:ehugg]]

This now works since the landing of bug 1162251