1024357 - There is no video in webRTC session between FxOS device (openC, HW decoder) and openh264 enabled browser (windows laptop)

Status: RESOLVED INVALID

(Core :: WebRTC: Audio/Video, defect)

Description

[Oscar Patiño González]

Attachment: log.txt

It seems that HW OMX decoder is not being initilized, as I get a NULL pointer. There is no video being displayed from my desktop browser into the openC device. Adding a log file as attachment

Comment 1

[Randell Jesup [:jesup] (needinfo me)]

Right now to enable it you need the patchset at https://hg.mozilla.org/users/rjesup_wgate.com/h264_omx

Note that testing it on a JB-based device such as ver v10G-2 on Flame will have major delay; we need KitKat-based base builds with updated firmaware to resolve the delay, which isn't ready yet.

Comment 2

[Randell Jesup [:jesup] (needinfo me)]

This also looks odd: configuring encoder 240x320 @ 244 fps, rate 300 kbps, though that might be a consequence of the failure.

Please try on a flame with v10G-2 (or later).

Comment 3

[John Lin [:jhlin][:jolin]]

Configuration failure, perhaps?

Oscar, could you please check your log and see if there was any ACodec error message like below? Thanks.

06-10 05:22:31.137  3163  3296 E ACodec  : Encoder could not be configured to emit SPS/PPS before IDR frames. (err -2147483648)

Comment 4

[Oscar Patiño González]

Attachment: ACodec logcat capture

Comment 5

[Oscar Patiño González]

Attachment: just for completeness, it is the same as logs but from start to finish

Comment 6

[Oscar Patiño González]

I think that my configuration is ok, but jesup's bitrate and framerate constraints were not applied. I have applied them but I got the same result. John, I have uploaded an ACodec log file, but couldn't see anything related with decoder. Just the encoder shows.
It seems that encoder is working but video from phone doesn't display on my laptop. I'm actually more focused on decoder right now, but maybe I should open another ticket.

Problem seems to be in WebrtcOMXH264VideoDecoder::Decode, function WebrtcOMXDecoder::ExtractPicDimensions don't get a valid resolution and mOMX is always a NULL pointer. Checked input frame and both buffer and length seems ok, but couldn't get dimensions info:

E/WebrtcOMXH264VideoCodec( 4208): Prepending SPS/PPS: 21 bytes, timestamp 3955846030, captureTimeMs 3712676480
E/WebrtcOMXH264VideoCodec( 4208): omxdecoder aInputImage._buffer 0xb047b000, aInputImage._length 732
E/WebrtcOMXH264VideoCodec( 4208): omxdecoder picture dimensions width: -1796805682 , height : -1320784648
E/WebrtcOMXH264VideoCodec( 4208): omxdecoder aInputImage._buffer 0xb047b000, aInputImage._length 780
E/WebrtcOMXH264VideoCodec( 4208): omxdecoder picture dimensions width: -1796803072 , height : -1320784624
E/WebrtcOMXH264VideoCodec( 4208): Encode frame: 240x320, timestamp 3955857010 (28865815), renderTimeMs 2430046909
E/WebrtcOMXH264VideoCodec( 4208): OMX: encoded frame (552): time 28865815 (2597923), flags x0
E/WebrtcOMXH264VideoCodec( 4208): Encoded frame: 552 bytes, 240x320, is_param 0, is_iframe 0, timestamp 3955857010, captureTimeMs 77582960
E/WebrtcOMXH264VideoCodec( 4208): omxdecoder aInputImage._buffer 0xb047b000, aInputImage._length 1030
E/WebrtcOMXH264VideoCodec( 4208): omxdecoder picture dimensions width: -1796800192 , height : -1320784600
E/WebrtcOMXH264VideoCodec( 4208): omxdecoder aInputImage._buffer 0xb047b000, aInputImage._length 1029
E/WebrtcOMXH264VideoCodec( 4208): omxdecoder picture dimensions width: -1796800192 , height : -1320784576
E/WebrtcOMXH264VideoCodec( 4208): omxdecoder aInputImage._buffer 0xb047b000, aInputImage._length 902
E/WebrtcOMXH264VideoCodec( 4208): omxdecoder picture dimensions width: -1796797582 , height : -1320784552
E/WebrtcOMXH264VideoCodec( 4208): omxdecoder aInputImage._buffer 0xb047b000, aInputImage._length 901
E/WebrtcOMXH264VideoCodec( 4208): omxdecoder picture dimensions width: -1796797582 , height : -1320784528
E/WebrtcOMXH264VideoCodec( 4208): omxdecoder aInputImage._buffer 0xb047b000, aInputImage._length 780
E/WebrtcOMXH264VideoCodec( 4208): omxdecoder picture dimensions width: -1796794792 , height : -1320784504

It seems that decoder is not receiving SPS/PPS and cannot figure out picture size. I don't know if there has been changes in openh264 plugin, but if any, I didn't apply them. My browser version has been always been working and I didn't make any change on it.

Comment 7

[John Lin [:jhlin][:jolin]]

(In reply to Oscar Patiño González from comment #6)
> I think that my configuration is ok, but jesup's bitrate and framerate
> constraints were not applied. I have applied them but I got the same result.
> John, I have uploaded an ACodec log file, but couldn't see anything related
> with decoder. Just the encoder shows.
> It seems that encoder is working but video from phone doesn't display on my
> laptop. I'm actually more focused on decoder right now, but maybe I should
> open another ticket.
> 
> Problem seems to be in WebrtcOMXH264VideoDecoder::Decode, function
> WebrtcOMXDecoder::ExtractPicDimensions don't get a valid resolution and mOMX
> is always a NULL pointer. Checked input frame and both buffer and length
> seems ok, but couldn't get dimensions info:
> 
> E/WebrtcOMXH264VideoCodec( 4208): Prepending SPS/PPS: 21 bytes, timestamp
> 3955846030, captureTimeMs 3712676480
> E/WebrtcOMXH264VideoCodec( 4208): omxdecoder aInputImage._buffer 0xb047b000,
> aInputImage._length 732
> E/WebrtcOMXH264VideoCodec( 4208): omxdecoder picture dimensions width:
> -1796805682 , height : -1320784648
> E/WebrtcOMXH264VideoCodec( 4208): omxdecoder aInputImage._buffer 0xb047b000,
> aInputImage._length 780
> E/WebrtcOMXH264VideoCodec( 4208): omxdecoder picture dimensions width:
> -1796803072 , height : -1320784624
> E/WebrtcOMXH264VideoCodec( 4208): Encode frame: 240x320, timestamp
> 3955857010 (28865815), renderTimeMs 2430046909
> E/WebrtcOMXH264VideoCodec( 4208): OMX: encoded frame (552): time 28865815
> (2597923), flags x0
> E/WebrtcOMXH264VideoCodec( 4208): Encoded frame: 552 bytes, 240x320,
> is_param 0, is_iframe 0, timestamp 3955857010, captureTimeMs 77582960
> E/WebrtcOMXH264VideoCodec( 4208): omxdecoder aInputImage._buffer 0xb047b000,
> aInputImage._length 1030
> E/WebrtcOMXH264VideoCodec( 4208): omxdecoder picture dimensions width:
> -1796800192 , height : -1320784600
> E/WebrtcOMXH264VideoCodec( 4208): omxdecoder aInputImage._buffer 0xb047b000,
> aInputImage._length 1029
> E/WebrtcOMXH264VideoCodec( 4208): omxdecoder picture dimensions width:
> -1796800192 , height : -1320784576
> E/WebrtcOMXH264VideoCodec( 4208): omxdecoder aInputImage._buffer 0xb047b000,
> aInputImage._length 902
> E/WebrtcOMXH264VideoCodec( 4208): omxdecoder picture dimensions width:
> -1796797582 , height : -1320784552
> E/WebrtcOMXH264VideoCodec( 4208): omxdecoder aInputImage._buffer 0xb047b000,
> aInputImage._length 901
> E/WebrtcOMXH264VideoCodec( 4208): omxdecoder picture dimensions width:
> -1796797582 , height : -1320784528
> E/WebrtcOMXH264VideoCodec( 4208): omxdecoder aInputImage._buffer 0xb047b000,
> aInputImage._length 780
> E/WebrtcOMXH264VideoCodec( 4208): omxdecoder picture dimensions width:
> -1796794792 , height : -1320784504
> 
> It seems that decoder is not receiving SPS/PPS and cannot figure out picture
> size. I don't know if there has been changes in openh264 plugin, but if any,
> I didn't apply them. My browser version has been always been working and I
> didn't make any change on it.

 Randy, does this sound like the problem you met when working on bug 996379? Thanks!

Comment 8

[Randy Lin [:rlin] - Be Mozillian on 2015/01/26]

I checked the connectivity between flame/desktop on today's central build.
H.264 video codec works well on both side. 
(By checking the OMXOutputDrain thread is alive through top -t)
Also pass the test on open_c.

Could you provide what reversion of gecko do you use?

Comment 9

[Oscar Patiño González]

I have tested my source with a previous gecko version (May 15th) and h264 bitstream is being decoded on the phone. Also, I get video coming from the phone decoded on my laptop. So there is no issue on browser.

I have captured a bitstream and it is a bit weird, because I'm receiving a wrongly formatted bitstream. I'm attaching it for you to have a look. As you can see an additional byte is introduced between the start NAL separator and the byte containing nal type info every time a nal is received, note 0x02 between 0x67 and 0x01:

00  00  00  01  02  67  42  00  1E  8C  8D  50  50  1E  88  00  
00  00  01  68  CE  3C  80  00  00  00  01  65  B8  00  04  00  
00  0C  E1  4C  34  84  07  

This snippet is the SPS PPS and first IDR, I suppose that they are sent in the same RTP packet, because after that, every nal shows 02 after the 00 00 00 01  code:

00  00  00  01  02  61  E0  00  40  00  9C  80  97  E7  28  B8  EE  FF  
20  4C  F9  72  E2  BF  66  E8  A8  47  

Here, again, 0x02 appears.
Now I suspect rtp unpacking is doing something weird??

Comment 10

[Oscar Patiño González]

Attachment: wrongly formatted h264 bitstream capture feeded to decoder

Comment 11

[Oscar Patiño González]

going to test again the May 15th version, It seems that my setup was wrong and only played vp8. Sorry guys!

Comment 12

[Oscar Patiño González]

Confirmed, revision from May 15th is working correctly. Current gecko version don't.

Comment 13

[John Lin [:jhlin][:jolin]]

May I know which desktop/browser version you're using?
At this moment OpenH264 support is not fully done (tracked under bug 948160) and the old Github repo we used for testing doesn't work well with current central build. And according to rlin, he uses a local browser build for his tests between phone and desktop.

Comment 14

[Oscar Patiño González]

I'm using a custom nightly version, it is a customized build with openh264 support.

Comment 15

[John Lin [:jhlin][:jolin]]

Payload in RTP packet (|payload_data| in [1]) should not contain start code. I think they will be inserted in [2] before sending to decoder. Could you please help check that on your Open C build?

[1] http://dxr.mozilla.org/mozilla-central/source/media/webrtc/trunk/webrtc/modules/rtp_rtcp/source/rtp_receiver_video.cc#226

[2] http://dxr.mozilla.org/mozilla-central/source/media/webrtc/trunk/webrtc/modules/video_coding/main/source/session_info.cc#141

Comment 16

[Oscar Patiño González]

Yes, I know, openh264 bitstreams don't include the sync bytes. I have captured the RTP packets received at phone for both version (the May version and the current one).

Both streams has the same structure, of course they are different but both have a 0x02 byte right before NAL starts. The problem is that in May's version this 0x02 byte is not considered as part of the h264 bitstream, while in current gecko version it does.

Also I suspect there might be a bug when adding the sync bytes to h264 bitstream, since I have found this in [1] that the first IDR seems to be splitted in two rtp packets but you can see that there's been inserted a sync code. (Watch it at 0x447 bytes offset). This is incorrect, 00 00 00 01 cannot be part of the stream unless it is a sync code, (and if so prevention bytes 0x03 are inserted), and there is no NAL starting byte there. 

I'm uploading both RTP captures in case you want to check what I've said.

[1] https://bugzilla.mozilla.org/attachment.cgi?id=8440616

Comment 17

[Oscar Patiño González]

Attachment: videodump_current_gecko.rtp

This file is rtp dump from the current gecko version.

Comment 18

[Oscar Patiño González]

Attachment: videodump_may_gecko.rtp

This file is an rtp dump from a May gecko version. In this version  h264 decoding is working

Comment 19

[John Lin [:jhlin][:jolin]]

(In reply to Oscar Patiño González from comment #16)
> Yes, I know, openh264 bitstreams don't include the sync bytes. I have
> captured the RTP packets received at phone for both version (the May version
> and the current one).

 Thanks!

> 
> Both streams has the same structure, of course they are different but both
> have a 0x02 byte right before NAL starts. The problem is that in May's
> version this 0x02 byte is not considered as part of the h264 bitstream,
> while in current gecko version it does.
> 
> Also I suspect there might be a bug when adding the sync bytes to h264
> bitstream, since I have found this in [1] that the first IDR seems to be
> splitted in two rtp packets but you can see that there's been inserted a
> sync code. (Watch it at 0x447 bytes offset). This is incorrect, 00 00 00 01
> cannot be part of the stream unless it is a sync code, (and if so prevention
> bytes 0x03 are inserted), and there is no NAL starting byte there. 
> 
> I'm uploading both RTP captures in case you want to check what I've said.
> 
> [1] https://bugzilla.mozilla.org/attachment.cgi?id=8440616

 The extra start code suggests that browser didn't send the RTP packets current central code expects (single NALU or FU-A). Could it be your custom nightly browser build out of date?

 I built (on Mac) Firefox using latest central source and OpenH264 plugin from [1]. It seems work fine with phone.

[1] https://github.com/ekr/openh264/tree/plugin4-merge

Comment 20

[Randell Jesup [:jesup] (needinfo me)]

[08:05]	jesup|laptop	https://github.com/ethanhugg/openh264/tree/plugin4-merge-gmpdir
[08:06]	jesup	opatino: Are you using the above branch to build?
[08:06]	jesup	With that, "make plugin".
[08:06]	jesup	that should work