890419 - Standalone benchmark for video

Status: NEW

(Core :: WebRTC: Audio/Video, defect, P4)

Comment 1

[Eric Rescorla (:ekr)]

Attachment: Standalone benchmark for video

Comment 2

[Eric Rescorla (:ekr)]

The above is a WIP. I'm going to add a control script to automatically run through a bunch of frame rate/frame size variations.

Comment 3

[Eric Rescorla (:ekr)]

Attachment: Standalone benchmark for video

Comment 4

[Eric Rescorla (:ekr)]

Here is the output of this benchmark:

Resolution      Rate    Backlog
176x144 5       1
176x144 10      1
176x144 15      1
176x144 20      1
176x144 25      1
176x144 30      2
176x144 40      482
320x240 5       1
320x240 10      1
320x240 15      1
320x240 20      2
320x240 25      2
320x240 30      3
320x240 40      501
640x480 5       1
640x480 10      2
640x480 15      2
640x480 20      2
640x480 25      4
640x480 30      48
1280x720        5       190

Comment 5

[Eric Rescorla (:ekr)]

c4 refers to a peak.

Comment 6

[Eric Rescorla (:ekr)]

Attachment: Graph of benchmark data

Comment 7

[Eric Rescorla (:ekr)]

You will also need a YUV video of around 30-60 seconds (the benchmark loops).
I captured mine by using quicktime and then mplayer, i.e.,

mplayer -nosound -endpos 30 -vo yuv4mpeg /mnt/hgfs/ubuntu12-64-2/EKR-talking.mov

yuvscaler can be found in the mjpegtools pkg.

Comment 8

[Eric Rescorla (:ekr)]

Here is a different set of benchmarks run with the following Google clip:

http://downloads.webmproject.org/ietf_tests/niklas_1280_720_30.yuv.xz

Note the big difference at 640x480.

Resolution      Rate    Backlog
176x144 5       1
176x144 10      1
176x144 15      1
176x144 20      1
176x144 25      2
176x144 30      2
176x144 40      483
320x240 5       1
320x240 10      1
320x240 15      1
320x240 20      2
320x240 25      2
320x240 30      2
320x240 40      504
640x480 5       163
1280x720        5       261

Comment 9

[Eric Rescorla (:ekr)]

Just looking at top, this doesn't appear to be overloaded. More investigation needed.

Comment 10

[Suhas]

Command to convert from YUVMPEG to YUV format (from Ekr):
   mplayer -demuxer rawvideo -rawvideo w=1280:h=720:format=i420 niklas_1280_720_30.yuv -vo yuv4mpeg

Output will be ins stream.yuv

Comment 11

[Suhas]

Attachment: Extend Conduit with CodecObserver to dump Incoming and Outgoing Framerates

Comment 12

[Suhas]

Comment on attachment 772439 [details] [diff] [review]
Extend Conduit with CodecObserver to dump Incoming and Outgoing Framerates

A patch to allow WebRTC Video Engine to report Incoming and Outgoing Framerate/Bitrate every second.

Comment 13

[Suhas]

Based on the patch above, the following tests were ran with the benchmark.

Test 1: Run http://downloads.webmproject.org/ietf_tests/niklas_1280_720_30.yuv.xz with VideoEngine Frmerate set to 60 Fps 
Observation:
 a) Outgoing FrameRate reported was at 29 Fps (This indicates setting maxFramerate on Engine didn't get reflected) 
 b) Incoming framerate varied between 2 Fps to 6 Fps.
 c) WebRTC log reported Drop frames with a reason being frameRate. I don't have exact count on number of drop-frames due to log file wrap up.


Test 2: Run http://downloads.webmproject.org/ietf_tests/niklas_1280_720_30.yuv.xz with VideoEngine Framerate set to 30 Fps

Observation:
a) Outgoing FrameRate reported was at 29 Fps (Again, this indicates setting maxFramerate on Engine didn't get reflected) 
 b) Incoming framerate varied between 2 Fps to 27 Fps.
 c) WebRTC log reported Drop frames with a reason being bitRate. I don't have exact count on number of drop-frames due to log file wrap up.

  

Test 3: Created a video using Quicktime worth 60 seconds and generated YUV format with resolution 176*244. Run the benchmark with this video file. This has VideoEngine Framerate set to 60 Fps

Observation:
a) Outgoing FrameRate reported was at 29 Fps (This indicates setting maxFramerate on Engine didn't get reflected) 
 b) Incoming framerate varied between 2 Fps to 9 Fps.
 c) WebRTC log reported Drop frames with a reason being bitrate. I don't have exact count on number of drop-frames due to log file wrap up.


Test 4: Same as Test 3 but with VideoEngine Framerate set to 30 Fps

Observation:
a) Outgoing FrameRate reported was at 18 - 29 Fps (This indicates setting maxFramerate on Engine didn't get reflected) 
 b) Incoming framerate varied between 3 Fps to 11 Fps.
 c) WebRTC log reported Drop frames with a reason being bitrate. I don't have exact count on number of drop-frames due to log file wrap up.


With this, more investigation is needed based on the clues from above.

Comment 14

[Suhas]

Attachment: Extend Conduit with CodecObserver to dump Incoming and Outgoing Framerates

Comment 15

[Eric Rescorla (:ekr)]

The benchmark wants MPEG-wrapper YUV and not raw YUV. You can make one with mplayer like so:

mplayer -demuxer rawvideo -rawvideo w=1280:h=720:format=i420 inputfile.yuv -vo yuv4mpeg

Adjust the parameters as appropriate. This will stuff the output in stream.yuv

Comment 16

[Suhas]

Hmm .. Interesting .. I had this flag turned off few days back and that didn't improve the backlogs observed on the Mac. 

I was seeing 2 places where the frames/packets were dropped.
One that applies to encoder logic of dropping frame on onyxif.c  and second one was with the frameDropper. In the earlier case dropping of the frame ends up in no encoded bitstream to be sent and hence framerate on the receiver was low. But the second one happens eventually where the leaky bucket adjusts its value so that the consecutive frames get dropped. Even this would affect the incoming framerate on the receiver.

Comment 17

[Suhas]

Also the above were observed for 176 * 144 and 30 FPS. The results were worst for the 640 * 480 cases though. 

Probably my test setup is wrong or something on my side. Not sure either.

Comment 18

[Ben Brittain (:bbrittain)]

Attachment: Rate vs Backlog

Comment 19

[Ben Brittain (:bbrittain)]

Attachment: CPU vs Backlog

Comment 20

[Ben Brittain (:bbrittain)]

The prior two attachments are representative of my results with the Peak.

How to reproduce,
Use a fresh copy of M-C (tip is 141367:ad0ae007aa9e at time of comment)
apply patch 'benchmark.patch' (this has everything needed with the exception of the video)
build & flash B2G
push video_benchmark, located in the dist/bin of your objdir
run run-benchmark.py (matplotlib required for graph generation)

Comment 21

[Ben Brittain (:bbrittain)]

Attachment: benchmark.patch

Comment 22

[Ben Brittain (:bbrittain)]

Attachment: Rate vs Backlog - v2

Comment 23

[Ben Brittain (:bbrittain)]

Attachment: CPU vs Backlog -v2

Comment 24

[Josh Aas]

(In reply to Ben Brittain (:bbrittain) from comment #21)
> Created attachment 786021 [details] [diff] [review]
> benchmark.patch

Can you please explain this attachment? Does this obsolete something else or add in some way?

Comment 25

[StevenLee[:slee]]

Attachment: video-benchmark.patch

Hi Ekr,

The patch is derived from bbrittain's. I change the policy of benchmark. My version is try to estimate the maximum capability of the device. I change some code in VPMVideoDecimator and VideoConduit. This is because
1. the input fps may be more than 30
2. the encoded bitrate may exceed the default maximum

Here is the output on unagi.
Resolution@Rate FPS
176x144@60      55.411500
320x240@60      21.547800
640x480@60      6.538580
1280x720@60     2.595920

Comment 26

[StevenLee[:slee]]

(In reply to StevenLee[:slee] from comment #25)
> The patch is derived from bbrittain's. I change the policy of benchmark. My
> version is try to estimate the maximum capability of the device. I change
> some code in VPMVideoDecimator and VideoConduit. This is because
> 1. the input fps may be more than 30
> 2. the encoded bitrate may exceed the default maximum
Some more details about the patch.
1. The previous version controls the input fps and makes it as 5, 15, 30, etc. But my patch keeps inputting the frames whenever the previous frame is encoded.
2. When inputting a frame by SendVideoFrame, it just copies the frame, wakes up the encoder thread and returns. For making input-frame thread and encoder thread cooperate synchronously, I check marker bit in Transport::SendRtpPacket. When marker bit is true, it can notify the input-frame thread to insert the next frame. If not, some frames may be over-written because GIPS uses only 2 buffers.

Comment 27

[Eric Rescorla (:ekr)]

Comment on attachment 8357025 [details] [diff] [review]
video-benchmark.patch

Review of attachment 8357025 [details] [diff] [review]:
-----------------------------------------------------------------

Is this still relevant? If so, please re-request review.

Comment 28

[Bulk Bug Changes for mreavy's org]

Mass change P3->P4 to align with new Mozilla triage process.

Comment 29

[BugBot [:suhaib / :marco/ :calixte]]

The bug assignee is inactive on Bugzilla, so the assignee is being reset.

Comment 30

[Suhaib Mujahid [:suhaib]]

Sorry, there was a problem with the detection of inactive users. I'm reverting the change.

Comment 31

[BugBot [:suhaib / :marco/ :calixte]]

The bug assignee is inactive on Bugzilla, so the assignee is being reset.