Open Bug 890419 Opened 7 years ago Updated 3 years ago

Standalone benchmark for video

Categories

(Core :: WebRTC: Audio/Video, defect, P4)

defect

Tracking

()

Blocking Flags:

People

(Reporter: ekr, Assigned: ekr)

References

(Blocks 1 open bug)

Details

Attachments

(9 files, 2 obsolete files)

No description provided.
Attached patch Standalone benchmark for video (obsolete) — Splinter Review
The above is a WIP. I'm going to add a control script to automatically run through a bunch of frame rate/frame size variations.
Attachment #771519 - Attachment is obsolete: true
Here is the output of this benchmark:

Resolution      Rate    Backlog
176x144 5       1
176x144 10      1
176x144 15      1
176x144 20      1
176x144 25      1
176x144 30      2
176x144 40      482
320x240 5       1
320x240 10      1
320x240 15      1
320x240 20      2
320x240 25      2
320x240 30      3
320x240 40      501
640x480 5       1
640x480 10      2
640x480 15      2
640x480 20      2
640x480 25      4
640x480 30      48
1280x720        5       190
c4 refers to a peak.
Attached image Graph of benchmark data
You will also need a YUV video of around 30-60 seconds (the benchmark loops).
I captured mine by using quicktime and then mplayer, i.e.,

mplayer -nosound -endpos 30 -vo yuv4mpeg /mnt/hgfs/ubuntu12-64-2/EKR-talking.mov

yuvscaler can be found in the mjpegtools pkg.
Here is a different set of benchmarks run with the following Google clip:

http://downloads.webmproject.org/ietf_tests/niklas_1280_720_30.yuv.xz

Note the big difference at 640x480.

Resolution      Rate    Backlog
176x144 5       1
176x144 10      1
176x144 15      1
176x144 20      1
176x144 25      2
176x144 30      2
176x144 40      483
320x240 5       1
320x240 10      1
320x240 15      1
320x240 20      2
320x240 25      2
320x240 30      2
320x240 40      504
640x480 5       163
1280x720        5       261
Just looking at top, this doesn't appear to be overloaded. More investigation needed.
Command to convert from YUVMPEG to YUV format (from Ekr):
   mplayer -demuxer rawvideo -rawvideo w=1280:h=720:format=i420 niklas_1280_720_30.yuv -vo yuv4mpeg

Output will be ins stream.yuv
Comment on attachment 772439 [details] [diff] [review]
Extend Conduit with CodecObserver to dump Incoming and Outgoing Framerates

A patch to allow WebRTC Video Engine to report Incoming and Outgoing Framerate/Bitrate every second.
Based on the patch above, the following tests were ran with the benchmark.

Test 1: Run http://downloads.webmproject.org/ietf_tests/niklas_1280_720_30.yuv.xz with VideoEngine Frmerate set to 60 Fps 
Observation:
 a) Outgoing FrameRate reported was at 29 Fps (This indicates setting maxFramerate on Engine didn't get reflected) 
 b) Incoming framerate varied between 2 Fps to 6 Fps.
 c) WebRTC log reported Drop frames with a reason being frameRate. I don't have exact count on number of drop-frames due to log file wrap up.


Test 2: Run http://downloads.webmproject.org/ietf_tests/niklas_1280_720_30.yuv.xz with VideoEngine Framerate set to 30 Fps

Observation:
a) Outgoing FrameRate reported was at 29 Fps (Again, this indicates setting maxFramerate on Engine didn't get reflected) 
 b) Incoming framerate varied between 2 Fps to 27 Fps.
 c) WebRTC log reported Drop frames with a reason being bitRate. I don't have exact count on number of drop-frames due to log file wrap up.

  

Test 3: Created a video using Quicktime worth 60 seconds and generated YUV format with resolution 176*244. Run the benchmark with this video file. This has VideoEngine Framerate set to 60 Fps

Observation:
a) Outgoing FrameRate reported was at 29 Fps (This indicates setting maxFramerate on Engine didn't get reflected) 
 b) Incoming framerate varied between 2 Fps to 9 Fps.
 c) WebRTC log reported Drop frames with a reason being bitrate. I don't have exact count on number of drop-frames due to log file wrap up.


Test 4: Same as Test 3 but with VideoEngine Framerate set to 30 Fps

Observation:
a) Outgoing FrameRate reported was at 18 - 29 Fps (This indicates setting maxFramerate on Engine didn't get reflected) 
 b) Incoming framerate varied between 3 Fps to 11 Fps.
 c) WebRTC log reported Drop frames with a reason being bitrate. I don't have exact count on number of drop-frames due to log file wrap up.


With this, more investigation is needed based on the clues from above.
Attachment #772439 - Attachment is obsolete: true
Assignee: ekr → snandaku
Assignee: snandaku → ekr
The benchmark wants MPEG-wrapper YUV and not raw YUV. You can make one with mplayer like so:

mplayer -demuxer rawvideo -rawvideo w=1280:h=720:format=i420 inputfile.yuv -vo yuv4mpeg

Adjust the parameters as appropriate. This will stuff the output in stream.yuv
Blocks: 896229
Hmm .. Interesting .. I had this flag turned off few days back and that didn't improve the backlogs observed on the Mac. 

I was seeing 2 places where the frames/packets were dropped.
One that applies to encoder logic of dropping frame on onyxif.c  and second one was with the frameDropper. In the earlier case dropping of the frame ends up in no encoded bitstream to be sent and hence framerate on the receiver was low. But the second one happens eventually where the leaky bucket adjusts its value so that the consecutive frames get dropped. Even this would affect the incoming framerate on the receiver.
Also the above were observed for 176 * 144 and 30 FPS. The results were worst for the 640 * 480 cases though. 

Probably my test setup is wrong or something on my side. Not sure either.
The prior two attachments are representative of my results with the Peak.

How to reproduce,
Use a fresh copy of M-C (tip is 141367:ad0ae007aa9e at time of comment)
apply patch 'benchmark.patch' (this has everything needed with the exception of the video)
build & flash B2G
push video_benchmark, located in the dist/bin of your objdir
run run-benchmark.py (matplotlib required for graph generation)
(In reply to Ben Brittain (:bbrittain) from comment #21)
> Created attachment 786021 [details] [diff] [review]
> benchmark.patch

Can you please explain this attachment? Does this obsolete something else or add in some way?
Blocks: 923364
Hi Ekr,

The patch is derived from bbrittain's. I change the policy of benchmark. My version is try to estimate the maximum capability of the device. I change some code in VPMVideoDecimator and VideoConduit. This is because
1. the input fps may be more than 30
2. the encoded bitrate may exceed the default maximum

Here is the output on unagi.
Resolution@Rate FPS
176x144@60      55.411500
320x240@60      21.547800
640x480@60      6.538580
1280x720@60     2.595920
Attachment #8357025 - Flags: feedback?(ekr)
(In reply to StevenLee[:slee] from comment #25)
> The patch is derived from bbrittain's. I change the policy of benchmark. My
> version is try to estimate the maximum capability of the device. I change
> some code in VPMVideoDecimator and VideoConduit. This is because
> 1. the input fps may be more than 30
> 2. the encoded bitrate may exceed the default maximum
Some more details about the patch.
1. The previous version controls the input fps and makes it as 5, 15, 30, etc. But my patch keeps inputting the frames whenever the previous frame is encoded.
2. When inputting a frame by SendVideoFrame, it just copies the frame, wakes up the encoder thread and returns. For making input-frame thread and encoder thread cooperate synchronously, I check marker bit in Transport::SendRtpPacket. When marker bit is true, it can notify the input-frame thread to insert the next frame. If not, some frames may be over-written because GIPS uses only 2 buffers.
No longer blocks: 923364
backlog: --- → webRTC+
Rank: 35
Priority: -- → P3
Comment on attachment 8357025 [details] [diff] [review]
video-benchmark.patch

Review of attachment 8357025 [details] [diff] [review]:
-----------------------------------------------------------------

Is this still relevant? If so, please re-request review.
Attachment #8357025 - Flags: feedback?(ekr)
Mass change P3->P4 to align with new Mozilla triage process.
Priority: P3 → P4
You need to log in before you can comment on or make changes to this bug.