Standalone benchmark for video

NEW
Assigned to

Status

()

Core
WebRTC: Audio/Video
P4
normal
Rank:
35
5 years ago
10 months ago

People

(Reporter: ekr, Assigned: ekr)

Tracking

(Blocks: 2 bugs)

unspecified
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(9 attachments, 2 obsolete attachments)

Comment hidden (empty)
(Assignee)

Comment 1

5 years ago
Created attachment 771519 [details] [diff] [review]
Standalone benchmark for video
(Assignee)

Comment 2

5 years ago
The above is a WIP. I'm going to add a control script to automatically run through a bunch of frame rate/frame size variations.
(Assignee)

Comment 3

5 years ago
Created attachment 771671 [details] [diff] [review]
Standalone benchmark for video
(Assignee)

Updated

5 years ago
Attachment #771519 - Attachment is obsolete: true
(Assignee)

Comment 4

5 years ago
Here is the output of this benchmark:

Resolution      Rate    Backlog
176x144 5       1
176x144 10      1
176x144 15      1
176x144 20      1
176x144 25      1
176x144 30      2
176x144 40      482
320x240 5       1
320x240 10      1
320x240 15      1
320x240 20      2
320x240 25      2
320x240 30      3
320x240 40      501
640x480 5       1
640x480 10      2
640x480 15      2
640x480 20      2
640x480 25      4
640x480 30      48
1280x720        5       190
(Assignee)

Comment 5

5 years ago
c4 refers to a peak.
(Assignee)

Comment 6

5 years ago
Created attachment 771675 [details]
Graph of benchmark data
(Assignee)

Comment 7

5 years ago
You will also need a YUV video of around 30-60 seconds (the benchmark loops).
I captured mine by using quicktime and then mplayer, i.e.,

mplayer -nosound -endpos 30 -vo yuv4mpeg /mnt/hgfs/ubuntu12-64-2/EKR-talking.mov

yuvscaler can be found in the mjpegtools pkg.
(Assignee)

Comment 8

5 years ago
Here is a different set of benchmarks run with the following Google clip:

http://downloads.webmproject.org/ietf_tests/niklas_1280_720_30.yuv.xz

Note the big difference at 640x480.

Resolution      Rate    Backlog
176x144 5       1
176x144 10      1
176x144 15      1
176x144 20      1
176x144 25      2
176x144 30      2
176x144 40      483
320x240 5       1
320x240 10      1
320x240 15      1
320x240 20      2
320x240 25      2
320x240 30      2
320x240 40      504
640x480 5       163
1280x720        5       261
(Assignee)

Comment 9

5 years ago
Just looking at top, this doesn't appear to be overloaded. More investigation needed.

Comment 10

5 years ago
Command to convert from YUVMPEG to YUV format (from Ekr):
   mplayer -demuxer rawvideo -rawvideo w=1280:h=720:format=i420 niklas_1280_720_30.yuv -vo yuv4mpeg

Output will be ins stream.yuv

Comment 11

5 years ago
Created attachment 772439 [details] [diff] [review]
Extend Conduit with CodecObserver to dump Incoming and Outgoing Framerates

Comment 12

5 years ago
Comment on attachment 772439 [details] [diff] [review]
Extend Conduit with CodecObserver to dump Incoming and Outgoing Framerates

A patch to allow WebRTC Video Engine to report Incoming and Outgoing Framerate/Bitrate every second.

Comment 13

5 years ago
Based on the patch above, the following tests were ran with the benchmark.

Test 1: Run http://downloads.webmproject.org/ietf_tests/niklas_1280_720_30.yuv.xz with VideoEngine Frmerate set to 60 Fps 
Observation:
 a) Outgoing FrameRate reported was at 29 Fps (This indicates setting maxFramerate on Engine didn't get reflected) 
 b) Incoming framerate varied between 2 Fps to 6 Fps.
 c) WebRTC log reported Drop frames with a reason being frameRate. I don't have exact count on number of drop-frames due to log file wrap up.


Test 2: Run http://downloads.webmproject.org/ietf_tests/niklas_1280_720_30.yuv.xz with VideoEngine Framerate set to 30 Fps

Observation:
a) Outgoing FrameRate reported was at 29 Fps (Again, this indicates setting maxFramerate on Engine didn't get reflected) 
 b) Incoming framerate varied between 2 Fps to 27 Fps.
 c) WebRTC log reported Drop frames with a reason being bitRate. I don't have exact count on number of drop-frames due to log file wrap up.

  

Test 3: Created a video using Quicktime worth 60 seconds and generated YUV format with resolution 176*244. Run the benchmark with this video file. This has VideoEngine Framerate set to 60 Fps

Observation:
a) Outgoing FrameRate reported was at 29 Fps (This indicates setting maxFramerate on Engine didn't get reflected) 
 b) Incoming framerate varied between 2 Fps to 9 Fps.
 c) WebRTC log reported Drop frames with a reason being bitrate. I don't have exact count on number of drop-frames due to log file wrap up.


Test 4: Same as Test 3 but with VideoEngine Framerate set to 30 Fps

Observation:
a) Outgoing FrameRate reported was at 18 - 29 Fps (This indicates setting maxFramerate on Engine didn't get reflected) 
 b) Incoming framerate varied between 3 Fps to 11 Fps.
 c) WebRTC log reported Drop frames with a reason being bitrate. I don't have exact count on number of drop-frames due to log file wrap up.


With this, more investigation is needed based on the clues from above.

Comment 14

5 years ago
Created attachment 772499 [details] [diff] [review]
Extend Conduit with CodecObserver to dump Incoming and Outgoing Framerates

Updated

5 years ago
Attachment #772439 - Attachment is obsolete: true

Updated

5 years ago
Assignee: ekr → snandaku

Updated

5 years ago
Assignee: snandaku → ekr
(Assignee)

Comment 15

5 years ago
The benchmark wants MPEG-wrapper YUV and not raw YUV. You can make one with mplayer like so:

mplayer -demuxer rawvideo -rawvideo w=1280:h=720:format=i420 inputfile.yuv -vo yuv4mpeg

Adjust the parameters as appropriate. This will stuff the output in stream.yuv
Blocks: 896229

Comment 16

5 years ago
Hmm .. Interesting .. I had this flag turned off few days back and that didn't improve the backlogs observed on the Mac. 

I was seeing 2 places where the frames/packets were dropped.
One that applies to encoder logic of dropping frame on onyxif.c  and second one was with the frameDropper. In the earlier case dropping of the frame ends up in no encoded bitstream to be sent and hence framerate on the receiver was low. But the second one happens eventually where the leaky bucket adjusts its value so that the consecutive frames get dropped. Even this would affect the incoming framerate on the receiver.

Comment 17

5 years ago
Also the above were observed for 176 * 144 and 30 FPS. The results were worst for the 640 * 480 cases though. 

Probably my test setup is wrong or something on my side. Not sure either.

Updated

5 years ago
Blocks: 861050
Created attachment 786010 [details]
Rate vs Backlog
Created attachment 786011 [details]
CPU vs Backlog
The prior two attachments are representative of my results with the Peak.

How to reproduce,
Use a fresh copy of M-C (tip is 141367:ad0ae007aa9e at time of comment)
apply patch 'benchmark.patch' (this has everything needed with the exception of the video)
build & flash B2G
push video_benchmark, located in the dist/bin of your objdir
run run-benchmark.py (matplotlib required for graph generation)
Created attachment 786027 [details]
Rate vs Backlog - v2
Created attachment 786030 [details]
CPU vs Backlog -v2

Comment 24

5 years ago
(In reply to Ben Brittain (:bbrittain) from comment #21)
> Created attachment 786021 [details] [diff] [review]
> benchmark.patch

Can you please explain this attachment? Does this obsolete something else or add in some way?

Comment 25

5 years ago
Created attachment 8357025 [details] [diff] [review]
video-benchmark.patch

Hi Ekr,

The patch is derived from bbrittain's. I change the policy of benchmark. My version is try to estimate the maximum capability of the device. I change some code in VPMVideoDecimator and VideoConduit. This is because
1. the input fps may be more than 30
2. the encoded bitrate may exceed the default maximum

Here is the output on unagi.
Resolution@Rate FPS
176x144@60      55.411500
320x240@60      21.547800
640x480@60      6.538580
1280x720@60     2.595920
Attachment #8357025 - Flags: feedback?(ekr)

Comment 26

5 years ago
(In reply to StevenLee[:slee] from comment #25)
> The patch is derived from bbrittain's. I change the policy of benchmark. My
> version is try to estimate the maximum capability of the device. I change
> some code in VPMVideoDecimator and VideoConduit. This is because
> 1. the input fps may be more than 30
> 2. the encoded bitrate may exceed the default maximum
Some more details about the patch.
1. The previous version controls the input fps and makes it as 5, 15, 30, etc. But my patch keeps inputting the frames whenever the previous frame is encoded.
2. When inputting a frame by SendVideoFrame, it just copies the frame, wakes up the encoder thread and returns. For making input-frame thread and encoder thread cooperate synchronously, I check marker bit in Transport::SendRtpPacket. When marker bit is true, it can notify the input-frame thread to insert the next frame. If not, some frames may be over-written because GIPS uses only 2 buffers.
No longer blocks: 923364

Updated

3 years ago
backlog: --- → webRTC+
Rank: 35
Priority: -- → P3
(Assignee)

Comment 27

3 years ago
Comment on attachment 8357025 [details] [diff] [review]
video-benchmark.patch

Review of attachment 8357025 [details] [diff] [review]:
-----------------------------------------------------------------

Is this still relevant? If so, please re-request review.
Attachment #8357025 - Flags: feedback?(ekr)
Mass change P3->P4 to align with new Mozilla triage process.
Priority: P3 → P4
You need to log in before you can comment on or make changes to this bug.