Closed Bug 1730020 Opened 5 months ago Closed 4 months ago

Add a mechanism to measure the media performance by recording the passed time on different stages

Categories

(Core :: Audio/Video: Playback, task, P2)

task

Tracking

()

RESOLVED FIXED
95 Branch
Tracking Status
firefox95 --- fixed

People

(Reporter: alwu, Assigned: alwu)

References

(Blocks 2 open bugs)

Details

Attachments

(4 files, 2 obsolete files)

In order to get one decoded data, we need to go through the following processes

  This represents the different stages that a media data will go through
   within the playback journey.
   
              |---|                      |---|           |------|
              Copy Demuxed            Copy Demuxed        Copy Decoded
              Data                      Data                Video
     |------------- |      |-----------------------------------|
        Request Demux         Request Decode
    |-----------------------------------------------------------|
      Request Data
   
    RequestData : Record the time where MediaDecoderStateMachine(MDSM) starts
    asking for a decoded data to MDSM receives a decoded data.
   
    RequestDemux : Record the time where MediaFormatReader(MFR) starts asking
    a demuxed sample to MFR received a demuxed sample. This stage is a sub-
    stage of RequestData.
   
    CopyDemuxedData : On some situations, we will need to copy the demuxed
    data, which is still not decoded yet so its size is still small. This
    records the time which we spend on copying data. This stage could happen
    multiple times, either being a sub-stage of RequestDemux (in MSE case), or
    being a sub-stage of RequestDecode (when sending data via IPC).
   
    RequestDecode : Record the time where MFR starts asking decoder to return
    a decoded data to MFR receives a decoded data. As the decoder might be
    remote, this stage might include the time spending on IPC trips. This stage
    is a sub-stage of RequestData.
   
    CopyDecodedVideo : If we can't reuse same decoder texture to the
    compositor, then we have to copy video data to to another sharable texture.
    This records the time which we spend on copying data. This stage is a sub-
    stage of RequestDecode.

Therefore, I would like to record the passed time on each stage to know how we can improve the overall performance. Although the critical part would definitely be the decoding part, we still want to know how long it takes. It's also good for us to have recorded data if we want to do the comparasion after we put improvement on the media pipeline.

Depends on D125140

Attachment #9240439 - Attachment description: Bug 1730020 - part1 : implement a class to record passed time for each for stages in the media pipeline. → Bug 1730020 - part1 : implement a class to record passed time for each stages in the media pipeline.
Attachment #9240439 - Attachment description: Bug 1730020 - part1 : implement a class to record passed time for each stages in the media pipeline. → Bug 1730020 - part1 : implement a class to record passed time for each for stages in the media pipeline.

I'm not sure if you're wanting to track samples through decoders, but FWIW I've looked at this a little before and have seen some approaches for libav and windows decoders:

  • For libavcodec the output AVFrame has a pkt_dts which should be copied from the input AVPacket to the associated output frame. This can be used to associate inputs to outputs.
  • For the Windows decoders, I've seen programs hack the input times so that instead of storing a time, they store an ID. Then on output that ID is used to match inputs to outputs, and also used to lookup the original presentation time for that frame.
Attachment #9240439 - Attachment description: Bug 1730020 - part1 : implement a class to record passed time for each for stages in the media pipeline. → Bug 1730020 - part1 : implement a new class to record the time which each stage of media pinepline spends on.

Depends on D126148

Depends on D126389

Attachment #9240440 - Attachment description: Bug 1730020 - part2 : report average result to telemetry. → Bug 1730020 - part2 : repord the average time for each stage.
Attachment #9242156 - Attachment description: Bug 1730020 - part3 : add profiler markers. → Bug 1730020 - part3 : add profiler markers to show additional information.
Attachment #9242592 - Attachment description: Bug 1730020 - part5 : add gtest. → Bug 1730020 - part4 : add gtest.
Attachment #9242730 - Attachment description: Bug 1730020 - part6 : remove unused variable 'videoDecodeStartTime'. → Bug 1730020 - part5 : remove unused variable 'videoDecodeStartTime'.
Attachment #9242537 - Attachment is obsolete: true
CopyDecodedVideo : If we can't reuse same decoder texture to the
compositor, then we have to copy video data to to another sharable texture.
This records the time which we spend on copying data. This stage is a sub-
stage of RequestDecode.

We might need to be careful about this data. This does not directly connected to GPU power consumption. GPU power consumption increases when GPU have more tasks. On windows, software video decoding seemed to consume more power than hardware video decoding from Bug 1733282. Removing video copy with hardware video decoder is connected to use YUV SwapChain as Overlay on Windows.

Attachment #9242156 - Attachment description: Bug 1730020 - part3 : add profiler markers to show additional information. → Bug 1730020 - part2 : add profiler markers to show additional information.
Attachment #9242592 - Attachment description: Bug 1730020 - part4 : add gtest. → Bug 1730020 - part3 : add gtest.
Attachment #9242730 - Attachment description: Bug 1730020 - part5 : remove unused variable 'videoDecodeStartTime'. → Bug 1730020 - part4 : remove unused variable 'videoDecodeStartTime'.
Attachment #9240440 - Attachment is obsolete: true
Pushed by alwu@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/8799f103c043
part1 : implement a new class to record the time which each stage of media pinepline spends on. r=bryce,padenot
https://hg.mozilla.org/integration/autoland/rev/085fb78aac06
part2 : add profiler markers to show additional information. r=padenot
https://hg.mozilla.org/integration/autoland/rev/b57ce3e0f39f
part3 : add gtest. r=padenot
https://hg.mozilla.org/integration/autoland/rev/580203e4b76a
part4 : remove unused variable 'videoDecodeStartTime'. r=bryce
Flags: needinfo?(alwu)
Pushed by alwu@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/be3ba5198e9b
part1 : implement a new class to record the time which each stage of media pinepline spends on. r=bryce,padenot
https://hg.mozilla.org/integration/autoland/rev/f61112a4def4
part2 : add profiler markers to show additional information. r=padenot
https://hg.mozilla.org/integration/autoland/rev/a49969885ed8
part3 : add gtest. r=padenot
https://hg.mozilla.org/integration/autoland/rev/78046ce4a00d
part4 : remove unused variable 'videoDecodeStartTime'. r=bryce
Regressions: 1734327
You need to log in before you can comment on or make changes to this bug.