1423253 - Kill NotifyPull for video tracks

Status: RESOLVED FIXED

(Core :: Audio/Video: MediaStreamGraph, enhancement, P3)

Description

[Andreas Pehrson [:pehrsons]]

Without durations for video chunks there's no need to know the duration of a frame before appending it to a track in the graph - we can just append when a new frame is available.

With durations and NotifyPull on track listeners in place we can switch all video producers relying on NotifyPull (canvas capture, peer connection, MediaEngineRemoteVideoSource) to be purely push-based, and remove NotifyPull() calls for video altogether.

Comment 1

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1423253 - Use a timer to make sure we keep sending video over a peer connection when there's no input. r?dminor

The webrtc-pc spec says:

If track is ended, or if the track's output is disabled, i.e. the track is
disabled and/or muted, the RTCRtpSender MUST send silence (audio),
black frames (video) or a zero-information-content equivalent.
In the case of video, the RTCRtpSender SHOULD send one black frame per second.

This patch covers this case, and the case when no frames reach the
MediaPipeline, for both direct and non-direct video listeners.

Comment 2

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1423253 - Throttle sending video frames by age instead of queue size. r?padenot

Comment 3

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1423253 - Change CanvasCaptureMediaStream's driver to push instead of pull. r?padenot

Comment 4

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1423253 - Change MediaEngineRemoteVideoSource to push instead of pull. r?padenot

Comment 5

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1423253 - Add a drift compensator to reclock video timestamps. r?padenot

Comment 6

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1423253 - Add drift compensation to MediaEncoder. r?padenot

Comment 7

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1423253 - Automatically progress a pushed video track to avoid blocking its stream. r?padenot

Comment 8

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1423253 - Remove support for direct audio listeners from AudioTrackEncoder. r?padenot

Comment 9

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1423253 - Route MediaEncoder::Suspend/Resume through the graph thread for less drift. r?padenot

Comment 10

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1423253 - Disregard VideoChunk durations in VideoTrackEncoder. r?padenot

Comment 11

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1423253 - Remove pulling from MediaEngineTabVideoSource and fix thread safety. r?padenot

Comment 12

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1423253 - Remove pulling (and mutex) from the default video source. r?padenot

Comment 13

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1423253 - Remove pulling (and mutex) from MediaPipelineReceiveVideo. r?padenot

Comment 14

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1423253 - Remove durations from VideoSegment::AppendFrame. r?padenot

VideoSegments still have durations, and they are still needed by the
MediaStreamGraph as it shuffles MediaSegments around.
They do not have a say in the wall-clock duration of video frames however.
Removing this should prevent any producers starting to add video chunks with
durations in the future.

Comment 15

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1423253 - Make a VideoTrackEncoder gtest more explicit. r?padenot

Comment 16

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1423253 - Disallow pulling of video. r?padenot

Comment 17

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1423253 - Remove unused SourceMediaStream::GetEndOfAppendedData. r?padenot

Comment 18

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1423253 - Disallow direct track listeners for audio tracks. r?padenot

Comment 19

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1423253 - Remove superfluous character from test_mr_pause_resume_video.html. r?jib

Comment 20

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1423253 - Pace future video frames in MediaPipeline. r?dminor!, r?padenot!

WebRTC.org doesn't handle receiving multiple future frames.
This will buffer future frames from a direct listener in MediaPipeline
and pass them on when the frame's wall clock timestamp has been reached.

Comment 21

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1423253 - Notify MediaStreamTrackListeners about tracks' enabled state. r?padenot

Comment 22

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1423253 - Remove DirectMediaStreamTrackListener::mMedia. r?padenot

It is unnecessary.

Comment 23

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1423253 - Move VideoSegment-specific logic out of VideoFrameContainer. r?padenot

It is only used in one place. Can just as well be inlined there.

Comment 24

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1423253 - Make future frames in a VideoFrameContainer black when a track is disabled. r?padenot

Comment 25

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1423253 - Make future frames in MediaPipeline black when a track is disabled. r?padenot

Comment 26

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1423253 - Improve MediaPipeline logging and fix formatting. r?dminor

Comment 27

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1423253 - Make future frames in VideoTrackEncoder black when a track is disabled. r?padenot

Comment 28

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1423253 - Fix test_mr_record_addtracked_stream.html recording a video that is too short. r?padenot

Comment 29

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1423253 - Put DecodedStreamData::mEOSVideoCompensation on the stack. r?padenot

Comment 30

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1423253 - Handle MediaDecoder pauses when future frames are already buffered. r?padenot

Comment 31

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1423253 - Remove mMutex from VideoFrameConverter. r?dminor

Comment 32

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1423253 - Drop old frames before they get processed in VideoFrameConverter. r?padenot

To avoid building up a queue of frames when the machine cannot keep up.

Comment 34

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1423253 - Move VideoFrameConverter to its own file. r?dminor

This lets us unit test it in a future patch.

Comment 35

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1423253 - Move YUVBufferGenerator into its own files so it can be used by other media gtests. r?dminor

Comment 36

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/35977d5dcf5c
Remove mMutex from VideoFrameConverter. r=dminor
https://hg.mozilla.org/integration/autoland/rev/d69f006bc09e
Use a timer to make sure we keep sending video over a peer connection when there's no input. r=dminor
https://hg.mozilla.org/integration/autoland/rev/9e13d07d7617
Change CanvasCaptureMediaStream's driver to push instead of pull. r=padenot
https://hg.mozilla.org/integration/autoland/rev/54b4c9399b81
Change MediaEngineRemoteVideoSource to push instead of pull. r=padenot
https://hg.mozilla.org/integration/autoland/rev/60d215858649
Add a drift compensator to reclock video timestamps. r=padenot
https://hg.mozilla.org/integration/autoland/rev/9478255cfe12
Add drift compensation to MediaEncoder. r=padenot
https://hg.mozilla.org/integration/autoland/rev/3de6381be3d9
Automatically progress a pushed video track to avoid blocking its stream. r=padenot
https://hg.mozilla.org/integration/autoland/rev/61a4762b830d
Remove support for direct audio listeners from AudioTrackEncoder. r=padenot
https://hg.mozilla.org/integration/autoland/rev/9e6eddfb4ef8
Route MediaEncoder::Suspend/Resume through the graph thread for less drift. r=padenot
https://hg.mozilla.org/integration/autoland/rev/2f7f881b7ede
Disregard VideoChunk durations in VideoTrackEncoder. r=padenot
https://hg.mozilla.org/integration/autoland/rev/e2a57da64b18
Remove pulling from MediaEngineTabVideoSource and fix thread safety. r=padenot
https://hg.mozilla.org/integration/autoland/rev/75a1569247b8
Remove pulling (and mutex) from the default video source. r=padenot
https://hg.mozilla.org/integration/autoland/rev/d8af2cceb3c8
Remove pulling (and mutex) from MediaPipelineReceiveVideo. r=padenot
https://hg.mozilla.org/integration/autoland/rev/387b32b7e55d
Remove durations from VideoSegment::AppendFrame. r=padenot
https://hg.mozilla.org/integration/autoland/rev/ba0778323644
Make a VideoTrackEncoder gtest more explicit. r=padenot
https://hg.mozilla.org/integration/autoland/rev/a838c502c66a
Disallow pulling of video. r=padenot
https://hg.mozilla.org/integration/autoland/rev/f8cf618572c6
Remove unused SourceMediaStream::GetEndOfAppendedData. r=padenot
https://hg.mozilla.org/integration/autoland/rev/d802933bc25e
Disallow direct track listeners for audio tracks. r=padenot
https://hg.mozilla.org/integration/autoland/rev/ffa939521c30
Remove superfluous character from test_mr_pause_resume_video.html. r=jib
https://hg.mozilla.org/integration/autoland/rev/2a52a7ee6bfc
Move VideoFrameConverter to its own file. r=dminor
https://hg.mozilla.org/integration/autoland/rev/5fdd6c8a0d2f
Move YUVBufferGenerator into its own files so it can be used by other media gtests. r=dminor
https://hg.mozilla.org/integration/autoland/rev/b5e813887d3f
Pace future video frames in MediaPipeline. r=dminor,padenot
https://hg.mozilla.org/integration/autoland/rev/1437c364fe99
Drop old frames before they get processed in VideoFrameConverter. r=padenot
https://hg.mozilla.org/integration/autoland/rev/c608a0db0063
Notify MediaStreamTrackListeners about tracks' enabled state. r=padenot
https://hg.mozilla.org/integration/autoland/rev/30552dd7fbfc
Remove DirectMediaStreamTrackListener::mMedia. r=padenot
https://hg.mozilla.org/integration/autoland/rev/81d85b902990
Move VideoSegment-specific logic out of VideoFrameContainer. r=padenot
https://hg.mozilla.org/integration/autoland/rev/f37f5f1fc1e7
Make future frames in a VideoFrameContainer black when a track is disabled. r=padenot
https://hg.mozilla.org/integration/autoland/rev/9019e6f0a6c3
Make future frames in MediaPipeline black when a track is disabled. r=padenot
https://hg.mozilla.org/integration/autoland/rev/9f3a6410605a
Improve MediaPipeline logging and fix formatting. r=dminor
https://hg.mozilla.org/integration/autoland/rev/7153764aded8
Make future frames in VideoTrackEncoder black when a track is disabled. r=padenot
https://hg.mozilla.org/integration/autoland/rev/e4a2813727b5
Fix test_mr_record_addtracked_stream.html recording a video that is too short. r=padenot
https://hg.mozilla.org/integration/autoland/rev/56569d42dd80
Put DecodedStreamData::mEOSVideoCompensation on the stack. r=padenot
https://hg.mozilla.org/integration/autoland/rev/ab393db34aa5
Handle MediaDecoder pauses when future frames are already buffered. r=padenot

Comment 37

[Paul Adenot (:padenot)]

Attachment: Bug 1423253 - Unbreak MINGW by defining WEBRTC_WIN manually.

Comment 38

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c0cf1a5594aa
Unbreak MINGW by defining WEBRTC_WIN manually.

Comment 39

[Paul Adenot (:padenot)]

Attachment: Bug 1423253 - Really unbreak MINGW builds for video TestVideoFrameConverter, by not building it when WebRTC is disabled.

Comment 40

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b6940071e197
Really unbreak MINGW builds for video TestVideoFrameConverter, by not building it when WebRTC is disabled.

Comment 41

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/35977d5dcf5c
https://hg.mozilla.org/mozilla-central/rev/d69f006bc09e
https://hg.mozilla.org/mozilla-central/rev/9e13d07d7617
https://hg.mozilla.org/mozilla-central/rev/54b4c9399b81
https://hg.mozilla.org/mozilla-central/rev/60d215858649
https://hg.mozilla.org/mozilla-central/rev/9478255cfe12
https://hg.mozilla.org/mozilla-central/rev/3de6381be3d9
https://hg.mozilla.org/mozilla-central/rev/61a4762b830d
https://hg.mozilla.org/mozilla-central/rev/9e6eddfb4ef8
https://hg.mozilla.org/mozilla-central/rev/2f7f881b7ede
https://hg.mozilla.org/mozilla-central/rev/e2a57da64b18
https://hg.mozilla.org/mozilla-central/rev/75a1569247b8
https://hg.mozilla.org/mozilla-central/rev/d8af2cceb3c8
https://hg.mozilla.org/mozilla-central/rev/387b32b7e55d
https://hg.mozilla.org/mozilla-central/rev/ba0778323644
https://hg.mozilla.org/mozilla-central/rev/a838c502c66a
https://hg.mozilla.org/mozilla-central/rev/f8cf618572c6
https://hg.mozilla.org/mozilla-central/rev/d802933bc25e
https://hg.mozilla.org/mozilla-central/rev/ffa939521c30
https://hg.mozilla.org/mozilla-central/rev/2a52a7ee6bfc
https://hg.mozilla.org/mozilla-central/rev/5fdd6c8a0d2f
https://hg.mozilla.org/mozilla-central/rev/b5e813887d3f
https://hg.mozilla.org/mozilla-central/rev/1437c364fe99
https://hg.mozilla.org/mozilla-central/rev/c608a0db0063
https://hg.mozilla.org/mozilla-central/rev/30552dd7fbfc
https://hg.mozilla.org/mozilla-central/rev/81d85b902990
https://hg.mozilla.org/mozilla-central/rev/f37f5f1fc1e7
https://hg.mozilla.org/mozilla-central/rev/9019e6f0a6c3
https://hg.mozilla.org/mozilla-central/rev/9f3a6410605a
https://hg.mozilla.org/mozilla-central/rev/7153764aded8
https://hg.mozilla.org/mozilla-central/rev/e4a2813727b5
https://hg.mozilla.org/mozilla-central/rev/56569d42dd80
https://hg.mozilla.org/mozilla-central/rev/ab393db34aa5
https://hg.mozilla.org/mozilla-central/rev/c0cf1a5594aa
https://hg.mozilla.org/mozilla-central/rev/b6940071e197