1034471 - [MediaRecorder] How to handle AudioChunks without channel counts or VideoChunks without size?

Status: NEW

(Core :: Audio/Video: Recording, defect, P3)

Description

[Benjamin Chen [:bechen]]

Fork from bug 991776 comment 56, comment 57.

roc:
If we get one of these AudioChunks containing no channels (silence), the encoder should still initialize and encode silence.

rlin:
For this case, what's the channel number should we use? 
ie, if OpusTrackEncoder init the opus encoder as channel = 1, then audio chunk start to feed the the stereo data, should we make the stereo PCM data as mono?
Or init the opus encoder as channel = 2 and make mono data as stereo?

== 
Similar to VideoChunk, if the VideoChunk was called SetNull(), the videoEncoder counld not initialize either.

Comment 1

[Benjamin Chen [:bechen]]

(In reply to Benjamin Chen [:bechen] from comment #0)
> Fork from bug 991776 comment 56, comment 57.
> 
> roc:
> If we get one of these AudioChunks containing no channels (silence), the
> encoder should still initialize and encode silence.
> 
> rlin:
> For this case, what's the channel number should we use? 
> ie, if OpusTrackEncoder init the opus encoder as channel = 1, then audio
> chunk start to feed the the stereo data, should we make the stereo PCM data
> as mono?
> Or init the opus encoder as channel = 2 and make mono data as stereo?
> 
> == 
> Similar to VideoChunk, if the VideoChunk was called SetNull(), the
> videoEncoder counld not initialize either.

Here comes up with some roughly ideas...
1. Dynamic changing the channel counts or video resolution
- as rlin says, need to consider the muxer/encoder abilities and the blob had been sent to JS.

2. Store some "encode information" in each MediaStream
- Once the MediaStream is attached to MSG, we can query the information from Source MediaStream.

3. Re-design the SetNull() function in VideoChunk and AudioChunk.
- Force the silence audio chunks have enough information for encoder. Either the black screen for video chunks.

How do you think?

Comment 2

[Maire Reavy [:mreavy]]

Note: we've seen cases where players get confused or fail to play after a resolution change, so avoiding that could be a win.

Perhaps in some cases it might be possible to defer encoding until the first non-Null frame to determine either the number of tracks or the resolution?  (and if an entire slice is Null, just choose something and then encode it)

Comment 3

[Bulk Bug Changes for mreavy's org]

Mass change P2->P3 to align with new Mozilla triage process.