[MediaRecorder] How to handle AudioChunks without channel counts or VideoChunks without size?

NEW
Unassigned

Status

()

P3
normal
Rank:
25
5 years ago
2 years ago

People

(Reporter: bechen, Unassigned)

Tracking

unspecified
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

5 years ago
Fork from bug 991776 comment 56, comment 57.

roc:
If we get one of these AudioChunks containing no channels (silence), the encoder should still initialize and encode silence.

rlin:
For this case, what's the channel number should we use? 
ie, if OpusTrackEncoder init the opus encoder as channel = 1, then audio chunk start to feed the the stereo data, should we make the stereo PCM data as mono?
Or init the opus encoder as channel = 2 and make mono data as stereo?

== 
Similar to VideoChunk, if the VideoChunk was called SetNull(), the videoEncoder counld not initialize either.
(Reporter)

Comment 1

5 years ago
(In reply to Benjamin Chen [:bechen] from comment #0)
> Fork from bug 991776 comment 56, comment 57.
> 
> roc:
> If we get one of these AudioChunks containing no channels (silence), the
> encoder should still initialize and encode silence.
> 
> rlin:
> For this case, what's the channel number should we use? 
> ie, if OpusTrackEncoder init the opus encoder as channel = 1, then audio
> chunk start to feed the the stereo data, should we make the stereo PCM data
> as mono?
> Or init the opus encoder as channel = 2 and make mono data as stereo?
> 
> == 
> Similar to VideoChunk, if the VideoChunk was called SetNull(), the
> videoEncoder counld not initialize either.

Here comes up with some roughly ideas...
1. Dynamic changing the channel counts or video resolution
- as rlin says, need to consider the muxer/encoder abilities and the blob had been sent to JS.

2. Store some "encode information" in each MediaStream
- Once the MediaStream is attached to MSG, we can query the information from Source MediaStream.

3. Re-design the SetNull() function in VideoChunk and AudioChunk.
- Force the silence audio chunks have enough information for encoder. Either the black screen for video chunks.

How do you think?
(Reporter)

Updated

5 years ago
Blocks: 1064535
Note: we've seen cases where players get confused or fail to play after a resolution change, so avoiding that could be a win.

Perhaps in some cases it might be possible to defer encoding until the first non-Null frame to determine either the number of tracks or the resolution?  (and if an entire slice is Null, just choose something and then encode it)
backlog: --- → webrtc/webaudio+
Rank: 25
Priority: -- → P2
Mass change P2->P3 to align with new Mozilla triage process.
Priority: P2 → P3
You need to log in before you can comment on or make changes to this bug.