1154213 - Same timestamps in audio recorded with media recorder api (in audio+video mode)(with vorbis)

Status: RESOLVED FIXED

(Core :: Audio/Video: Recording, defect, P2)

Description

[Stéphane Schoorens]

User Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:37.0) Gecko/20100101 Firefox/37.0
Build ID: 20150403141925

Steps to reproduce:

Recording a stream (audio + video) with the media recorder api
var session = {
        audio: true,
        video: {
            mozMediaSource: source || "screen",
            mediaSource: source || "screen"
        }
    };

    navigator.getUserMedia(session, 
        function (stream) {
            var mediaRecorder = new MediaRecorder(stream);
            mediaRecorder.start();
        });


Actual results:

timestamps in the audio track are the same each seconds.

Cmd to analyse it:
gst-launch-0.10 filesrc location=record.webm ! decodebin ! audioconvert ! identity ! alsasink -v -m
Result:
/GstPipeline:pipeline0/GstIdentity:identity0: last-message = "chain   ******* (identity0:sink) (4096 bytes, timestamp: 0:00:00.000000000, duration: 0:00:00.021333333, offset: -1, offset_end: -1, flags: 0 ) 0x7efef00130e0"
/GstPipeline:pipeline0/GstIdentity:identity0: last-message = "chain   ******* (identity0:sink) (4096 bytes, timestamp: 0:00:00.000000000, duration: 0:00:00.021333333, offset: -1, offset_end: -1, flags: 0 ) 0x7efef00738f0"
/GstPipeline:pipeline0/GstIdentity:identity0: last-message = "chain   ******* (identity0:sink) (4096 bytes, timestamp: 0:00:00.000000000, duration: 0:00:00.021333333, offset: -1, offset_end: -1, flags: 0 ) 0x7efef00738f0"
/GstPipeline:pipeline0/GstIdentity:identity0: last-message = "chain   ******* (identity0:sink) (4096 bytes, timestamp: 0:00:00.000000000, duration: 0:00:00.021333333, offset: -1, offset_end: -1, flags: 0 ) 0x7efef0096040"
/GstPipeline:pipeline0/GstIdentity:identity0: last-message = "chain   ******* (identity0:sink) (4096 bytes, timestamp: 0:00:00.000000000, duration: 0:00:00.021333333, offset: -1, offset_end: -1, flags: 0 ) 0x7efef0085210"
/GstPipeline:pipeline0/GstIdentity:identity0: last-message = "chain   ******* (identity0:sink) (4096 bytes, timestamp: 0:00:00.000000000, duration: 0:00:00.021333333, offset: -1, offset_end: -1, flags: 0 ) 0x7efef00855d0"
/GstPipeline:pipeline0/GstIdentity:identity0: last-message = "chain   ******* (identity0:sink) (4096 bytes, timestamp: 0:00:00.000000000, duration: 0:00:00.021333333, offset: -1, offset_end: -1, flags: 0 ) 0x7efef0085210"
/GstPipeline:pipeline0/GstIdentity:identity0: last-message = "chain   ******* (identity0:sink) (4096 bytes, timestamp: 0:00:00.000000000, duration: 0:00:00.021333333, offset: -1, offset_end: -1, flags: 0 ) 0x7efef0085210"
/GstPipeline:pipeline0/GstIdentity:identity0: last-message = "chain   ******* (identity0:sink) (4096 bytes, timestamp: 0:00:01.000000000, duration: 0:00:00.021333333, offset: -1, offset_end: -1, flags: 0 ) 0x7efef0096540"
/GstPipeline:pipeline0/GstIdentity:identity0: last-message = "chain   ******* (identity0:sink) (4096 bytes, timestamp: 0:00:01.000000000, duration: 0:00:00.021333333, offset: -1, offset_end: -1, flags: 0 ) 0x7efef0096180"
/GstPipeline:pipeline0/GstIdentity:identity0: last-message = "chain   ******* (identity0:sink) (4096 bytes, timestamp: 0:00:01.000000000, duration: 0:00:00.021333333, offset: -1, offset_end: -1, flags: 0 ) 0x7efef0085490"
/GstPipeline:pipeline0/GstIdentity:identity0: last-message = "chain   ******* (identity0:sink) (4096 bytes, timestamp: 0:00:01.000000000, duration: 0:00:00.021333333, offset: -1, offset_end: -1, flags: 0 ) 0x7efef00b7190"
/GstPipeline:pipeline0/GstIdentity:identity0: last-message = "chain   ******* (identity0:sink) (4096 bytes, timestamp: 0:00:01.000000000, duration: 0:00:00.021333333, offset: -1, offset_end: -1, flags: 0 ) 0x7efef00130e0"
/GstPipeline:pipeline0/GstIdentity:identity0: last-message = "chain   ******* (identity0:sink) (4096 bytes, timestamp: 0:00:01.000000000, duration: 0:00:00.021333333, offset: -1, offset_end: -1, flags: 0 ) 0x7efef0096680"


Expected results:

timestamps need to be set on each audio frame. this didn't happen with an audio only record.

Comment 1

[Stéphane Schoorens]

This also happen while recording camera+audio instead of screen+audio

Comment 2

[vkrishnamoorthy]

Andrew should this be filed under Web API?

Comment 3

[Andrew Overholt [:overholt]]

(In reply to [e-mail] [:Vishy] from comment #2)
> Andrew should this be filed under Web API?

Let's try Media.

Comment 4

[Guang-De Lin]

Attachment: Handle timestamps of video/webm vorbis track encoding.

Audio-only recording is audio/ogg with opus encoding.
Audio frame duration will be treated correctly in OpusTrackEncoder/OggWriter.

video/webm recording is vp8/vorbis while audio track timecode is not handled.
My patch tries to write timestamp in VorbisTrackEncoder,
and insert timecode of audio frames in EbmlComposer.

With my patch the recorded webm files can be played normally with sound by VLC.
Without it the recorded files will be played by VLC silently. 

Hello Maire,
I don't know exactly who could be the reviewer.
Do you have any idea? Thanks.

Comment 5

[Maire Reavy [:mreavy]]

Comment on attachment 8688817 [details] [diff] [review]
Handle timestamps of video/webm vorbis track encoding.

Review of attachment 8688817 [details] [diff] [review]:
-----------------------------------------------------------------

This looks good to me.  It make sense to have a second review from any of the following: jesup, roc, or jya.

Comment 6

[Maire Reavy [:mreavy]]

Randell -- If you have time today, can you review this?

Comment 7

[Maire Reavy [:mreavy]]

From my perspective, we're good to land this.

Comment 8

[Guang-De Lin]

Attachment: Handle timestamps of video/webm vorbis track encoding. r=mreavy, r=rjesup

carry r+ from mreavy and rjesup.

try submission:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=6662c7085516

Comment 9

[Pulsebot]

https://hg.mozilla.org/integration/mozilla-inbound/rev/ac214b95a784

Comment 10

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/ac214b95a784

Comment 11

[Stéphane Schoorens]

Amazing =p. Thanks all