Closed Bug 1664010 Opened 4 years ago Closed 4 years ago

Audio samples (Opus) written to webm have corrupt sample start times.

Categories

(Core :: Audio/Video: Recording, defect, P2)

defect

Tracking

()

RESOLVED FIXED

People

(Reporter: akochendorfer, Assigned: pehrsons)

References

(Regressed 1 open bug)

Details

Attachments

(1 file)

Eric Burns

11:05 AM (2 hours ago)

to me, Luchin, Derek, Zac
Hi Andy,

Thanks for the help! We'd love to chat with your media developers about this one if they could spare a few minutes. Here's a write-up from one of our engineers:

Title:
Audio samples (Opus) written to webm have corrupt sample start times.

Details:
We have observed that Firefox 79 and 80 running on Windows 10, on certain hardware, will write audio samples with incorrect timestamps to webm files when recording audio from HTML5. Opus samples written to the webm container have incorrect timestamps but correct durations.

Repro steps:
We were not able to repro with some Win10 desktop machines. We were able to repro with Win10 running on Lenovo P51 laptop. The repro machine tested both with the internal mic and an external Logitech c930e device. Repro steps are as follows:

Browse to Panopto.it
Configure to capture audio (optionally can capture video)
Record for 10s
Click to download the webm after recording

Expected:

Webm is an a good format, can be played by any webm player

Actual:

Most players cannot play the webm but browsers can so long as a seek operation does not occur.
You can confirm there is an issue easily by running an ffmpeg command: ffmpeg -i {input webm} output.mp4
    An offending file will output warnings like "[mp4 @ 00000222dfc86380] Non-monotonous DTS in output stream 0:1; previous: 4168, current: 3896; changing to 4169. This may result in incorrect timestamps in the output file."
You can dump the sample information in ffprobe: ffprobe -show_frames -select_streams a:0 -of csv {webm media file} > output.csv
    If you look at the output csv you will see that the durations of the samples are correct but the sample start times are broken. In our repro, we observed the pkt start time to increase by ~6ms per packet while each packet had a duration value of ~20ms. 
    If you calculate the start time based on the duration of the samples then the start times will be correct.

If you have questions about this bug, please contact Derek Sessions at dsessions@panopto.com or Zac Rumford at zrumford@panopto.com. Thank you!

Andreas, do you understand why this would be hardware dependent ? It looks like the most standard and basic usage of MediaRecorder is trivially broken, and we might be disseminating invalid webm files.

Severity: -- → S2
Flags: needinfo?(apehrson)
Priority: -- → P1

Odd. I note that we set duration based on the actual number of frames the encoder pulled[1], whereas timestamps are based on incrementing a fixed packet size (20ms)[2].

But we also assert that these are the same [3].

I'll have to assume that for this report they've hit a path where that assert fails.

[1] https://searchfox.org/mozilla-central/rev/0c97a6410ff018c22e65a0cbe4e5f2ca4581b22e/dom/media/encoder/OpusTrackEncoder.cpp#370,373
[2] https://searchfox.org/mozilla-central/rev/0c97a6410ff018c22e65a0cbe4e5f2ca4581b22e/dom/media/encoder/OpusTrackEncoder.cpp#425-427
[3] https://searchfox.org/mozilla-central/rev/0c97a6410ff018c22e65a0cbe4e5f2ca4581b22e/dom/media/encoder/OpusTrackEncoder.cpp#388


Looking at the quoted error message the times in question are 4168, current: 3896. The only supported opus samplerate [5] for which these are divisible is 8000. This might not be the samplerate used because we can start at an offset, but I nevertheless gave it try, forcing the samplerate to 8000 by the pref "media.cubeb.force_sample_rate".

This gave me the error

Input #0, matroska,webm, from '/home/pehrsons/Downloads/panopto2.webm':
  Metadata:
    encoder         : QTmuxingAppLibWebM-0.0.1
  Duration: N/A, start: -0.001000, bitrate: N/A
    Stream #0:0(eng): Video: vp8, yuv420p(progressive), 1280x720, SAR 1:1 DAR 16:9, 60 tbr, 1k tbn, 1k tbc (default)
    Stream #0:1(eng): Audio: opus, 48000 Hz, stereo, fltp (default)
Stream mapping:
  Stream #0:0 -> #0:0 (vp8 (native) -> h264 (libx264))
  Stream #0:1 -> #0:1 (opus (native) -> aac (native))
Press [q] to stop, [?] for help
[aac @ 0x5592e0a64d80] Queue input is backward in time
    Last message repeated 7 times
Too many packets buffered for output stream 0:1.
[aac @ 0x5592e0a64d80] Qavg: 48085.945
[aac @ 0x5592e0a64d80] 2 frames left in the queue on closing
Conversion failed!

Not identical, but similar in that time goes backwards. What's weird here is that the samplerate for the stream is 48k, and nb_samples for packets is based on that too (they're 960 == 20ms). dts times however, are in chronological packet order [6, 2, 6, 9, 12, 16, 19, 22, etc.].

This seems completely wrong. At this point I'd say audio recording is broken when the output device (aka cubeb preferred samplerate) is 8k, 12k, 16k or 24k. The most common samplerates should be 44.1k and 48k so I'm reducing this to a P2. I'll reach out to the contacts to verify that this is the reason they're hitting issues.

[5] https://searchfox.org/mozilla-central/rev/0c97a6410ff018c22e65a0cbe4e5f2ca4581b22e/dom/media/encoder/OpusTrackEncoder.cpp#39-40

Assignee: nobody → apehrson
Status: NEW → ASSIGNED
Flags: needinfo?(apehrson)
Priority: P1 → P2
OS: Windows 10 → All

A couple of errors found in comment 3. I mentioned packet durations are based on 48k, but that was nb_samples. pkt_duration seems to be in milliseconds (20). The error there seems to be that dts time went from 6 back to 2. In that light, "completely wrong" is a bit of an overstatement.

The reported example (dts from 4168 to 3896) does not occur that early on in the stream either, so perhaps this analysis is barking up the wrong tree. I'll wait until I hear back from the panopto contacts.

One second is a bit short given that starting bluetooth input devices takes
several seconds on some platforms. Bug 1586370 may have worsened this since we
now process silence while waiting for an input to give us data.

I'm not sure this is the right fix for the issue reported. Testing on panopto's side was so far a bit inconclusive so I'm waiting to see if they have more chances to reproduce.

Keywords: leave-open
Pushed by pehrsons@gmail.com: https://hg.mozilla.org/integration/autoland/rev/b9d2b93a0c31 Increase the wait time before default-initing in AudioTrackEncoder. r=bryce

This was not the right fix for the reported issue (panopto could repro) but with bug 1667728 fixed they have not been able to repro again. All in all I consider this fixed.

Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Depends on: 1667728
Resolution: --- → FIXED
Regressions: 1699235
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: