(In reply to Andri Yngvason from comment #0) > I've implemented a WebCodecs based H.264 decoder for noVNC: https://github.com/novnc/noVNC/pull/1876 Hi Andri, Could you create a test page and upload it here, or create a minimal test environment with steps-to-reproduce, maybe? It would greatly help us diagnose the issue. > After a few frames have been decoded, the error callback is called and decoding stops. We are seeing errors like these on the command line: > ``` > [RDD 15365: MediaPDecoder #1]: D/FFmpegVideo FFMPEG: Frame decode takes 0.35 ms average decode time 1.45 ms frame duration 0.00 average frame duration 0.00 decoded 12 frames > [RDD 15365: MediaPDecoder #1]: D/FFmpegVideo FFMPEG: slow decode: failed to decode in time (decoded late 12) > ``` > What appears to be happening here is that Firefox doesn't receive a `duration` parameter with each encoded frame chunk, and sets the `duration` to zero instead of disabling the "slow decode" code path or deriving `duration` from `timestamp`. > > According to the specification, the `duration` parameter is optional, so it should not be relied upon for such calculations. Additionaly, I am failing to see why `null` or `0` `duration` causes the issue. Those logs are [for statistics](https://searchfox.org/mozilla-central/rev/1cc09ebc0b32f69fd30057582671b0811f2cb9c8/dom/media/platforms/ffmpeg/FFmpegVideoDecoder.cpp#1012-1016,1030-1031) only. On my Ubuntu machine, both [video-decode-display sample](https://github.com/w3c/webcodecs/blob/6bd81184785cebcfa5cb5dbd6739db605476c8e0/samples/video-decode-display/demuxer_mp4.js#L102) or [videoDecoder-codec-specific WPT](https://github.com/web-platform-tests/wpt/blob/e630e188af8bf60b94489506daa62d30a7b092df/webcodecs/videoDecoder-codec-specific.https.any.js#L218) works fine with forcing `duration` to `null` or `0`. They have similar logs, but the decoding works fine. ``` 0:14.39 pid:15010 [RDD 15390: MediaPDecoder #3]: D/FFmpegVideo FFMPEG: Frame decode takes 2.36 ms average decode time 46.24 ms frame duration 0.00 average frame duration 0.00 decoded 2 frames 0:14.39 pid:15010 [RDD 15390: MediaPDecoder #3]: D/FFmpegVideo FFMPEG: slow decode: failed to decode in time (decoded late 2) ``` > Side note: It looks like the decoder object goes into an unrecoverable state when this happens, so the only way forward is to instantiate a new one. This means that we end up in a loop where the decoder gets switched to sw, fails, gets recreated and configured, get switched to sw, fails, etc. Did you see any issue in the logs with `MOZ_LOG="WebCodecs:5"`?
Bug 1906761 Comment 2 Edit History
Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.
(In reply to Andri Yngvason from comment #0) > I've implemented a WebCodecs based H.264 decoder for noVNC: https://github.com/novnc/noVNC/pull/1876 Hi Andri, Could you create a test page and upload it here, or create a minimal test environment with steps-to-reproduce, maybe? It would greatly help us diagnose the issue. > After a few frames have been decoded, the error callback is called and decoding stops. We are seeing errors like these on the command line: > ``` > [RDD 15365: MediaPDecoder #1]: D/FFmpegVideo FFMPEG: Frame decode takes 0.35 ms average decode time 1.45 ms frame duration 0.00 average frame duration 0.00 decoded 12 frames > [RDD 15365: MediaPDecoder #1]: D/FFmpegVideo FFMPEG: slow decode: failed to decode in time (decoded late 12) > ``` > What appears to be happening here is that Firefox doesn't receive a `duration` parameter with each encoded frame chunk, and sets the `duration` to zero instead of disabling the "slow decode" code path or deriving `duration` from `timestamp`. > > According to the specification, the `duration` parameter is optional, so it should not be relied upon for such calculations. Additionaly, I am failing to see why `null` or `0` `duration` causes the issue. Those logs are [for statistics](https://searchfox.org/mozilla-central/rev/1cc09ebc0b32f69fd30057582671b0811f2cb9c8/dom/media/platforms/ffmpeg/FFmpegVideoDecoder.cpp#1012-1016,1030-1031) only. On my Ubuntu machine, both [video-decode-display sample](https://github.com/w3c/webcodecs/blob/6bd81184785cebcfa5cb5dbd6739db605476c8e0/samples/video-decode-display/demuxer_mp4.js#L102) or [videoDecoder-codec-specific WPT](https://github.com/web-platform-tests/wpt/blob/e630e188af8bf60b94489506daa62d30a7b092df/webcodecs/videoDecoder-codec-specific.https.any.js#L218) works fine with forcing `duration` to `null` or `0`. They have similar logs, but the decoding works fine. ``` 0:14.39 pid:15010 [RDD 15390: MediaPDecoder #3]: D/FFmpegVideo FFMPEG: Frame decode takes 2.36 ms average decode time 46.24 ms frame duration 0.00 average frame duration 0.00 decoded 2 frames 0:14.39 pid:15010 [RDD 15390: MediaPDecoder #3]: D/FFmpegVideo FFMPEG: slow decode: failed to decode in time (decoded late 2) ``` > Side note: It looks like the decoder object goes into an unrecoverable state when this happens, so the only way forward is to instantiate a new one. This means that we end up in a loop where the decoder gets switched to sw, fails, gets recreated and configured, get switched to sw, fails, etc. Did you see any issue in the logs with `MOZ_LOG="WebCodecs:5"`?
(In reply to Andri Yngvason from comment #0) > I've implemented a WebCodecs based H.264 decoder for noVNC: https://github.com/novnc/noVNC/pull/1876 Hi Andri, Could you create a test page demonstrating the issue and upload it here, or create a minimal test environment with steps-to-reproduce? It would greatly help us diagnose the issue. > After a few frames have been decoded, the error callback is called and decoding stops. We are seeing errors like these on the command line: > ``` > [RDD 15365: MediaPDecoder #1]: D/FFmpegVideo FFMPEG: Frame decode takes 0.35 ms average decode time 1.45 ms frame duration 0.00 average frame duration 0.00 decoded 12 frames > [RDD 15365: MediaPDecoder #1]: D/FFmpegVideo FFMPEG: slow decode: failed to decode in time (decoded late 12) > ``` > What appears to be happening here is that Firefox doesn't receive a `duration` parameter with each encoded frame chunk, and sets the `duration` to zero instead of disabling the "slow decode" code path or deriving `duration` from `timestamp`. > > According to the specification, the `duration` parameter is optional, so it should not be relied upon for such calculations. Additionaly, I am failing to see why `null` or `0` `duration` causes the issue. Those logs are [for statistics](https://searchfox.org/mozilla-central/rev/1cc09ebc0b32f69fd30057582671b0811f2cb9c8/dom/media/platforms/ffmpeg/FFmpegVideoDecoder.cpp#1012-1016,1030-1031) only. On my Ubuntu machine, both [video-decode-display sample](https://github.com/w3c/webcodecs/blob/6bd81184785cebcfa5cb5dbd6739db605476c8e0/samples/video-decode-display/demuxer_mp4.js#L102) or [videoDecoder-codec-specific WPT](https://github.com/web-platform-tests/wpt/blob/e630e188af8bf60b94489506daa62d30a7b092df/webcodecs/videoDecoder-codec-specific.https.any.js#L218) works fine with forcing `duration` to `null` or `0`. They have similar logs, but the decoding works fine. ``` 0:14.39 pid:15010 [RDD 15390: MediaPDecoder #3]: D/FFmpegVideo FFMPEG: Frame decode takes 2.36 ms average decode time 46.24 ms frame duration 0.00 average frame duration 0.00 decoded 2 frames 0:14.39 pid:15010 [RDD 15390: MediaPDecoder #3]: D/FFmpegVideo FFMPEG: slow decode: failed to decode in time (decoded late 2) ``` > Side note: It looks like the decoder object goes into an unrecoverable state when this happens, so the only way forward is to instantiate a new one. This means that we end up in a loop where the decoder gets switched to sw, fails, gets recreated and configured, get switched to sw, fails, etc. Did you see any issue in the logs with `MOZ_LOG="WebCodecs:5"`?
(In reply to Andri Yngvason from comment #0) > I've implemented a WebCodecs based H.264 decoder for noVNC: https://github.com/novnc/noVNC/pull/1876 Hi Andri, Could you create a test page demonstrating the issue and upload it here, or create a minimal test environment with steps-to-reproduce? It would greatly help us diagnose the issue. > After a few frames have been decoded, the error callback is called and decoding stops. We are seeing errors like these on the command line: > ``` > [RDD 15365: MediaPDecoder #1]: D/FFmpegVideo FFMPEG: Frame decode takes 0.35 ms average decode time 1.45 ms frame duration 0.00 average frame duration 0.00 decoded 12 frames > [RDD 15365: MediaPDecoder #1]: D/FFmpegVideo FFMPEG: slow decode: failed to decode in time (decoded late 12) > ``` > What appears to be happening here is that Firefox doesn't receive a `duration` parameter with each encoded frame chunk, and sets the `duration` to zero instead of disabling the "slow decode" code path or deriving `duration` from `timestamp`. > > According to the specification, the `duration` parameter is optional, so it should not be relied upon for such calculations. Additionaly, I am failing to see why `null` or `0` `duration` causes the issue. Those logs are [for statistics](https://searchfox.org/mozilla-central/rev/1cc09ebc0b32f69fd30057582671b0811f2cb9c8/dom/media/platforms/ffmpeg/FFmpegVideoDecoder.cpp#1012-1016,1030-1031) only. Yes, `duration` is optional. On my Ubuntu machine, both [video-decode-display sample](https://github.com/w3c/webcodecs/blob/6bd81184785cebcfa5cb5dbd6739db605476c8e0/samples/video-decode-display/demuxer_mp4.js#L102) or [videoDecoder-codec-specific WPT](https://github.com/web-platform-tests/wpt/blob/e630e188af8bf60b94489506daa62d30a7b092df/webcodecs/videoDecoder-codec-specific.https.any.js#L218) works fine with forcing `duration` to `null` or `0`. They have similar logs, but the decoding works fine. ``` 0:14.39 pid:15010 [RDD 15390: MediaPDecoder #3]: D/FFmpegVideo FFMPEG: Frame decode takes 2.36 ms average decode time 46.24 ms frame duration 0.00 average frame duration 0.00 decoded 2 frames 0:14.39 pid:15010 [RDD 15390: MediaPDecoder #3]: D/FFmpegVideo FFMPEG: slow decode: failed to decode in time (decoded late 2) ``` > Side note: It looks like the decoder object goes into an unrecoverable state when this happens, so the only way forward is to instantiate a new one. This means that we end up in a loop where the decoder gets switched to sw, fails, gets recreated and configured, get switched to sw, fails, etc. Did you see any issue in the logs with `MOZ_LOG="WebCodecs:5"`?