Open Bug 2043895 Opened 1 month ago Updated 1 month ago

mediaCapabilities.decodingInfo is still slow

Categories

(Core :: Audio/Video, defect, P2)

Product:
Core
Component:
Audio/Video
Version:
Firefox 153
Type:
defect

Tracking

()

People

(Reporter: bugzilla.mozilla.org.wireless475, Unassigned)

References

(Depends on 4 open bugs, Blocks 1 open bug)

Details

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:152.0) Gecko/20100101 Firefox/152.0

Steps to reproduce:

After triaging an issue for playback starting much slower on Firefox than Chromium with Shaka player. I arrived at https://bugzilla.mozilla.org/show_bug.cgi?id=1676902, which seems to be closed despite the performance still not being usable in some cases.

Quick test:

(async () => {
    let N = 30;
    let ms = t => t.toFixed(2);
    let base = {
        type: 'media-source',
        video: {
            contentType: 'video/mp4; codecs="avc1.640028"',
            width: 1920, height: 1080, framerate: 30,
            bitrate: 5_000_000,
        },
    };

    // 1. cold: first call after page load (RDD warmup included)
    let t0 = performance.now();
    await navigator.mediaCapabilities.decodingInfo(structuredClone(base));
    console.log(`cold:     ${ms(performance.now() - t0)} ms`);

    // 2. uncached: vary bitrate per call
    t0 = performance.now();
    for (let i = 0; i < N; i++) {
        let c = structuredClone(base);
        c.video.bitrate = 5_000_000 + i + 1;
        await navigator.mediaCapabilities.decodingInfo(c);
    }
    console.log(`uncached: ${ms((performance.now() - t0) / N)} ms/call  (N=${N})`);

    // 3. cached: same config repeatedly
    let fixed = structuredClone(base);
    fixed.video.bitrate = 7_500_000;
    await navigator.mediaCapabilities.decodingInfo(fixed);  // prime
    t0 = performance.now();
    for (let i = 0; i < N; i++) {
        await navigator.mediaCapabilities.decodingInfo(fixed);
    }
    console.log(`cached:   ${ms((performance.now() - t0) / N)} ms/call  (N=${N})`);
})();

Actual results:

Chromium results:

cold:     1.60 ms
uncached: 0.09 ms/call  (N=30)
cached:   0.07 ms/call  (N=30)

Firefox Nightly Version 153.0a1 (2026-05-30) (64-bit) results:

cold:     16.00 ms
uncached: 18.87 ms/call  (N=30)
cached:   17.33 ms/call  (N=30)

Expected results:

Firefox should have more comparable performance. In some cases playback was delayed by tens of seconds because of the volume of available stream types that had to be checked.

Thanks for the detailed benchmark and for tracking this back to bug 1676902.

I can confirm the cost and I have looked into where it comes from. For a non-DRM video configuration, navigator.mediaCapabilities.decodingInfo() currently instantiates a real MediaDataDecoder for each query -- it creates the decoder, calls Init(), reads the hardware-acceleration status, then shuts it down. On Windows the H.264 decoder is a Media Foundation decoder set up across the RDD/utility process, which dominates the ~17 ms you measured, and the per-call work is serialized, so Shaka's many parallel probes queue up. The result is not cached, which is why your cached run costs the same as uncached. (Bug 1676902's earlier improvement was on the codec-agnostic/WebM path, so it never touched this Windows H.264 instantiation cost.)

The key observation: of the three result fields, supported already comes from a cheap in-process capability lookup and smooth is a constant -- only powerEfficient currently needs the live decoder, and only to read its hardware-accelerated flag. The WebRTC branch of this same API already computes powerEfficient without instantiating a decoder. So the proposed fix is:

  1. Compute powerEfficient from the capability/HW-decode signal instead of spinning up and tearing down a decoder, mirroring the WebRTC path. (One dependency: in the content process the remote support query does not yet distinguish HW vs SW decode -- tracked by bug 1754239 -- so the HW-accel signal has to come from the same source the WebRTC path uses, or ride on that bug.)
  2. Add a configuration-keyed result cache as a complementary safety net, so repeated identical probes (exactly Shaka's pattern) are effectively free.

This is a real performance problem worth fixing. Ashley (:az) — adding a needinfo to see whether you'd like to take this, since it sits squarely in your Media Capabilities area. If you don't have the bandwidth, we can route it to :alwu instead. Keeping S3/P2 — functional, but a meaningful startup-latency regression vs. Chromium on Windows DASH/Shaka.

Blocks: video-startup
Severity: -- → S3
Flags: needinfo?(azebrowski)
Priority: -- → P2
Blocks: media-capabilities
No longer blocks: video-startup

I won't be able to get to this immediately - :alwu or :pehrsons, would you be able to take a closer look?

Flags: needinfo?(apehrson)
Flags: needinfo?(alwu)

FWIW, I closed Bug 1676902 as worksforme because that particular test of video/webm; codecs=vp8 was now fast. But I didn't have my thinking hat on - it was an example from a larger enumeration test of video and audio for fingerprinting - so not all codecs may have improved over the years

and indeed 'video/mp4; codecs="avc1.640028"' didn't move in the PoC 1676902 (note you need to remove the blank dimensions size since that's invalid in FF149+)

              this PoC                PoC 1676902 webm vp8           PoC 1676902 mp4 avc1.640028
chrome148
    cold:     3.80 ms                 4.799999952316284 ms           5.100000023841858 ms
    uncached: 0.16 ms/call  (N=30)
    cached:   0.11 ms/call  (N=30)
nightly153
    cold:     11.75 ms                10.21100000000024 ms           234.45400000000154 ms
    uncached: 2.81 ms/call  (N=30)    3.3997999999992317 ms rerun    220.85879999998724 ms rerun
    cached:   2.31 ms/call  (N=30)
FF102
    cold:     10.37 ms                 72.41029999999955 ms          220.14689999999973 ms
    uncached: 2.75 ms/call  (N=30)     49.83520000000135 ms rerun    201.61880000000383 ms rerun
    cached:   2.38 ms/call  (N=30)
FF95
    cold:     43.09 ms                 99.30589999999938 ms          268.40469999999914 ms
    uncached: 2.53 ms/call  (N=30)     80.84870000000228 ms rerun    233.44219999999768 ms rerun
    cached:   2.79 ms/call  (N=30)

Perhaps we should create a new issue to test all audio/video codecs combos vs chrome for possible perf improvements - edit: and assign actionable findings to Bug 1884199

Thanks for the report!

One would imagine it goes quicker if you run the requests in parallel as opposed to serially, but that does not seem to be the case, see bug 1533252.

I upped N to 300 and got a logging profile.

On my macbook the time is ~5ms per request. You can see in the profile it takes 4.8ms for RDD to reply to decoder init in one instance. The reason is the platform call to allocate the platform VideoToolbox decoder.

Caching in the content process seems like a good solution to this problem. But we cannot just cache results to the specific request. We'd need to query the codec for its capabilities (supported resolution range, supported framerate range, supported bitrate range, etc.) and cache those. That may need a bit of a refactor so wouldn't be a quick fix.

However, I do not understand why we create a decoder instance to check for support. Doing a bit of archaeology it seems this decoder creation stems from a time when we ran a benchmark on the decoder to be able to fill the decodingInfo request's smooth attribute. We no longer do that, and PDMFactory::Supports is able to answer whether a codec is hw-backed or not. If PDMFactory::Supports can run quicker we should be able to do a quick fix.

Keeping the needinfo to answer this.

Flags: needinfo?(azebrowski)

The first problem is RemoteDecoderModule cannot answer powerEfficient and ditto for OmxDecoderModule. Bug 1754239 is referenced as intended to fix those cases, but it didn't. It seems its dependency, bug 1766307, fixed the other cases.

Status: UNCONFIRMED → NEW
Ever confirmed: true
Flags: needinfo?(apehrson)
Flags: needinfo?(alwu)

I have some patches written with a new path behind pref.

(Linux) Without:

cold:     25.00 ms
uncached: 24.83 ms/call  (N=30)
cached:   23.17 ms/call  (N=30)

(Same Linux) With:

cold:     5.00 ms
uncached: 0.07 ms/call  (N=30)
cached:   0.13 ms/call  (N=30)

It is faster but loses some plumbing for resolution/framerate support. It is unclear to me how particular codec instance creation was about those bits in the first place. I'll proceed with landing behind the pref and we can evaluate/improve this in a second step.

I discussed the accuracy of this approach with padenot and it is lacking a bit. The other question is if we want that much accuracy. I'll proceed with putting this up behind a pref.

(In reply to bugzilla.mozilla.org.wireless475 from comment #0)
In some cases playback was delayed by tens of seconds because of the volume of available stream types that had to be checked.

If you have an example of this, please link me up.

A shaka player demo I tried had 2 audio variants and 12 video variants, totaling 24 MediaCapabilities queries in 100ms. On my linux box per comment 6 it would have been maybe 500ms, but still far from "tens of seconds".

Flags: needinfo?(bugzilla.mozilla.org.wireless475)

(In reply to Andreas Pehrson [:pehrsons] from comment #8)

(In reply to bugzilla.mozilla.org.wireless475 from comment #0)
In some cases playback was delayed by tens of seconds because of the volume of available stream types that had to be checked.

If you have an example of this, please link me up.

A shaka player demo I tried had 2 audio variants and 12 video variants, totaling 24 MediaCapabilities queries in 100ms. On my linux box per comment 6 it would have been maybe 500ms, but still far from "tens of seconds".

It's pretty hard to test right now, since I found this while using Piped (alternative Youtube frontend) and the numbers of encodings and other options were usually causing visible delay for the DASH and SABR protocols. SABR is currently in development, DASH is not really accessible these days and the whole project relies on self hosting right now. I also patched out this behaviour by using isTypeSupported specifically for Firefox since it seemed like Shaka could get away with it for now. This deprives the player of information needed to determine the best pick for a scenario and I consider it a hack.

In short, I can only only point you to the project, but chances are you will not have much luck with the codecs you get and we already have a workaround in place for it. I think that the benchmark is proof enough for the problem. Youtube has a lot of qualities, video codecs and audio variants. This causes a lot of possible combinations which are explored with decodingInfo.

Thanks for investigating this issue.

Flags: needinfo?(bugzilla.mozilla.org.wireless475)

(In reply to bugzilla.mozilla.org.wireless475 from comment #9)

It's pretty hard to test right now, since I found this while using Piped (alternative Youtube frontend) and the numbers of encodings and other options were usually causing visible delay for the DASH and SABR protocols.

If you see "visible delay" meaning "tens of seconds" anywhere please provide a link as it probably means there's a bug somewhere else. For the worst-case I've seen here (25ms per query) "tens of seconds" means we'd have to see at least 400 queries, e.g. the cartesian product of 10 audio streams and 40 video streams. That seems unlikely for any manifest.

If you want a short-term workaround, don't do the cartesian product of all audio and video streams for query. Audio and video decoders are completely decoupled, at least in Firefox, so you could query them in isolation. In the example of 10 audio and 40 video streams that would make 50 queries instead of 400, shaving away 87.5% of the time spent. Audio would also be much quicker to answer, so probably more like 90%.

(In reply to Andreas Pehrson [:pehrsons] from comment #10)

(In reply to bugzilla.mozilla.org.wireless475 from comment #9)

It's pretty hard to test right now, since I found this while using Piped (alternative Youtube frontend) and the numbers of encodings and other options were usually causing visible delay for the DASH and SABR protocols.

If you see "visible delay" meaning "tens of seconds" anywhere please provide a link as it probably means there's a bug somewhere else. For the worst-case I've seen here (25ms per query) "tens of seconds" means we'd have to see at least 400 queries, e.g. the cartesian product of 10 audio streams and 40 video streams. That seems unlikely for any manifest.

If you want a short-term workaround, don't do the cartesian product of all audio and video streams for query. Audio and video decoders are completely decoupled, at least in Firefox, so you could query them in isolation. In the example of 10 audio and 40 video streams that would make 50 queries instead of 400, shaving away 87.5% of the time spent. Audio would also be much quicker to answer, so probably more like 90%.

It varies a lot. Tens of seconds is the extreme. It's likely due to some videos having multiple languages. Videos would usually take around 1-2 seconds at least to load. This still doesn't change the fact that it should be addressed. Even a couple of seconds is unpleasant for users.

I cannot provide you with a link since the public instances are not functional. On top of that, the most recent build has the previously mentioned hack, that solves it. I do my testing on a private instance that I intentionally avoid exposing to the internet and that has been patched for 1-2 weeks at this point. I can say that replacing decodingInfo made all videos play without a significant delay for me. Chromium always played without significant delays for me. Videos with a larger following (ex: Linus Tech Tips) seemed to be the ones most affected(likely had AI dubs). I am fairly certain this is the root cause.

Thanks for the workaround suggestion, the current one works for now and I would prefer to only touch it once I need to remove it. I do not work on Shaka.

You need to log in before you can comment on or make changes to this bug.