1967793 - Unable to determine hardware acceleration with WebCodecs

Status: RESOLVED FIXED

(Core :: Audio/Video: Web Codecs, defect)

Description

[Luke Curley]

Attachment: false-compatibility.png

Steps to reproduce:

I'm making a website that utilizes WebCodecs. I need to determine the encoding codec and always want to use hardware encoding (when available) as battery life is more important than fidelity.

1. Loop through all codecs based on efficiency: av1, vp9, hevc, h254, vp8.
2. Check if the browser supports hardware acceleration, if so use it.
3. Repeat the loop but this time with software acceleration.

I'm using the hardwareAcceleration: "prefer-hardware" field to determine if hardware acceleration is supported. Here's an example of one of the checks:

VideoEncoder.isConfigSupported({
    codec: "av01.0.08M.08",
    width: 1280,
    height: 720,
    hardwareAcceleration: "prefer-hardware",
}).then(console.log)

Actual results:

On Chrome, I correctly use HEVC (on my Macbook) or H.264 because they have hardware accelerated encoding.

However on Firefox, all (supported) codecs show up as hardware accelerated. This results in trying to perform a software AV1/VP9 encode which understandably does not work well. I found this issue because my laptop was spinning at 100% CPU and the resulting video was unwatchable due to dropped frames. I looked at the stack trace in the profiler and sure enough, it's performing a software VP9 encode.

I attached a screenshot of the difference in reported compatibility between Chrome and Firefox.

Expected results:

Ideally, Firefox would report that the configuration is not supported if hardware acceleration is not available.

But this is an intended feature to prevent fingerprinting. In that case, if this "hardwareAcceleration" is not allowed, then it would be nice to get some sort of error to indicate that the field does nothing. I could then loop over codecs in a different order (ex. H264 first) so I don't make a horrible choice if software encoding is used.

What Firefox is doing is allowed by the W3C spec, as it says the field may be ignored. I'll file an issue with the W3C on this front.

I also looked into the Media Capabilities API but it's not clear how it intersects with WebCodecs. It requires a container but WebCodecs is containerless, so putting something like "webm/av1" might return unsupported for other reasons. I'm also not sure if it runs into the same fingerprinting issues (probably) so I didn't spend much time on it.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Core::Audio/Video: Web Codecs' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 2

[Luke Curley]

Filed a W3C issue with more information: https://github.com/w3c/webcodecs/issues/896

Comment 3

[Paul Adenot (:padenot)]

Attachment: Bug 1967793 - Properly implement hardware check in VideoEncoderConfig.isConfigSupported. r?#media-playback-reviewers,tjr

Comment 4

[Paul Adenot (:padenot)]

Attachment: Bug 1967793 - Add skeleton code for the decoder side, for now no-op. r?#media-playback-reviewers

The implementation will be made functional in 1968022.

Comment 5

[Paul Adenot (:padenot)]

Luke, thanks for filing a bug, this was simply something that wasn't implemented. I just implemented it for the encoding side, with provision for the users to decide to not reveal the capabilities of their machine.

The decoding side needs a refactoring and will be implemented in bug 1968022 (that you can follow if you want).

Comment 6

[Paul Adenot (:padenot)]

Attachment: Bug 1967793 - Add a mochitest for VideoEncoderConfig.isConfigSupported. r?#media-playback-reviewers

This is a mochitest because very tied to our infra.

Comment 7

[Luke Curley]

(In reply to Paul Adenot (:padenot) from comment #5)

Luke, thanks for filing a bug, this was simply something that wasn't implemented. I just implemented it for the encoding side, with provision for the users to decide to not reveal the capabilities of their machine.

The decoding side needs a refactoring and will be implemented in bug 1968022 (that you can follow if you want).

Whew, thanks so much.

It would still be nice to know when resistfingerprinting is enabled so I can assume software encodes instead. But yeah I don't know the best way to implement that with the current specification.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6d383fe964d7
Properly implement hardware check in VideoEncoderConfig.isConfigSupported. r=tjr,media-playback-reviewers,alwu
https://hg.mozilla.org/integration/autoland/rev/3bfdb0258314
Add skeleton code for the decoder side, for now no-op. r=media-playback-reviewers,alwu
https://hg.mozilla.org/integration/autoland/rev/d4fe7ec9bdb4
Add a mochitest for VideoEncoderConfig.isConfigSupported. r=media-playback-reviewers,alwu
https://hg.mozilla.org/integration/autoland/rev/c2454de86ff1
apply code formatting via Lando

Comment 9

[Cristina Horotan [:chorotan]]

Backed out for causing multiple failures

Backout link

Push with failures

Failure log mda failure
Failure log bustage

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/cd2179edcc63
Revert "Bug 1967793: apply code formatting via Lando" for causing multiple failures

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1fdc8d90f80e
Properly implement hardware check in VideoEncoderConfig.isConfigSupported. r=tjr,media-playback-reviewers,alwu
https://hg.mozilla.org/integration/autoland/rev/893ae4d96d5c
Add skeleton code for the decoder side, for now no-op. r=media-playback-reviewers,alwu
https://hg.mozilla.org/integration/autoland/rev/ebd57d9d2564
Add a mochitest for VideoEncoderConfig.isConfigSupported. r=media-playback-reviewers,alwu
https://hg.mozilla.org/integration/autoland/rev/413bdd77fce3
apply code formatting via Lando

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5c620d3fef10
Revert "Bug 1967793: apply code formatting via Lando" for causing codecs failures in audio-encoder-codec-specific.https.any.html.

Comment 13

[Serban Stanca [:SerbanS]]

Reverted this because it was causing codecs failures in audio-encoder-codec-specific.https.any.html.

• Backout link
• Push with failures
• Failure Log
• Failure line: TEST-UNEXPECTED-PRECONDITION_FAILED | /webcodecs/audio-encoder-codec-specific.https.any.html | Test the Opus DTX flag works. - Opus not supported

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/8d53582c1f41
https://hg.mozilla.org/integration/autoland/rev/672938523852
Properly implement hardware check in VideoEncoderConfig.isConfigSupported. r=tjr,media-playback-reviewers,alwu
https://github.com/mozilla-firefox/firefox/commit/d1030bd7df00
https://hg.mozilla.org/integration/autoland/rev/598db5125d73
Add skeleton code for the decoder side, for now no-op. r=media-playback-reviewers,alwu
https://github.com/mozilla-firefox/firefox/commit/77eacc454d3f
https://hg.mozilla.org/integration/autoland/rev/1ababe54c300
Add a mochitest for VideoEncoderConfig.isConfigSupported. r=media-playback-reviewers,alwu

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/80e2c08d20fb
https://hg.mozilla.org/integration/autoland/rev/ff0b4d1cc61e
Revert "Bug 1967793 - Add a mochitest for VideoEncoderConfig.isConfigSupported. r=media-playback-reviewers,alwu" for causing codecs failures @ video-encoder-h26x-annexb.https.any.html?h264_annexb_hardware

Comment 16

[amarc]

Backed out for causing codecs failures @ video-encoder-h26x-annexb.https.any.html?h264_annexb_hardware

• Backout link
• Push with failures
• Failure Log

Comment 17

[BugBot [:suhaib / :marco/ :calixte]]

There are some r+ patches which didn't land and no activity in this bug for 2 weeks.
:padenot, could you have a look please?
If you still have some work to do, you can add an action "Plan Changes" in Phabricator.
For more information, please visit BugBot documentation.

Comment 18

[Paul Adenot (:padenot)]

Attachment: Bug 1967793 - Properly implement hardware check in VideoEncoderConfig.isConfigSupported. r?#media-playback-reviewers,tjr

Comment 19

[Paul Adenot (:padenot)]

Attachment: Bug 1967793 - Properly return encoding support in FFmpegEncoderModule. r?aosmond

Comment 20

[Paul Adenot (:padenot)]

Attachment: Bug 1967793 - Properly return encoder support in WMFEncoderModule. r?chunmin,aosmond

Comment 21

[Paul Adenot (:padenot)]

Attachment: Bug 1967793 - Adjust WPT expectations. r?#media-playback-reviewers

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/4090910e48ac
https://hg.mozilla.org/integration/autoland/rev/0595030292bc
Properly implement hardware check in VideoEncoderConfig.isConfigSupported. r=tjr,media-playback-reviewers,alwu
https://github.com/mozilla-firefox/firefox/commit/5b78bdc1a5ba
https://hg.mozilla.org/integration/autoland/rev/fbb79e04e432
Add skeleton code for the decoder side, for now no-op. r=media-playback-reviewers,alwu
https://github.com/mozilla-firefox/firefox/commit/3f28fe0d4b37
https://hg.mozilla.org/integration/autoland/rev/efe1532ce4d8
Add a mochitest for VideoEncoderConfig.isConfigSupported. r=media-playback-reviewers,alwu
https://github.com/mozilla-firefox/firefox/commit/2432f61c6c8e
https://hg.mozilla.org/integration/autoland/rev/7c98de7cc264
Properly return encoding support in FFmpegEncoderModule. r=media-playback-reviewers,alwu
https://github.com/mozilla-firefox/firefox/commit/e82dce81263a
https://hg.mozilla.org/integration/autoland/rev/22f49cfe19b8
Properly return encoder support in WMFEncoderModule. r=chunmin
https://github.com/mozilla-firefox/firefox/commit/10e77eb47e2b
https://hg.mozilla.org/integration/autoland/rev/cd26472e0bd3
Adjust WPT expectations. r=media-playback-reviewers,alwu

Comment 23

[Cristina Horotan [:chorotan]]

https://hg.mozilla.org/mozilla-central/rev/0595030292bc
https://hg.mozilla.org/mozilla-central/rev/fbb79e04e432
https://hg.mozilla.org/mozilla-central/rev/efe1532ce4d8
https://hg.mozilla.org/mozilla-central/rev/7c98de7cc264
https://hg.mozilla.org/mozilla-central/rev/22f49cfe19b8
https://hg.mozilla.org/mozilla-central/rev/cd26472e0bd3

Comment 24

[Thorin [:thorin]]

(In reply to Luke Curley from comment #7)

It would still be nice to know when resistfingerprinting is enabled so I can assume software encodes instead. But yeah I don't know the best way to implement that with the current specification.

Hi Luke. Can I ask why you need to assume software encodes when RFP is enabled (if you detected that)

Everyone else, sorry for the noise :)

Comment 25

[Luke Curley]

(In reply to Thorin [:thorin] from comment #24)

(In reply to Luke Curley from comment #7)

It would still be nice to know when resistfingerprinting is enabled so I can assume software encodes instead. But yeah I don't know the best way to implement that with the current specification.

Hi Luke. Can I ask why you need to assume software encodes when RFP is enabled (if you detected that)

Everyone else, sorry for the noise :)

Of course.

If hardware acceleration is supported, then I can encode AV1/VP9 etc in real-time. Otherwise, I need to use a simpler codec like H264/VP8 in order to achieve real-time encoding. I discovered this bug because my CPU was pegged at 100% performing a software VP9 encode and only able to produce a few frames per second. I switched to H264 which presumably uses hardware encoding but would also be less disruptive for software encoding.

I suppose I could switch codec if the output frame rate is significantly worse than the input frame rate. It's just a better user experience if I can avoid the guess work and use an adequate codec from the onset.

For decoding it doesn't matter as much based on the benchmarks I've seen. Meanwhile the difference between hardware and software encoding is an order of magnitude especially for for newer codecs.

Comment 26

[Thorin [:thorin]]

Thanks, didn't see a mention of RFP :) I'm not a video codec expert, but I work with RFP

• We don't do much spoofing in capabilities from memory, and we don't control the codecs available at all per platform. And AFAIK we don't mess with hardware encoding/decoding
• Talking about framerates, RFP's limits framerate: RAF (requestAnimationFrame) to 60FPS

Still trying to understand how RFP is impacting your choices here: I've been working on making RFP not "break" things and that includes websites altering their behaviour around it. Perhaps I could spin this up in another issue for RFP improvement, if you're willing to indulge

nfi tjr for some input

Comment 27

[Luke Curley]

(In reply to Thorin [:thorin] from comment #26)

Thanks, didn't see a mention of RFP :) I'm not a video codec expert, but I work with RFP

• We don't do much spoofing in capabilities from memory, and we don't control the codecs available at all per platform. And AFAIK we don't mess with hardware encoding/decoding
• Talking about framerates, RFP's limits framerate: RAF (requestAnimationFrame) to 60FPS

Still trying to understand how RFP is impacting your choices here: I've been working on making RFP not "break" things and that includes websites altering their behaviour around it. Perhaps I could spin this up in another issue for RFP improvement, if you're willing to indulge

nfi tjr for some input

Oh I just made an assumption that RFP was involved, but it turned out that hardware encoding wasn't properly hooked up. I imagine hardware codec support could be used for fingerprinting because it depends on the graphics card, operating system, drivers, etc. Especially when codec profiles, constraints, etc are introduced.