1859400 - HEVC file causes DoS of HEVC playback

Status: RESOLVED FIXED

(Core :: Audio/Video: Playback, defect, P3)

Description

[Tyson Smith [:tsmith]]

Attachment: testcase.mp4

Found with m-c 20231016-f64b858317d1 (--enable-debug --enable-fuzzing)

Attempting to play the test case hangs attempting to load(?). From this point on it is not possible to play HEVC files without relaunching the browser.

Comment 1

[Alastor Wu [:alwu]]

This test file seems crashing the whole GPU process, I am still not sure if it should be a security issue. But for safe, making this bug as a sec bug for now.

Comment 2

[Daniel Veditz [:dveditz]]

This is nightly-only and surely it will be fixed before we ship. if we had to rate it we'd call it sec-low if anything, but we don't need to hide it.

Comment 3

[BugBot [:suhaib / :marco]]

The severity field is not set for this bug.
:jimm, could you have a look please?

For more information, please visit BugBot documentation.

Comment 4

[BugBot [:suhaib / :marco]]

The severity field is not set for this bug.
:jimm, could you have a look please?

For more information, please visit BugBot documentation.

Comment 5

[BugBot [:suhaib / :marco]]

This bug prevents fuzzing from making progress; however, it has low severity. It is important for fuzz blocker bugs to be addressed in a timely manner (see here why?).
:alwu, could you consider increasing the severity?

For more information, please visit BugBot documentation.

Comment 6

[mirh]

Still happening in 131b4

Comment 7

[Alastor Wu [:alwu]]

Is anyone still be able to reproduce this issue? I couldn’t reproduce it on the latest central build. Below are the testing steps I followed, and the browser did not hang or crash.

python3 -m fuzzfetch --central --asan --fuzzing -n firefox
python3 -m grizzly.replay ./firefox/firefox testcase.mp4

Comment 8

[mirh]

I didn't test the nightly.. but it's not like the whole browser hanged or crashed. Only whatever subsequent attempt of decoding a video got stuck.
Can the fuzzer even access the HEVC MFT plugin?

Comment 9

[Tyson Smith [:tsmith]]

(In reply to mirh from comment #8)

Can the fuzzer even access the HEVC MFT plugin?

This is a mutation fuzzer and the browser is running the output, so yes.

(In reply to Alastor Wu [:alwu] from comment #7)

Is anyone still be able to reproduce this issue?

Yes, I am able to reproduce the issue. I tested with m-c 20241001-0546d4eb6429. (Windows 11 with HW HEVC support)

and the browser did not hang or crash.

The process of loading the content (?) hung, instead of failing. This means every attempt will need to wait to timeout which greatly reduces the iteration rate.

Comment 10

[Alastor Wu [:alwu]]

(In reply to Alastor Wu [:alwu] from comment #7)

Is anyone still be able to reproduce this issue?

Yes, I am able to reproduce the issue. I tested with m-c 20241001-0546d4eb6429. (Windows 11 with HW HEVC support)

The process of loading the content (?) hung, instead of failing. This means every attempt will need to wait to timeout which greatly reduces the iteration rate.

Hmm the latest Nightly is also from 10/01, but when I opened the file, the process (tab) didn't either hang or become non-responsive. As you mention timeout, is fuzzy-test waiting for specific event? In addition, would you mind to follow this instruction to capture a media profile for me when the issue happens? Thank you.

Comment 11

[Tyson Smith [:tsmith]]

(In reply to Alastor Wu [:alwu] from comment #10)

Hmm the latest Nightly is also from 10/01, but when I opened the file, the process (tab) didn't either hang or become non-responsive. As you mention timeout, is fuzzy-test waiting for specific event?

It should not have anything to do with the fuzzing framework since the test is completely standalone. FWIW this is basically a corrupted mp4 file.

In addition, would you mind to follow this instruction to capture a media profile for me when the issue happens? Thank you.

https://share.firefox.dev/3XKMq6F

I opened the tab dragged the testcase.mp4 file and waited a few seconds then closed the tab. During that time the tab window is black and the loading indicator on the tab is active.

Comment 12

[Alastor Wu [:alwu]]

(In reply to Tyson Smith [:tsmith] from comment #11)

It should not have anything to do with the fuzzing framework since the test is completely standalone. FWIW this is basically a corrupted mp4 file.

Oh, sorry I'm not clear enough. What I mean is, what is the test doing exactly? Just play a media file to see if any load or error event? How is a test considered finished?

In addition, would you mind to follow this instruction to capture a media profile for me when the issue happens? Thank you.

https://profiler.firefox.com/from-browser/calltree/?globalTrackOrder=0w9&hiddenGlobalTracks=2w7&hiddenLocalTracksByPid=6668-0w8awc~26064-0w37wa~26220-01~16936-0w2~13068-0w4~28336-0w4~28040-0w4~3068-0w4~19224-013w57wc~2384-0w356&thread=a&v=10

This doesn't seem a valid url, did you upload it?

Comment 13

[Tyson Smith [:tsmith]]

Fixed.

Also the browser hangs on shutdown:

• open browser
• drag and drop testcase.mp4
• close browser

Comment 14

[Alastor Wu [:alwu]]

From the investigation so far, I think this is a driver/graphic card issue. From the profile you uploaded, we can see the CPU usage was very high in the media supervisor thread in the GPU process. The operations were all stuck in the stack of HEVCDECODER_STORE.dll, which didn't happen on my two Windows laptops [1]

[1]
Windows 10 : https://share.firefox.dev/4ePGa4y
Windows 11 : https://share.firefox.dev/4gLY7md

---

Tyson, would you mind help me do following things?

• Could you provide about:support for the device on which the issue can be reproduced?
• Do you have other devices running fuzzing? Does this issue happen on other devices as well?
• Did you install this HEVC extension on your device?

Thanks!

Comment 15

[Tyson Smith [:tsmith]]

Attachment: about-support.txt

Comment 16

[Tyson Smith [:tsmith]]

(In reply to Alastor Wu [:alwu] from comment #14)

• Could you provide about:support for the device on which the issue can be reproduced?

Attached.

• Do you have other devices running fuzzing? Does this issue happen on other devices as well?

Yes.

• Did you install this HEVC extension on your device?

No, but I do have HEVC Video Extensions from Device Manufacturer that I did not explicitly install.

Comment 17

[David [:Ceremony]]

This bug causes issues on aliexpress as reported with bug 1956680

Comment 18

[David [:Ceremony]]

I've dived into it further and it seems to be caused by a more recent version of the windows HEVC decoder you have currently installed:
I had both "HEVC Video Extensions from Device Manufacturer" (v2.4.39.0) and "HEVC Video Extension" (v2.4.37.0) installed simultaneously. The memory issue was present in Firefox.

Then I first uninstalled the later "non-device manufacturer" variant and the memory issue was still there.
Then I also removed the "device manufacturer" variant and hevc no longer worked at all (duh).

I then grabbed a much older "HEVC Video Extensions from Device Manufacturer" (v2.0.51931.0) off archive.org and installed it via powershell after unpacking:
Add-AppxPackage "Microsoft.HEVCVideoExtension_2.0.61931.0_neutral_~_8wekyb3d8bbwe.appxBundle"

With this version, there was no memory or playback issue of hevc within firefox.
However, after checking the windows store for updates, it decided to update this older version to the previous v2.4.39.0 version.

Comment 19

[Alastor Wu [:alwu]]

Thanks. I’ve replied regarding the memory leak issue in bug 1956680, and it would be better to keep all memory leak–related discussion consolidated there. Since this bug appears to be addressing a different issue, I think we can continue tracking it separately.

Comment 20

[Paul Adenot (:padenot)]

Attachment: Bug 1859400 - Filter user_data_unregistered SEI from HEVC AnnexB extradata on Windows. r?alwu

When converting HVCC extradata to AnnexB format for prepending to
keyframes, skip non-parameter-set NALUs (e.g. SEI). A corrupted file
with a large PREFIX_SEI_NUT in the hvcC box caused the Windows HEVC
hardware decoder to hang indefinitely.

Comment 21

[Paul Adenot (:padenot)]

Attachment: Bug 1859400 - Add regression tests for Windows HEVC hvcc SEI filtering. r?alwu

Comment 22

[Alastor Wu [:alwu]]

Attachment: about:conifg (my hardware setup can't reproduce the issue)

Just leave my about:config for future investigation, my setup can't reproduce the issue.

Comment 23

[Pulsebot]

Pushed by [e-mail]:
https://github.com/mozilla-firefox/firefox/commit/122d8e167e1c
https://hg.mozilla.org/integration/autoland/rev/1b9cb9824317
Filter user_data_unregistered SEI from HEVC AnnexB extradata on Windows. r=alwu
https://github.com/mozilla-firefox/firefox/commit/abcf3efb3ed5
https://hg.mozilla.org/integration/autoland/rev/cfef9906f94d
Add regression tests for Windows HEVC hvcc SEI filtering. r=alwu
https://github.com/mozilla-firefox/firefox/commit/3917f8be9c6f
https://hg.mozilla.org/integration/autoland/rev/783ad498988d
apply code formatting via Lando

Comment 24

[rperta]

https://hg.mozilla.org/mozilla-central/rev/1b9cb9824317
https://hg.mozilla.org/mozilla-central/rev/cfef9906f94d
https://hg.mozilla.org/mozilla-central/rev/783ad498988d