1796391 - Utility AudioDecoder not starting under MSIX

Status: VERIFIED FIXED

(Core :: Audio/Video: Playback, defect)

Description

[:gerard-majax]

STR:

1. Build m-c with EARLY_BETA_OR_EARLIER= in build/defines.sh and remove a1 in browser/config/version_display
2. mack repackage msix --sign
3. Install the MSIX package: https://firefox-source-docs.mozilla.org/browser/installer/windows/installer/MSIX.html#local-developer-builds
4. Open about:processes
5. Load https://paul.cx/public/z.mp3

Expected:
Audio plays

Actual:
No audio for several seconds then some audio. Looking in about:processes one can see no Utility AudioDecoder process and decoding is happening on Content process?

Comment 1

[:gerard-majax]

Attachment: Bug 1796391 - Force init signed policy rules for delayed mitigations on MSIX r?bobowen!

Comment 2

[:gerard-majax]

Attachment: 0001-Bug-1796391-Do-not-enforce-delayed-MITIGATION_FORCE_beta.patch

Comment 3

[:gerard-majax]

Attachment: 0001-Bug-1796391-Do-not-enforce-delayed-MITIGATION_FORCE_release.patch

Comment 4

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Core::Audio/Video: Playback' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 5

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1780796

Comment 6

[:gerard-majax]

Attachment: 0001-Bug-1796391-Do-not-enforce-delayed-MITIGATION_F_beta.patch

Comment 7

[:gerard-majax]

Attachment: 0001-Bug-1796391-Do-not-enforce-delayed-MITIGATIO_release.patch

Comment 8

[:gerard-majax]

Attachment: 0001-Bug-1796391-Always-init-signed-policy-rules-r-b_beta.patch

Comment 9

[:gerard-majax]

Attachment: 0001-Bug-1796391-Always-init-signed-policy-rules-_release.patch

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/89d3bd40e892
Always init signed policy rules r=bobowen

Comment 11

[:gerard-majax]

Attachment: 0001-Bug-1796391-Force-init-signed-policy-rules-for-_beta.patch

Comment 12

[:gerard-majax]

Attachment: 0001-Bug-1796391-Force-init-signed-policy-rules-f_release.patch

Comment 13

[Cristian Tuns]

Backout as requested by gerard-majax
Backout link: https://hg.mozilla.org/integration/autoland/rev/87a8a14e507fc1ff9d72c9c2bcd6d413c2765a7a

Comment 14

[:gerard-majax]

Comment on attachment 9299518 [details] [diff] [review]
0001-Bug-1796391-Force-init-signed-policy-rules-for-_beta.patch

Beta/Release Uplift Approval Request

• User impact if declined: bug 1795909: Firefox stops working completely
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: Build MSIX and normal package, install, try to play some MP3 and AAC.
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Simple change and manually tested by several people
• String changes made/needed: N/A
• Is Android affected?: No

https://treeherder.mozilla.org/jobs?repo=try&revision=357435bec733bf8253fe1aa9815fac4f80640420

Comment 15

[:gerard-majax]

Comment on attachment 9299519 [details] [diff] [review]
0001-Bug-1796391-Force-init-signed-policy-rules-f_release.patch

Beta/Release Uplift Approval Request

• User impact if declined: bug 1795909: Firefox stops working completely
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: Build MSIX and normal package, install, try to play some MP3 and AAC.
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Simple change and manually tested by several people
• String changes made/needed: N/A
• Is Android affected?: No

https://treeherder.mozilla.org/jobs?repo=try&revision=2c90ae829c0688754d3b391738cd30500433503b

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2a5a0f3b5111
Force init signed policy rules for delayed mitigations on MSIX r=bobowen

Comment 17

[Donal Meehan [:dmeehan]]

Comment on attachment 9299518 [details] [diff] [review]
0001-Bug-1796391-Force-init-signed-policy-rules-for-_beta.patch

Approved for 107.0b3.

Comment 18

[Donal Meehan [:dmeehan]]

https://hg.mozilla.org/releases/mozilla-beta/rev/485b82dcc037

Comment 19

[Iulian Moraru]

https://hg.mozilla.org/mozilla-central/rev/2a5a0f3b5111

Comment 20

[Brindusa Tot, Desktop QA]

Making custom Firefox builds is usually very time consuming for Desktop QA, which is why the team relies on Engineering providing these builds when verifying a fix requires them.
Alexandre, we'd be happy to take a look at this if you can provide an affected build for us to reproduce the initial issue, and a build that contains the patch so that we can verify it does not reproduce anymore.

Comment 21

[:gerard-majax]

Builds are on treeherder, search for msixs for 64 bits

Comment 22

[:gerard-majax]

And broken one can be installed manually from https://archive.mozilla.org/pub/firefox/releases/106.0/win64/multi/Firefox%20Setup%20106.0.msix

Comment 23

[Alexandru Trif, Desktop Test Engineering [:atrif]]

Reproduced the issue with Firefox 106.0 MSIX on Windows 10x64. Opening https://paul.cx/public/z.mp3 does plays the audio only after some loading and there is no Utility > Audio Decoder displayed inside about:processes.
The issue is verified fixed on Windows 10x64 and Windows 11x64 with Firefox 107.0b4 (20221023135421) MSIX and 108.0a1 (20221024093150) MSIX from treeeherder. The mp3 audio file and AAC audio files are played as expected.

On Windows 10x64 the Utility > Generic Audio Decoder is displayed inside about:processes with Firefox 107.0b4 and Utility > Windows Media Framework Audio Decoder is displayed for the latest Nightly. On Windows 11x64 the decoder the Windows Media Framework Audio Decoder is displayed inside about:processes with Firefox 107.0b4 and Utility > Generic Audio Decoder is displayed for the latest Nightly. I have also take a look at 105.0.3 MSIX and it seems that it shows Utility Audio Decoder inside about:processes. Just to be extra safe here, does the decoder name matter or it's just important that the decoder is displayed inside about:processes when playing the mp3/aac files? Thank you!

Comment 24

[:gerard-majax]

(In reply to Alexandru Trif, QA [:atrif] from comment #23)

Reproduced the issue with Firefox 106.0 MSIX on Windows 10x64. Opening https://paul.cx/public/z.mp3 does plays the audio only after some loading and there is no Utility > Audio Decoder displayed inside about:processes.
The issue is verified fixed on Windows 10x64 and Windows 11x64 with Firefox 107.0b4 (20221023135421) MSIX and 108.0a1 (20221024093150) MSIX from treeeherder. The mp3 audio file and AAC audio files are played as expected.

I dont think beta4 is a good candidate, because you need EARLY_BETA_OR_LATER and it's switched between beta6 and beta7. That's why I mentionned the need for builds.

On Windows 10x64 the Utility > Generic Audio Decoder is displayed inside about:processes with Firefox 107.0b4 and Utility > Windows Media Framework Audio Decoder is displayed for the latest Nightly. On Windows 11x64 the decoder the Windows Media Framework Audio Decoder is displayed inside about:processes with Firefox 107.0b4 and Utility > Generic Audio Decoder is displayed for the latest Nightly. I have also take a look at 105.0.3 MSIX and it seems that it shows Utility Audio Decoder inside about:processes. Just to be extra safe here, does the decoder name matter or it's just important that the decoder is displayed inside about:processes when playing the mp3/aac files? Thank you!

You should see both decoders. But about:processes can take some time to update, you might need to reload.

Comment 25

[Alexandru Trif, Desktop Test Engineering [:atrif]]

(In reply to Alexandre LISSY :gerard-majax from comment #24)

I dont think beta4 is a good candidate, because you need EARLY_BETA_OR_LATER and it's switched between beta6 and beta7. That's why I mentionned the need for builds.

I will check this on 107.0b7 as well then.

You should see both decoders. But about:processes can take some time to update, you might need to reload.

Yup, you are right, I somehow missed it in the first place. Thank you!

Comment 26

[Pascal Chevrel:pascalc]

Comment on attachment 9299519 [details] [diff] [review]
0001-Bug-1796391-Force-init-signed-policy-rules-f_release.patch

Approved for 106.0.2, thanks.

Comment 27

[Pascal Chevrel:pascalc]

https://hg.mozilla.org/releases/mozilla-release/rev/3f8994667e76

Comment 28

[Alexandru Trif, Desktop Test Engineering [:atrif]]

Verified the issue with Firefox 106.0.2 (20221024211236) MSIX from comment 27 on Windows 10x64 21H1 and Windows 11x64 21H2. The mp3 audio file and AAC audio files are played as expected and both Utility > Generic Audio Decoder and Windows Media Framework Audio Decode are displayed inside about:processes after playing the sound. I will leave this open and reverify this again on 107.0b7 when available and on 106.0.2 once it lands officially on Microsoft Store.

Comment 29

[Alexandru Trif, Desktop Test Engineering [:atrif]]

Verified this again with Firefox 106.0.2 (20221025065831) MSIX installed from Microsoft Store on two Windows 11x64 and two Windows 10x64 machines. The build is started accordingly and the mp3 audio file and AAC audio files are played as expected. Both Utility > Generic Audio Decoder and Windows Media Framework Audio Decode are displayed inside about:processes after playing the sound.

Comment 30

[Yannis Juglaret [:yannis]]

Just for archiving -- our current understanding of this problem is as follows. Our code contained paths where it will try to load non-Microsoft-signed libraries with the ProcessSignaturePolicy process mitigation policy being active. On regular installs and in our CI, failures with this mitigation are silent, they just generate an ETW event and make LoadLibrary return NULL -- and our code was relying on this behavior. But given our current config for the MSIX package, these failures seem doubly fatal, meaning they provoke the two things described below.

1. They set the package status to 2, which implies that next time we try to launch the app we will not even try to start Firefox and just fail.
   Here is a part of the corresponding kernel call stack caught in procmon and manually annotated:

    0xfffff80763d78dd3 0x1c0008dd3 CI.dll KappxpSetPackageStatus
    0xfffff80763d79481 0x1c0009481 CI.dll KappxpNotifyPackagedFile
    0xfffff80763df1e2c 0x1c0081e2c CI.dll KappxNotifyIntegrityFailureInPackagedProcess
    0xfffff80763dc7a17 0x1c0057a17 CI.dll CipReportAndReprieveUMCIFailure
    0xfffff80763dc2daa 0x1c0052daa CI.dll CiValidateImageHeader
    (... 00000034`cf3fd5a0 ntdll.dll LdrLoadDll)

2. They report an exception -- which maybe crashes our process although we are not 100% sure of the flow of that exception.
   Here is a part of the corresponding userland call stack caught in WinDbg:

    00 00000034`cf3fc868 00007fff`0b637c19 ntdll!NtRaiseException+0x14
    01 00000034`cf3fc870 00007fff`0b636cc8 ntdll!WerpBreakIntoDebuggerIfPresent+0x2d
    02 00000034`cf3fc8a0 00007fff`0b6267b5 ntdll!RtlReportException+0x38
    03 00000034`cf3fc920 00007fff`0b5f9f15 ntdll!LdrAppxHandleIntegrityFailure+0x1b5
    04 00000034`cf3fd060 00007fff`0b561770 ntdll!LdrpMapDllNtFileName+0x99321
    05 00000034`cf3fd160 00007fff`0b56153f ntdll!LdrpMapDllFullPath+0xe0
    06 00000034`cf3fd2f0 00007fff`0b578fd4 ntdll!LdrpProcessWork+0x77
    07 00000034`cf3fd340 00007fff`0b56932c ntdll!LdrpLoadDllInternal+0x1a0
    08 00000034`cf3fd3e0 00007fff`0b57a95a ntdll!LdrpLoadDll+0xb0
    09 00000034`cf3fd5a0 00007ff6`5bdf6e8e ntdll!LdrLoadDll+0xfa

The proposed patches appear to have fixed the problem.

Comment 31

[Alexandru Trif, Desktop Test Engineering [:atrif]]

Verified fixed with Firefox 107.0b9 (20221102185936) MSIX installed from treeherder on Windows 11x64 and Windows 10x64. The build is started accordingly and the mp3 audio file and AAC audio files are played as expected. Both Utility > Generic Audio Decoder and Windows Media Framework Audio Decode are displayed inside about:processes after playing the sound.