1751450 - Unsigned DLLs used during per-user installation on Windows, prevents use of AppLocker

Status: RESOLVED FIXED

(Firefox :: Installer, defect, P3)

Description

[nopsled9090]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36

Steps to reproduce:

Apply AppLocker rules allowing EXE and DLL files signed by Microsoft or Mozilla.
Run latest "Firefox Installer.exe", ignore the UAC prompt and install for the user.
If AppLocker is in block mode, the installation fails;
If AppLocker is in audit mode, the installation succeeds but Windows logs a bunch of events indicating that operations would have failed in block mode. The following unsigned DLLs are used during Firefox installation. They should be signed by Mozilla. Note that these are only part of the installer - the files that get installed are all signed. The problem is that the installer and any updates will fail when app control is in effect. It's not feasible to use hash rules to allow those DLLs because each time the files get rebuilt their hashes change.
ACCESSCONTROL.DLL
APPLICATIONID.DLL
CITYHASH.DLL
INVOKESHELLVERB.DLL
NSJSON.DLL
SHELLLINK.DLL
SYSTEM.DLL
CERTCHECK.DLL
INETBGDL.DLL
UAC.DLL
USERINFO.DLL
WEBBROWSER.DLL

Actual results:

Installation fails because the unsigned DLLs are blocked by application control.

Expected results:

Installation should succeed if the DLLs used by the installer are signed by Mozilla the way most of the Mozilla files are.

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Firefox::Installer' component, and is moving the bug to that component. Please revert this change in case you think the bot is wrong.

Comment 2

[Nick Alexander :nalexander [he/him]]

Thanks for the clear report. It's my understanding that these unsigned DLLs are packed into the installer EXE, which is itself signed. They are then unpacked before being consumed, opening a window for an adversary to replace or modify them. That changes any security consideration: there's no "data at rest" problem where-by an adversary can impact Firefox users at scale by distributing a compromised Firefox installer, only a race condition that can potentially be exploited by an adversary with write-access to the targeted user's device.

I think there are many, many ways for an adversary with write-access to the targeted user's device to inject code into the Installer process; this may not change that reality. So I'm not sure we will care to fix this. But out of an abundance of caution, I've marked this as a sec ticket and we will work with our sec team to triage this.

Thanks again!

Comment 3

[Nick Alexander :nalexander [he/him]]

I will add that these DLLs are in-tree, for historical reasons; we could sign the versions in tree if we wanted to. One reason we might not want to is that we sign different Firefox versions with different keys, and that would be quite hard to accommodate.

Bug 1616712 might be relevant here: that meta ticket talks about producing the DLLs, not signing them, but it's in the same vein...

Comment 4

[nopsled9090]

The issue is not about an adversary replacing the DLLs and compromising the installation. It's that an organization that wants to implement application control (e.g., using AppLocker or Windows Defender Application Control) and wants to create rules allowing end users to install the Firefox browser on their own cannot do so. It's easy to create rules for files that are digitally signed by Mozilla and have the product name "Firefox" in the version resource, and those rules will work for future installers. But files that aren't digitally signed must be identified by their hashes, which change every time they're built again, so it's impossible to create rules ahead of time for future versions.

Comment 5

[Nick Alexander :nalexander [he/him]]

(In reply to nopsled9090 from comment #4)

The issue is not about an adversary replacing the DLLs and compromising the installation. It's that an organization that wants to implement application control (e.g., using AppLocker or Windows Defender Application Control) and wants to create rules allowing end users to install the Firefox browser on their own cannot do so. It's easy to create rules for files that are digitally signed by Mozilla and have the product name "Firefox" in the version resource, and those rules will work for future installers. But files that aren't digitally signed must be identified by their hashes, which change every time they're built again, so it's impossible to create rules ahead of time for future versions.

I hear your use case. I'm not sure it's particularly common. My concern is that unsigned binaries have security implications.

Comment 6

[Nick Alexander :nalexander [he/him]]

mhowell: has signing the NSIS plugin DLLs ever come up?

Comment 7

[nopsled9090]

BTW - all the EXE and DLL files that the installer leaves on the system (the browser executable files) are all signed, and all but two of those files that are signed by Mozilla have the product name "Firefox" in the version resource. So it's easy to create AppLocker/WDAC rules for Firefox once it's installed, but it's impossible to create rules for the installer or (AFAIK) any updater.

Bottom line is that this is an important bug to fix and one could consider it a security ISSUE, but it's not a security VULNERABILITY in the product.

Comment 8

[nopsled9090]

(In reply to Nick Alexander :nalexander [he/him] from comment #5)

I hear your use case. I'm not sure it's particularly common. My concern is that unsigned binaries have security implications.

Application control is becoming increasingly important, and it's also being made more available. See https://github.com/Microsoft/AaronLocker which has gotten a fair amount of adoption and interest.

Comment 9

[Molly Howell (she/her) (mostly inactive)]

(In reply to Nick Alexander :nalexander [he/him] from comment #6)

mhowell: has signing the NSIS plugin DLLs ever come up?

Hmm. Not that I can recall. I don't know of any technical reason why we couldn't sign them though. It's not like any of those are generated at runtime or anything, they're just extracted out of the archive. The only problem that immediately comes to mind is that some of those files come from the NSIS distribution, they aren't all built by us, so we'd end up putting Mozilla signatures on unmodified NSIS files. I'm not sure if that's something we should really get into or not.

Comment 10

[nopsled9090]

That's a good point about signing files built elsewhere. One could state, though, that a validated signature doesn't assert "We compiled and linked this from scratch," but instead asserts, "This file has not been modified since we signed it and shipped it." If the NSIS files you incorporate into the installer have problems, they're still your problems whether you sign them or not.
For the record, though, IANAL.

Comment 11

[[e-mail] (:bhearsum)]

(In reply to nopsled9090 from comment #4)

The issue is not about an adversary replacing the DLLs and compromising the installation. It's that an organization that wants to implement application control (e.g., using AppLocker or Windows Defender Application Control) and wants to create rules allowing end users to install the Firefox browser on their own cannot do so. It's easy to create rules for files that are digitally signed by Mozilla and have the product name "Firefox" in the version resource, and those rules will work for future installers. But files that aren't digitally signed must be identified by their hashes, which change every time they're built again, so it's impossible to create rules ahead of time for future versions.

I realize you're specifically asking about the current exe installer, but if it works for them, you could consider directing your users to the new-ish MSIX package instead. It's available through the Microsoft Store.

Comment 12

[Nick Alexander :nalexander [he/him]]

(In reply to Molly Howell (she/her) [:mhowell] from comment #9)

(In reply to Nick Alexander :nalexander [he/him] from comment #6)

mhowell: has signing the NSIS plugin DLLs ever come up?

Hmm. Not that I can recall. I don't know of any technical reason why we couldn't sign them though. It's not like any of those are generated at runtime or anything, they're just extracted out of the archive. The only problem that immediately comes to mind is that some of those files come from the NSIS distribution, they aren't all built by us, so we'd end up putting Mozilla signatures on unmodified NSIS files. I'm not sure if that's something we should really get into or not.

This would be an interesting attack vector. Suppose Firefox, say XUL.dll, composes a third-party library subsequently found to be vulnerable. An adversary could load Firefox's trusted XUL.dll to access the vulnerable functionality, with said functionality signed by Mozilla. I don't think this is worth worrying about: said adversary could presumably just compose the vulnerable code themselves, since they already have some code execution ability.

Given that, it's not clear to me what the ramifications of signing an NSIS plugin that we didn't compile could be. I suppose it's now possible to get a small unit of code that chains to a trust root accepted in default Windows configurations? Again, an adversary needs exececute to do something with that.

I think if we're going to address this, the only cost-effective solution will be to sign the in-tree plugin DLLs, presumably with Mozilla's release signing key. I can imagine schemes that are more granular but they're going to be challenging to implement.

bhearsum: can you suggest who in relman/releng would have an opinion or otherwise bless signing the DLLs in tree?

Comment 13

[Molly Howell (she/her) (mostly inactive)]

(In reply to Nick Alexander :nalexander [he/him] from comment #12)

Given that, it's not clear to me what the ramifications of signing an NSIS plugin that we didn't compile could be. I suppose it's now possible to get a small unit of code that chains to a trust root accepted in default Windows configurations? Again, an adversary needs exececute to do something with that.

Worth noting here is that one of the plugins I'm talking about is System, which is basically arbitrary code execution as a library. It's tricky to call it though, because you'd have to prepare its arguments in the form of an NSIS stack, so I'm not sure if it could be done from rundll32 or anything like that.

Comment 14

[[e-mail] (:bhearsum)]

(In reply to Nick Alexander :nalexander [he/him] from comment #12)

bhearsum: can you suggest who in relman/releng would have an opinion or otherwise bless signing the DLLs in tree?

Aki is the best starting point from RelEng, but Hal probably wants to be looped in here as well.

I want to add a couple of things as well:

• We don't have any examples of signing something we didn't build, but we do have something close-ish. The OpenH264 plugins that we use are "built" by us, but only in the sense of compiling them. The source comes largely from Cisco (with a few tweaks, it appears). So, this may not be entirely untrodden ground.
• I briefly looked at how we might do this, and I think it fits in with the current set of tasks. We already post the unsigned builds in setup.exe from the unsigned Build task, so we should be able to use the existing Build Signing task to sign. We'd have to adjust where downstream tasks get that from, and maybe a few other things but it seems largely non-invasive.

Comment 15

[Aki Sasaki (not active)]

Some thoughts:

First, I don't think there will be issues here, because we distribute the DLLs and sign the outer installers, but IANAL. Hopefully any vetting of these DLLs we have to do have already been done before we decided to distribute them.

Second, our authenticode key is shared between nightly and release, so if that's the concern, we don't have to worry about that. If the concern is around the dep/production split, then yes, this may be valid.

Third, we could use a toolchain approach here, if these files don't change regularly. These tasks would rebuild every ~1 year or on source change. For example, we sign the geckodriver toolchain binary in this task. We'd need to download that binary to use in our repackage or other downstream task.

Fourth, in the build task, we create a zipfile and a stub and non-stub installer EXE, which we then sign. It looks like at least some of the DLLs specified in this bug are included in the installer EXEs. The signing logic explodes zipfiles (skipping some blocklisted filename patterns and signs the inner files, but only signs the outer file for EXEs. If the DLLs are included in the above files but skipped due to that logic, we could either alter that logic, or create zipfiles, and sign the innards, and sign the repackaged EXE in a 3rd task. (If the setup EXEs need to be included in the repackaged installer, then we may need additional tasks; hoping not.)

Signing the files and landing them in-tree seems like the least amount of work here, if there are no significant concerns.

Comment 16

[Hal Wine [:hwine] use NI!]

From the signing infrastructure perspective, I don't see any secops issues.

How confident are we that the in-tree items would not need resigning when we renew the code-signing cert? Certainly, we could do so, but only if it makes it onto the "renew the cert" checklist :)

Comment 17

[nopsled9090]

When a signature is timestamped, the signature remains valid even after the code-signing certificate has expired, so there should be no need to re-sign the files.

Comment 18

[nopsled9090]

(... and the signatures of Firefox signed files are all timestamped...)

Comment 19

[Aki Sasaki (not active)]

Most likely we will sign these files through adhoc-signing. We may want to wait for the results of bug 1745467 first.

Comment 20

[nopsled9090]

How long do you anticipate the process might take? Could it be released in days? Weeks? Months?
Thanks to all of you for the work you're doing on this.

Comment 21

[Nick Alexander :nalexander [he/him]]

I am fine with ad-hoc signing and putting these DLLs directly into the tree. On Slack, :bhearsum asked if and how this would impact updating these DLLs. These DLLs are very infrequently updated, so I'm not particularly concerned about the ongoing cost of ad-hoc signing. As to whether updates would regress the signing status, I think we could add a test (or a lint) ensuring that these DLLs keep a signature and that it's valid for some period into the future -- say 6 or 12 months. So if ad-hoc signing meets with approval more broadly, it makes sense to me.

Comment 22

[Nick Alexander :nalexander [he/him]]

(In reply to nopsled9090 from comment #20)

How long do you anticipate the process might take? Could it be released in days? Weeks? Months?
Thanks to all of you for the work you're doing on this.

Weeks to months, with no particular guarantee. Remember that the time from landing through to making it to Firefox Release is also 1-2 months.

Comment 23

[nopsled9090]

"ensuring that these DLLs keep a signature and that it's valid for some period into the future -- say 6 or 12 months." --> the signatures are timestamped so they should remain valid indefinitely, shouldn't they?

Comment 24

[[e-mail] (:bhearsum)]

I am fine with ad-hoc signing and putting these DLLs directly into the tree. On Slack, :bhearsum asked if and how this would impact updating these DLLs. These DLLs are very infrequently updated, so I'm not particularly concerned about the ongoing cost of ad-hoc signing. As to whether updates would regress the signing status, I think we could add a test (or a lint) ensuring that these DLLs keep a signature and that it's valid for some period into the future -- say 6 or 12 months. So if ad-hoc signing meets with approval more broadly, it makes sense to me.

I'm a bit more concerned about newly added DLLs, but any test that checks the signature on everything in the directory would probably catch that too. Maybe a README would help, too.

Comment 25

[nopsled9090]

If you want, I can write up the repro details for testing against an AppLocker policy that allows Mozilla-signed binaries. If provided with a candidate build, I can also perform tests on my own system and report results. Happy to help out in any way I can.

Comment 27

[Mike Kaply [:mkaply]]

This is apparently going to become more urgent because of Smart App Control

https://support.microsoft.com/en-us/topic/what-is-smart-app-control-285ea03d-fa88-4d56-882e-6698afdb7003

We have reports from a partner that a Firefox fire was blocked during install.

Smart App Control is being rolled out in Windows 11 2H22

Comment 28

[Mike Kaply [:mkaply]]

Attachment: Screenshot of errors

Here's some screenshots of the errors as well as the error from the Error console.

The errors showed up when doing an update from the about dialog.

Comment 29

[:Gijs (he/him)]

Flagging this for re-triage based on mkaply's comments.

Comment 30

[Mike Kaply [:mkaply]]

Our partner is going to need this fairly quickly due to shipping schedules.

Comment 31

[Nick Rishel [:nrishel]]

Discussed in our weekly meeting, assigning to bhearsum

Comment 32

[[e-mail] (:bhearsum)]

(In reply to Aki Sasaki [:aki] (he/him) (UTC-6) from comment #15)

First, I don't think there will be issues here, because we distribute the DLLs and sign the outer installers, but IANAL. Hopefully any vetting of these DLLs we have to do have already been done before we decided to distribute them.

Given the new reasons to fix this soon, I'd like to consider taking this option - at least for the very short term. The DLLs in question were vetted when they were landed, and have been shipped for a number of years -- signing does not change the risks here IMO.

Doing this would mean that try builds would also contain these DLLs signed with our release cert. However, you wouldn't be able to push to try and get new code signed (because these would be pre-signed DLLs landed in the tree).

What do you need to get this moving? I'd be happy to make an adhoc-signing pull request if it's helpful.

Third, we could use a toolchain approach here, if these files don't change regularly. These tasks would rebuild every ~1 year or on source change. For example, we sign the geckodriver toolchain binary in this task. We'd need to download that binary to use in our repackage or other downstream task.

I certainly wouldn't object to this, but these files are so rarely changed or added that it may not be worth it. It could be multiple years before we need to sign anything again.

Comment 33

[Aki Sasaki (not active)]

What do you need to get this moving? I'd be happy to make an adhoc-signing pull request if it's helpful.

+1, this seems like the most expedient route to fixing this bug.

Comment 34

[[e-mail] (:bhearsum)]

Attachment: Bug 1751450: update all NSIS plugin DLLs with signed copies.

Comment 35

[Mike Kaply [:mkaply]]

Do we have any idea what the UserInfo.dll referenced in the error message is? That one isn't coming from our files.

Comment 36

[[e-mail] (:bhearsum)]

(In reply to Mike Kaply [:mkaply] from comment #35)

Do we have any idea what the UserInfo.dll referenced in the error message is? That one isn't coming from our files.

It looks like this is one of the plugins packaged with NSIS (not checked into our tree). Which we can probably also find a way to sign -- thanks for flagging this part explicitly.

Comment 37

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4da0a636e40e
update all NSIS plugin DLLs with signed copies. r=mhowell

Comment 38

[Nick Alexander :nalexander [he/him]]

(In reply to [e-mail] (:bhearsum) from comment #36)

(In reply to Mike Kaply [:mkaply] from comment #35)

Do we have any idea what the UserInfo.dll referenced in the error message is? That one isn't coming from our files.

It looks like this is one of the plugins packaged with NSIS (not checked into our tree). Which we can probably also find a way to sign -- thanks for flagging this part explicitly.

In Slack, Ben suggests modifying the NSIS fetch task to pull a ZIP that we host and contains ad-hoc signed copies of the NSIS plugins (including UserInfo.dll). This sounds like a reasonable way to solve the immediate issue.

Comment 39

[[e-mail] (:bhearsum)]

Attachment: nsis-3.07-signed.zip

This is a copy of the NSIS 3.07 that we're currently using that contains signed versions of the embedded DLLs (both the ANSI and Unicode versions).

The signing was done through the RelEng adhoc signing system in this revision: https://github.com/mozilla-releng/adhoc-signing/commit/9dd8495695495f612f7adfb58c8f922958486390

Comment 40

[[e-mail] (:bhearsum)]

Attachment: Bug 1751450: pull NSIS Windows source from our own rehosted version, with signed DLLs. r?nalexander!,aki!

This is a short term measure. In the medium or long term we can do this signing in a separate Task after pulling the original source.

Comment 41

[Nick Alexander :nalexander [he/him]]

mhowell: would you mind either verifying yourself that the signatures in the DLLs are the only changes in:

https://bugzilla.mozilla.org/attachment.cgi?id=9278166
https://downloads.sourceforge.net/project/nsis/NSIS%203/3.07/nsis-3.07.zip

Or suggesting how I might do so? Thanks!

Comment 42

[Molly Howell (she/her) (mostly inactive)]

What I'm doing is just opening each pair of files in a hex editor that has a comparison function and looking at the diff in that form. That shows me that the only differences are the relevant bits of the PE optional header, namely the file checksum and the offset/length for the signature table, and the addition at the end of the file of a block of data which forms the signature itself. It so happens that I know from experience what these things look like in a hex editor, so that's a fast way for me to do this. The specific tool I'm using is Beyond Compare, because it has a nice UI for going through a whole directory quickly, but any hex editor with a file compare tool works (personally I like HxD).

Comment 43

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b46b70eeaa6f
pull NSIS Windows source from our own rehosted version, with signed DLLs. r=nalexander,aki

Comment 44

[Cristian Tuns]

Backed out for causing fetch bustages

• Backout link
• Push with failures
• Failure Log
• Failure line: Exception: Archive type not supported for attachment.cgi

Comment 45

[Nick Alexander :nalexander [he/him]]

glandium: purely for your awareness: we've made some awkward temporary changes to sign parts of the NSIS toolchain. I know you've done work in the area and are probably closest to the owner of this toolchain so I wanted you to be aware. We'll try to get a better signing solution in H2.

Comment 46

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/4da0a636e40e

Comment 47

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4832094b40da
pull NSIS Windows source from our own rehosted version, with signed DLLs. r=nalexander,aki

Comment 48

[[e-mail] (:bhearsum)]

Comment on attachment 9278150 [details]
Bug 1751450: update all NSIS plugin DLLs with signed copies.

Beta/Release Uplift Approval Request

• User impact if declined: Some users will not be able to install Firefox; impact to partners
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: Ensure installer still works. (I have already done this on autoland.)
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): The only changes here are signing of some DLLs that we already ship -- the DLLs themselves are otherwise identical.
• String changes made/needed:
• Is Android affected?: No

Comment 49

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9278150 [details]
Bug 1751450: update all NSIS plugin DLLs with signed copies.

Approved for 101.0rc2.

Comment 50

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-release/rev/958beba6884b
https://hg.mozilla.org/releases/mozilla-release/rev/cf10641e190b

Comment 51

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/4832094b40da

Comment 52

[[e-mail] (:bhearsum)]

Just to make it crystal clear in the final comment: what we did here was a short term fix. The medium term fix for this is tracked in https://bugzilla.mozilla.org/show_bug.cgi?id=1771192

Comment 53

[Catalin Sasca, Desktop Test Engineering [:csasca]]

Tried reproducing the issue on Windows 11 by adding rules for the installer executable and dlls provided by Mozilla (and enforcing them too), but had no luck in seeing the issues while installing Firefox. Nopsled, could you provide a step by step on how to easily reproduce the issue? Thank you!

Comment 54

[nopsled9090]

(In reply to Catalin Sasca, QA [:csasca] from comment #53)

Tried reproducing the issue on Windows 11 by adding rules for the installer executable and dlls provided by Mozilla (and enforcing them too), but had no luck in seeing the issues while installing Firefox. Nopsled, could you provide a step by step on how to easily reproduce the issue? Thank you!

In addition to the repro steps described in my initial submission:

• AppLocker is configurable through local/domain GPO only on Enterprise SKUs, not on Win11 Pro or Home.

• Make sure you are using only Publisher rules for the Firefox files - don't create hash rules for the unsigned files.

• Make sure that AppLocker is fully operational before trying to repro - specifically:

  • Make sure you configure the appidsvc to be automatic-start and ensure that it's running
  • Take a look in the AppLocker EXE/DLL event log and make sure that AppLocker events are being logged:
    Applications and Services Logs\Microsoft\Windows\AppLocker\EXE and DLL
    You should see at least a bunch of Information events for all the EXE and DLL files that are loaded for execution. If you don't, then AppLocker isn't working yet.

Comment 55

[nopsled9090]

(In reply to Catalin Sasca, QA [:csasca] from comment #53)

Tried reproducing the issue on Windows 11 by adding rules for the installer executable and dlls provided by Mozilla (and enforcing them too), but had no luck in seeing the issues while installing Firefox. Nopsled, could you provide a step by step on how to easily reproduce the issue? Thank you!

Oh - I assume you already know how to configure AppLocker rules through the Local Security Policy GUI, including how to enable and enforce DLL rules? If you don't enforce the DLL rules, you won't reproduce the issue. The problem is with unsigned DLLs, not EXEs.

Comment 56

[nopsled9090]

(In reply to Catalin Sasca, QA [:csasca] from comment #53)

Tried reproducing the issue on Windows 11 by adding rules for the installer executable and dlls provided by Mozilla (and enforcing them too), but had no luck in seeing the issues while installing Firefox. Nopsled, could you provide a step by step on how to easily reproduce the issue? Thank you!

Just tried again. It looks like the latest published Firefox installer download includes the bug fix described above, so this bug will no longer repro.

Comment 57

[Catalin Sasca, Desktop Test Engineering [:csasca]]

Thank you for the clarifications. I tried reproducing only on the Pro version of Windows 11. Also thanks for verifying that it no longer repro. If you have time can you verify as well on 102 ? (I assume you referred to 101 as the latest published Firefox installer).