1745467 - Firefox 91 ESR Setup for Windows shipping with vulnerable SHA1 digital signature

Status: VERIFIED FIXED

(Release Engineering :: General, defect)

Description

[javascriptjedi]

User Agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0

Steps to reproduce:

Downloaded Firefox Setup 91.4.0esr.exe from Mozilla website (ftp).

Actual results:

Upon inspection of installer package, I found that digital signature was obsolete, vulnerable SHA1 digest.

Expected results:

You should be using SHA256 digest. You do provide SHA256 sums for these executables in the ftp download tree, but the signature on the EXE itself is still SHA1.

Here's a link about SHA1 being vulnerable from 2019:
https://www.computerworld.com/article/3173616/the-sha1-hash-function-is-now-completely-unsafe.html

Comment 1

[:Gijs (he/him)]

bhearsum, can you forward this to the right folks?

Comment 2

[[e-mail] (:bhearsum)]

I will note that we are signing with a SHA256 certificate, just not with SHA256 digests.

This is a RelEng issue, but IIRC, the reason we hadn't upgraded yet was because we were supporting XP SP2 until fairly recently, which did not support sha256 digests. Our minimum version now is Windows 7, so it's possible we may be able to change this now.

There is some context for this scattered in https://bugzilla.mozilla.org/show_bug.cgi?id=1079858.

Comment 3

[Frederik Braun [:freddy]]

This bug has just been reported again from an external contributor who I am giving access to the bug with this comment.
Aki, can you help getting this triaged?

Comment 5

[saelic]

(In reply to [e-mail] (:bhearsum) from comment #2)

This is a RelEng issue, but IIRC, the reason we hadn't upgraded yet was because we were supporting XP SP2 until fairly recently, which did not support sha256 digests. Our minimum version now is Windows 7, so it's possible we may be able to change this now.

I would like to point out two things:

1. The last version of Firefox which supported Windows XP and Vista was 52.9.0esr, which, according to [1][2], was released on 2018-06-26 and which is not being supported since 2018-09-05. This should be the last version which used SHA-1 hash in Authenticode.
   The next version of Firefox, namely version 60, which was released on 2018-01-23, already did not support Windows XP and Vista and therefore should be the first version which was signed using SHA-256 or higher. So we are 4 years backwards on this.

2. Even if for some reason you want to keep the SHA-1 based signature, then you can dual-sign the setup file with both SHA-1 and SHA-256 signatures - as many of Windows XP compatible programs do to this day. Look at [3] for reference.

In conclusion, we should switch to SHA-256 (or even SHA-512, there is no reason why not) RIGHT AWAY, without any further considerations. This step should be made 4 years ago.

[1] https://support.mozilla.org/en-US/kb/firefox-esr-release-cycle
[2] https://en.wikipedia.org/wiki/Firefox_version_history#Firefox_52_through_59
[3] https://textplain.files.wordpress.com/2016/01/image6.png?w=380&h=142

Comment 6

[[e-mail] (:bhearsum)]

I am no longer with the RelEng team so I can't comment on current priorities, but I'd like to politely provide a little bit of additional background.

RelEng generally has many competing priorities that are not always obvious from the outside. Additionally, changes like this are not usually a matter of just flipping a switch - our signing is done with non-traditional tools to support our scale, which often means new feature has to be done on them to support things like this.

And this is perhaps less relevant today, but I do want to note that we did look into signing with both algorithms when we originally switched to SHA256 certificates and found that versions of Windows that we supported at the time did not cope with them correctly. There's a bunch of background on this starting with https://bugzilla.mozilla.org/show_bug.cgi?id=1079858#c18.

Comment 7

[saelic]

I don't blame anyone for this, I'm sure that you know what you are doing.

But I can also assure you, that to this day, 2022-01-19:

1. Windows 7 and later correctly support dual-signed files with SHA-1, SHA-256, SHA-384 and SHA-512.
2. Windows NT 4, 2000, XP, 2003 and Vista can correctly validate MD5 and SHA-1 Authenticode signatures (not applicable here, just for reference for older topics).

Comment 8

[Geoff Brown [:gbrown]]

The bug title and original report specify a particular release (which is great -- I appreciate the specific example), but I note the SHA1 digest is used for all Window releases, for both the installer and stub installer.

Comment 9

[Geoff Brown [:gbrown]]

I don't know the signing code well, but I'm learning. So far I note that it is easy to use either SHA1 or SHA256 digests, but not both. It appears there is a way to configure SHA1+SHA256, but the result only has a SHA1 digest. I'm investigating why...

Comment 10

[saelic]

Skip SHA-1 entirely! One signature based on SHA-256 or SHA-512 is enough.
Every Microsoft binary is signed only with SHA-256 for example.

I would personally go with SHA-512 because there is no reason why not to choose stronger algorithm in this case.

Comment 11

[Geoff Brown [:gbrown]]

Sorry for the long delay here -- there have been some unexpected interrupts!

Things I've learned:

• Firefox needs to support Windows 7, Windows 8, Windows 10, and Windows 11
• I believe Windows 8, Windows 10, and Windows 11 always supported SHA-256
• Windows 7 did not initially support SHA-256, but that was a long time ago. Updates are available for Windows 7 that provide SHA-256 support, and those updates have been required on Windows 7 since 2019 in order to support, for instance, Windows updates: https://support.microsoft.com/en-us/topic/2019-sha-2-code-signing-support-requirement-for-windows-and-wsus-64d1c82d-31ee-c273-3930-69a4cde8e64f
• Our signing framework (signingscript, winsign, autograph) has good existing support for SHA-1 or SHA-256 digests but practically no support for dual signing. Adding SHA-1 + SHA-256 dual signature support seems very complicated.
• Prior art (https://bugzilla.mozilla.org/show_bug.cgi?id=1079858#c24) suggests that dual signatures may not always work.

All of this is pointing to just switching from the SHA-1 to SHA-256 digest, with some QA to double-check our Windows 7 beliefs. I'm trying to make that happen as fast as I can.

Comment 12

[Geoff Brown [:gbrown]]

I came across bug 1531165, with both reassuring test results and outstanding concerns about unpatched Windows 7 users.

Comment 13

[saelic]

Why do you even care about users of unsupported version of Windows? Windows 7 is out of support sice January 2020, those who bought paid support till January 2023 are obligated to install necessary patches to be able to install updates, so they have SHA-256 support because of this.
If somebody doesn't will to update, it's their problem and the Firefox project must not suffer from this in any way.

Comment 14

[Geoff Brown [:gbrown]]

Attachment: Bug 1745467 - Update signing formats for Windows; r=

Change signing formats to use SHA-256 digests in the Windows installer signature.

Comment 15

[Geoff Brown [:gbrown]]

:avaida - There are concerns that this change will cause problems for the Windows installer on unpatched Windows 7 machines. We think it will be fine on Windows 7 with SP1. Can you help us verify the change on Windows 7 with/without SP1?

There is a try run at https://treeherder.mozilla.org/jobs?repo=try&revision=e6c632947d103713578289de541730b28ead13e4 and we are specifically concerned about the signed installers, like https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/AqrPZh2sT1yJ3f5fQkuOTQ/runs/0/artifacts/public/build/target.stub-installer.exe .

Comment 16

[saelic]

I'm always ready to help in the area of security, but in this case I need very detailed explanation, why there is a need to support Windows 7 and Windows 7 SP1.
Both of those systems are not supported by Microsoft for general public. Those who paid for paid support till January 2023 are required to install patch which adds support for Authenticode signatures based on SHA-2. I don't see any reason to support unpatched systems. Just as Windows XP and Windows Vista support was dropped long time ago, this is the time to look at Windows 7 support in very narrow manner.

Checkout the latest, paid patch for Windows 7: http://download.windowsupdate.com/c/msdownload/update/software/secu/2022/01/windows6.1-kb5010451-x64_992659d57fa1fab0b083f29b7e323cf5cb856f24.msu
It is signed with SHA-256 ONLY. So if Microsoft is doing that I don't see any reason that Mozilla should be different in this area.

Comment 17

[Alexandru Trif, Desktop QA [:atrif]]

Hello! We verified that Firefox is properly installed using builds from comment 15 on Windows 7x64 SP1 Ultimate (build 7601)/ Windows 7x64 SP1 Professional (build 7601) and Windows 7x64 Professional (build 7600) installed under VM. Following installers were tested on all above-mentioned systems:

• target.installer.exe x86 and target.stub-installer.exe from Windows 2012 Shippable-> rs (B)
• target.installer.exe x64 from Windows 2012 x64 Shippable-> rs (B)
• target.installer.msi x86 from Windows 2012 Shippable MSis (N) and target.installer.msi x64 from Windows 2012 x64 Shippable-> MSis (N)

Geoff Brown, is there anything else that should be tested here? If there is please let us know. Thank you!

Comment 18

[Ryan VanderMeulen [:RyanVM]]

(In reply to saelic from comment #16)

I'm always ready to help in the area of security, but in this case I need very detailed explanation, why there is a need to support Windows 7 and Windows 7 SP1.

~18% of our users are still on Windows 7, so any discussions around dropping support require more thought than just "it isn't supported anymore".
https://data.firefox.com/dashboard/hardware

We appreciate your report. Please trust that we're taking it seriously. But please also trust that we have various considerations that need to be weighed here.

Comment 19

[Alexandru Trif, Desktop QA [:atrif]]

Attachment: win7-updates.txt

Attaching the updates list from both Win7x64 SP1 machines.

Comment 20

[Geoff Brown [:gbrown]]

Thank you so much :atrif, that is good news.

I notice both lists in comment 19 have KB4474419 and KB4490628. Would it be possible to test without those updates applied?

Comment 21

[Ciprian Georgiu, Desktop QA]

(In reply to Geoff Brown [:gbrown] from comment #20)

Thank you so much :atrif, that is good news.

I notice both lists in comment 19 have KB4474419 and KB4490628. Would it be possible to test without those updates applied?

Yes, we retested today on the same Firefox try builds mentioned in comment 17, after uninstalling KB4474419 and KB4490628 updates from Win 7 x64 Pro with SP1. We can confirm that the installations properly worked on this system.

We've also check this on a virtual machine, with a clean install of Win 7 x64 Pro without SP1, and with no updates installed. The Firefox try builds have been properly installed.

Let us know please if you need anything else tested here!

Comment 22

[Ciprian Georgiu, Desktop QA]

Attachment: win7-sp1_No_KB4474419_and_KB4490628.txt

Attaching the system info containing the updates list of the both Win 7 SP1 machines.

Comment 23

[Geoff Brown [:gbrown]]

Interesting! Thanks QA. I note these findings are also consistent with https://bugzilla.mozilla.org/show_bug.cgi?id=1531165#c12 and https://bugzilla.mozilla.org/show_bug.cgi?id=1531165#c13.

Comment 24

[Geoff Brown [:gbrown]]

Romain - In this bug we want to modernize the signature format used for Firefox installers on Windows. Currently, installers are signed with a sha-1 digest; we want to change to a sha-256 digest, for improved security.

Reportedly, the original Windows 7 did not support sha-256, but support was added in SP1. There are also hot fixes for Windows 7 that provide sha-256 support. In all of our testing, our new installers with sha-256 digests work fine, even on Windows 7 without SP1 and with sha-256 hot fixes uninstalled.

Current Microsoft updates for Windows 7 are signed with sha-256 only (comment 16), and Microsoft has required sha-256 support to get updates since 2019 (comment 11).

In short, I think sha-256 digests are safe and appropriate to use, with only a theoretical risk to a very few unpatched Windows 7 users. Your thoughts?

Comment 25

[saelic]

but support was added in SP1

That's wrong, support for SHA-2 based signatures was added with the release of KB4474419.

In short, I think sha-256 digests are safe and appropriate to use, with only a theoretical risk to a very few unpatched Windows 7 users. Your thoughts?

If you are worried about this, just add info about the change in release notes and a link to the patch.
https://www.catalog.update.microsoft.com/Search.aspx?q=KB4474419

---

Aside from this, I strongly suggest that you prepare your release process to support N-number of Authenticode signatures - and maybe even classic PGP signatures, especially for Linux releases. This topic would be closed long time ago if adding a second signature was supported by release process. So, as Windows supports many Authenticode signatures for a single file, your release process should take this fact into account for future use with newer signature algorithms.

---

And for the record:

• SHA-2 family, with SHA-2-256 and SHA-2-512, was introduced in the year 2001,
• SHA-3 family was introduced in October 2012,
• SHA-1 use is deprecated since year 2010,
• update for Windows 7, which adds support for SHA-2 based Authenticode signatures, was released in October 2019.

And now, in March 2022, we have such problems. That really tells something about the state of the industry.

Comment 26

[saelic]

Correction: update for Windows 7, which adds support for SHA-2 based Authenticode signatures, was released in March 2019.
https://support.microsoft.com/help/4474419

Comment 27

[Geoff Brown [:gbrown]]

(In reply to saelic from comment #25)

but support was added in SP1

That's wrong, support for SHA-2 based signatures was added with the release of KB4474419.

Agreed. Thanks for the clarification. KB4474419 was "for Windows 7 SP1", but not actually part of SP1. I would expect KB4474419 to be installed by most current users on SP1, but it's unlikely we can verify that, whereas :RT probably has some idea of how many Firefox users are on SP1.

Comment 28

[saelic]

In short, I think sha-256 digests are safe and appropriate to use, with only a theoretical risk to a very few unpatched Windows 7 users. Your thoughts?

This is how Kaspersky dealt with this [1]:

The SHA-1 module signature algorithm is deprecated by Microsoft. Update KB4474419 is required for successful installation of Kaspersky Endpoint Security on a computer running the Microsoft Windows 7 operating system. For more details about this update, visit the Microsoft technical support website (https://support.microsoft.com/en-us/help/4474419/sha-2-code-signing-support-update).

[1] https://aes.s.kaspersky-labs.com/endpoints/keswin11/11.8.0.384/english-21.5.11.384.0.12.0/3531313638307c44454c7c4e554c4c/ReleaseNotes.txt

Comment 29

[Romain Testard [:RT]]

(In reply to Geoff Brown [:gbrown] from comment #24)

Romain - In this bug we want to modernize the signature format used for Firefox installers on Windows. Currently, installers are signed with a sha-1 digest; we want to change to a sha-256 digest, for improved security.

Reportedly, the original Windows 7 did not support sha-256, but support was added in SP1. There are also hot fixes for Windows 7 that provide sha-256 support. In all of our testing, our new installers with sha-256 digests work fine, even on Windows 7 without SP1 and with sha-256 hot fixes uninstalled.

Current Microsoft updates for Windows 7 are signed with sha-256 only (comment 16), and Microsoft has required sha-256 support to get updates since 2019 (comment 11).

In short, I think sha-256 digests are safe and appropriate to use, with only a theoretical risk to a very few unpatched Windows 7 users. Your thoughts?

82% of monthly installs are on 6.1.7601 (SP1) and 18% are on 6.1.7600 - does the proposed change imply that the 18% of monthly installs on 6.1.7600 will break? https://sql.telemetry.mozilla.org/queries/84638/source#209607

Comment 30

[Geoff Brown [:gbrown]]

(In reply to Romain Testard [:RT] from comment #29)

82% of monthly installs are on 6.1.7601 (SP1) and 18% are on 6.1.7600 - does the proposed change imply that the 18% of monthly installs on 6.1.7600 will break? https://sql.telemetry.mozilla.org/queries/84638/source#209607

I'm sorry, but it's complicated.

https://support.microsoft.com/en-us/topic/2019-sha-2-code-signing-support-requirement-for-windows-and-wsus-64d1c82d-31ee-c273-3930-69a4cde8e64f suggests that KB4474419 is required for sha-2 support on Windows 7, but that doesn't explicitly talk about application installers, only Windows updates. Statements like those noted in comment 28 also suggest that KB4474419 is required. But I can't find anything that explicitly says, "your app installer won't work on Windows 7 unless KB4474419 is installed".

KB4474419 has been available since 2019. Given its importance to Windows updates, I suspect it is very widely used.

QA can't find any problem with sha-2 signed Firefox, even without KB4474419 installed. That's a bit of a puzzle, and also somewhat reassuring.

I'd like to go ahead with check-in and let this ride the trains -- see what the reaction is on Nightly and Beta. We should add something like comment 28 to the Firefox release notes, just in case.

Comment 31

[saelic]

But I can't find anything that explicitly says, "your app installer won't work on Windows 7 unless KB4474419 is installed".

I can say what will happen: If SHA-2 is not supported, then digital signature verification will fail, the UAC prompt will be with orange label telling user that software publisher cannot be verified, but installation can be continued.
This is the same situation as in case with installers that are not digitally signed at all.

Comment 32

[Geoff Brown [:gbrown]]

Ryan, Romain - Do you have any other questions? OK for me to go ahead?

Comment 33

[Ryan VanderMeulen [:RyanVM]]

I'd be fine with landing it on Nightly and seeing what shakes out.

Comment 34

[Romain Testard [:RT]]

Do we have the necessary telemetry to validate that installs and updates will work as expected if we enable this? I'd suspect that Win7 users on unpatched OS won't be the first to raise bugs so we should rely on telemetry to validate that out theory is right I think?

My understanding of what our theory is: Implementing a SHA256 digest won't break more than X% of installs or Y% of updates on target clients
My understanding of what our metrics are: Share of successful Win7 6.1.7600 installs (I'm pretty sure we have the necessary telemetry) and share of Win7 6.1.7600 updates (I'm unclear if we have this information?)

Comment 35

[[e-mail] (:bhearsum)]

(In reply to Romain Testard [:RT] from comment #34)

share of Win7 6.1.7600 updates (I'm unclear if we have this information?)

I'm not sure sure if we have this. If not, can we use OS version + currently installed buildid as a proxy? (ie: if people fail to update they'll either get stuck on an old buildid, or possibly even stop reporting in at all.)

Comment 36

[Geoff Brown [:gbrown]]

(In reply to Romain Testard [:RT] from comment #34)

My understanding of what our theory is: Implementing a SHA256 digest won't break more than X% of installs or Y% of updates on target clients

Right. Further, X and Y should be pretty close to zero: Only those who are running Windows 7 without KB4474419 may be affected; they should get a warning and be offered a way to proceed.

If telemetry does show an uptick in install or update problems on Windows 7, especially if those are skewed to earlier Win7 versions, we should have a closer look.

OK for me to go ahead?

Comment 37

[Romain Testard [:RT]]

I can see install success rate for target clients on https://sql.telemetry.mozilla.org/queries/84813/source#210012 and target clients broken down per app version on https://sql.telemetry.mozilla.org/queries/84815/source#210022
I'm OK to go ahead with the change on Nightly so long as letting it go to Beta is gated on reviewing that the above success metrics look right.

Comment 38

[saelic]

I bet that 100% of potentially affected users will even wouldn't notice, that the UAC prompt changed to orange colour. If they are so backwards with software versions, then they do not care about security at all and will click "Yes" on anything.

Sorry guys, but, with all the respect for you, this is taking too long with too much unnecessary thinking.

Comment 39

[Geoff Brown [:gbrown]]

Thanks Romain.

ni myself to monitor telemetry.

Comment 41

[Aki Sasaki (not active)]

https://hg.mozilla.org/mozilla-central/rev/3fa902696620

Comment 42

[Geoff Brown [:gbrown]]

Telemetry (comment 37) looks good to me.

Comment 43

[Geoff Brown [:gbrown]]

Release Note Request (optional, but appreciated)
[Why is this notable]:
There is concern that some older, unpatched Windows 7 users may require a Windows hot fix to continue using Firefox updates (lots of discussion above); let's make the fix obvious.
[Affects Firefox for Android]:
[Suggested wording]:
Beginning in this release, the Firefox installer for Windows is signed with a SHA-256 digest, rather than SHA-1. Update KB4474419 is required for successful installation on a computer running the Microsoft Windows 7 operating system. For more details about this update, visit the Microsoft technical support website (https://support.microsoft.com/en-us/help/4474419/sha-2-code-signing-support-update).
[Links (documentation, blog post, etc)]:

Comment 44

[Dianna Smith [:diannaS]]

Release note added, please see https://www.mozilla.org/firefox/100.0a1/releasenotes/

Comment 45

[saelic]

(In reply to Geoff Brown [:gbrown] from comment #43)

Update KB4474419 is required for successful installation on a computer running the Microsoft Windows 7 operating system.

That's WRONG. It is not required for successful installation, it is required for successful signature validation.

These are two totally different things, as Windows doesn't block installation of user-mode software which is unsigned/not properly signed/with invalid signature/with signature based on unsupported hash.
Valid signature is required only when installing kernel-mode modules, for example drivers (requirement firstly introduced in Windows Vista) - but again, it has nothing to do with user's software.

As I stated above, users without KB4474419 will not even notice that UAC prompt will be in orange colour - they will click "Yes" on anything or even better, have UAC disabled, so won't even notice anything. You can't expect more from users running Windows 7 in 2022.

Comment 46

[Geoff Brown [:gbrown]]

(In reply to saelic from comment #45)

(In reply to Geoff Brown [:gbrown] from comment #43)

Update KB4474419 is required for successful installation on a computer running the Microsoft Windows 7 operating system.

That's WRONG. It is not required for successful installation, it is required for successful signature validation.

Yes, I know. But many Firefox users won't understand "signature validation". I want to keep the message simple: If you are on Windows 7 and you are concerned because something funny is happening during installation, make sure you have KB4474419.

Comment 47

[saelic]

OK, in this case this is fine.

If I understand correctly, the new signature will be applied from version 10.00.
What about updates to earlier versions, especially the ESR ones?

Comment 48

[Ciprian Georgiu, Desktop QA]

We have also verified this bug on Nightly builds: Windows 2012 Shippable-> rs (B), Windows 2012 x64 Shippable-> rs (B), Windows 2012 Shippable MSis (N) and Windows 2012 x64 Shippable-> MSis (N). Tested with Win 7 x64, without KB4474419 and KB4490628 updates installed. No issues have been encountered with these builds.

Comment 49

[Geoff Brown [:gbrown]]

Comment on attachment 9265566 [details]
Bug 1745467 - Update signing formats for Windows; r=

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration: Provides improved signature integrity on Windows. Including in esr restores consistency to the signature format used across releases.
• User impact if declined:
• Fix Landed on Version: 100
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): We recently closely monitored version 100 signature verification on Windows -- no sign of trouble from this patch.

Comment 50

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9265566 [details]
Bug 1745467 - Update signing formats for Windows; r=

Approved for 91.10esr.

Comment 51

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr91/rev/a9d3fc6cf9e2

Comment 52

[Ciprian Georgiu, Desktop QA]

This is verified as fixed on esr 91.10esr, running Win 7 SP1 and Win 7 without SP1, with KB4474419 and KB4490628 uninstalled. No issues were found while installing the following builds from treeherder:

• Windows 2012 Shippable-> rs (B)
• Windows 2012 x64 Shippable-> rs (B)
• Windows 2012 Shippable MSis (N)
• Windows 2012 x64 Shippable-> MSis (N)

Also, we've tried the official builds from archive.mozilla.org to make sure that everything works as expected:

• Firefox Setup 91.10.0esr.exe - win32
• Firefox Setup 91.10.0esr.msi - win32
• Firefox Setup 91.10.0esr.exe - win64
• Firefox Setup 91.10.0esr.msi - win64

Comment 53

[Geoff Brown [:gbrown]]

Thank you!

Comment 54

[javascriptjedi]

Verified signature fixed. Thanks.