1691782 - Fix missing vcruntime140_1.dll by shipping/running redistributable installer

Status: RESOLVED FIXED

(Firefox :: Installer, defect, P3)

Description

[(No longer employed by Mozilla) Aaron Klotz]

I have seen three reports of this on Reddit in the past week. I'm filing this under installer even though for us it's also kind of a releng thing.

From what I can tell, vcruntime140_1.dll was introduced by VC2019 Update 3.

In at least one Reddit report, this issue started after they installed the VC Runtime Redistributable for the benefit of another program they were installing on their machine.

I have tried reproducing this myself, but have not yet been able to.

A few thoughts as to why this might be affecting us:

• The latest VC2019 redistributables are also shared with VC2017 and VC2015. We ship VC2017 redistributables in our bin directory, so installing the newest VC2019 version may interfere with the ones we ship;
• The dynamic linker might be preferring the newer, installed variant of the CRT DLLs when loading, but is getting confused during dependency resolution because of our local copies.
• We're probably not shipping the latest and greatest version of the CRT, missing out on bugs, security fixes, and incompatibilities such as the one in this bug.
• Shipping the CRT the way we do is not recommended by Microsoft, nor by me personally. As I have mentioned in other bugs, we really should be running the redistributable CRT installer.

Comment 1

[Alice Wyman]

Mozilla Support includes similar user reports. See https://support.mozilla.org/en-US/questions/firefox?tagged=bug1624546&show=all
(see also bug 1624546).

Comment 2

[Adam Gashlin (he/him) [:agashlin] (ex-moco)]

See bug 1624546 comment 18 for STR.

I agree that it would be better to use the redistributable installer, I'm trying to work through how it would work from an installer and updater perspective.

---

We would probably want to run the redist installer as part of an update, maybe only the first time it appears. This would also mean keeping the redist installer in the install directory. The installer Microsoft is shipping for 2015, 2017, 2019 now is around 14MB on x86 and x64, just 2017 is similar. Is there a "minimum" version anymore?

If we run the redist installer, but don't remove our own shipped copy of the redist, will the system redist be consistently used? If it turns out to be harmless to keep our own copy, then some of the complications below go away.

As far as I know, the redist installer can only be run by an administrator, so some other method will be required if we want to keep supporting non-administrator installations. These represent about 4 to 5% of new 87.0 installs (internal link to a installation telemetry event query, which only includes actual Firefox startups).

• We could keep shipping our copy as well, though I'm not 100% sure this won't cause confusion with the system redist. That's 2MB of Microsoft DLLs in our directory (excluding d3dcompiler_47.dll, which I think we need anyway), 0.5MB 7-zipped.

• Bad idea 1: Conditionally unpack them for a single user install, or remove them in an all users install, and maintain this during updates.

• Bad idea 2: We could stop shipping the copy of the redist, run the installer for new installs, and but not for existing installs. The updater could "forget" them rather than remove. This would cause trouble down the line if we ever need to change the redist.

What installer? We don't need to ship 2019, but the unified 2015, 2017, 2019 installer is the only one Microsoft is officially advertising now. We can probably still find a 2017 installer (it looks like https://aka.ms/vs/15/release/VC_Redist.x86.exe is supported?), but I'm not sure that it won't have the same issues. Will it work on Windows 7? Speaking of which, since the redist (or at least the UCRT) gets updated by WU once it's installed system-wide, would relying on the system redist cause us issues down the line? Probably not, everyone in the ecosystem is depending on backwards compatibility.

What about uninstallation? We'd probably just want to leave the system install alone, if it's even possible to remove. There may be a refcount system, but we don't want to break stuff elsewhere on the system that needed the redist without installing it.

---

It seems much simpler to just fix bug 1624546 to include the additional files for now, and that would be needed to protect non-admin installs anyway. I want to keep this bug around for discussion about how to deal with the redistributable installer in the future, though. If you disagree with the severity feel free to adjust.

Comment 3

[(No longer employed by Mozilla) Aaron Klotz]

(In reply to Adam Gashlin (he/him) [:agashlin] from comment #2)

As far as I know, the redist installer can only be run by an administrator, so some other method will be required if we want to keep supporting non-administrator installations.

I suspect this isn't a possibility, but I'll throw it out there just in case it is: Is it possible to ask Windows Update to select and install the redist without elevating?

Comment 4

[Adam Gashlin (he/him) [:agashlin] (ex-moco)]

Another wrinkle: for arm64 we still need to ship a few x86 components (for which we include vcruntime), so we might want to include that installer as well. This doesn't need to be handled completely before we can make progress on the vast majority, though.

(In reply to Aaron Klotz [:aklotz] from comment #3)

(In reply to Adam Gashlin (he/him) [:agashlin] from comment #2)

As far as I know, the redist installer can only be run by an administrator, so some other method will be required if we want to keep supporting non-administrator installations.

I suspect this isn't a possibility, but I'll throw it out there just in case it is: Is it possible to ask Windows Update to select and install the redist without elevating?

It looks like there's a group policy to allow non-admin users to operate Windows Update, but I doubt there's a way to enable that without being an admin already. This answer says it doesn't work since Windows 7, anyway.

Even if we could, I think Windows Update only handles updating the UCRT. I was able to find KBs for the URCT itself, but not for the other components. After installing a redist and updating Windows 10 (from 2004 19041.746 to 20H2 19042.906), vcruntime140.dll was still the version I installed (14.25.28508.3, timestamped 1/8/2020), running the latest installer updates it by over a year (to 14.25.29913.0 timestamped 3/11/2021). Windows Update alone updated ucrtbase.dll from 10.0.19041.546 to 10.0.19041.789.

Comment 5

[Andrea Marchesini [:baku]]

I'm not sure this is the same issue we see with the Mozilla VPN project, but it seems that on Windows, when the VPN client is uninstalled, Firefox stops working because vcruntime140_1.dll is removed. The issue doesn't seem to be related to the Mozilla VPN installer, because nothing custom is done to remove the vcruntime140_1.dll and any other C/C++ apps work as expected. It's just firefox.

The Mozilla VPN installer is generated using this WXS config file: https://github.com/mozilla-mobile/mozilla-vpn-client/blob/main/windows/installer/MozillaVPN.wxs

I increase the priority to P1 because it's a blocker for the VPN client. If you disagree, I'm fine. But it would be great if we can debug this issue together.

Comment 6

[Adam Gashlin (he/him) [:agashlin] (ex-moco)]

(In reply to Andrea Marchesini [:baku] from comment #5)

I'm not sure this is the same issue we see with the Mozilla VPN project, but it seems that on Windows, when the VPN client is uninstalled, Firefox stops working because vcruntime140_1.dll is removed. The issue doesn't seem to be related to the Mozilla VPN installer, because nothing custom is done to remove the vcruntime140_1.dll

The VPN installer uses a vcredist merge module, my guess is this is the issue I discussed in bug 1624546 comment 18. Once you install the VPN look in HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\SharedDLLs in regedit, is the value vcruntime140_1.dll = 1 while concrt140.dll has > 1? If so, then when the VPN is uninstalled, vcruntime140_1.dll gets its refcount decremented, and so it is removed, but concrt140.dll has been updated and still depends on it. This is a bug in how MSMs work and it's why they shouldn't be used for system libraries like this.

Do you see this effect on a clean system? I'd expect it to only happen when the system had also had an earlier vcredist (e.g. 2017) installed. (You can check this by seeing if the system had vcruntime140.dll, but not vcruntime140-1.dll, before installing the VPN).

and any other C/C++ apps work as expected. It's just firefox.

Firefox is set up to prefer DLLs from system32 for security reasons, see bug 1624546 comment 21, so it won't use the DLLs it ships with (which don't depend on vcruntime140_1.dll). This is likely not the case with the other C/C++ programs that still work.

You might try fixing this the way LibreOffice did, by making the redist permanent, this adds an extra +1 to the refcount so the elements are not removed, but I don't know if there's a way to do this within the wxs. Probably a better way is to use the full redist MSI, which is careful to not let the refcounts get out of sync. Ideally this is also what Firefox itself would be doing.

Comment 8

[Gian-Carlo Pascutto [:gcp]]

This bug is marked P1 but has no assignee. Comment 5 sounds extremely scary.

Comment 9

[Robin Steuber (she/her) [:bytesized]]

:agashlin was going to take this, but now no longer works here. I'm not sure when someone else will be able to take it.

I'm setting the priority to P3, for now, to reflect the fact our timeline for the next few cycles currently does not include it. I'll put it on our team's agenda to discuss a new priority for this.

Comment 10

[Haik Aftandilian [:haik]]

I recently ran into this on a new Windows 11 laptop. After uninstalling some of the free trial bloatware, Firefox failed to start due to the missing vcruntimeXXX.dll. Installing Visual Studio community edition with C++ dependencies worked around the problem. Should we reconsider the priority on this?

Comment 11

[Nick Alexander :nalexander [he/him] (PTO August 22-September 12)]

(In reply to Haik Aftandilian [:haik] from comment #10)

I recently ran into this on a new Windows 11 laptop. After uninstalling some of the free trial bloatware, Firefox failed to start due to the missing vcruntimeXXX.dll. Installing Visual Studio community edition with C++ dependencies worked around the problem. Should we reconsider the priority on this?

This is somewhat on me: I've been busy! Just a status update: I picked up Adam's patches and worked through the test failures, so those aren't blocking anything. The sticking point here are some final details on introducing new manifests vs. embedding the manifest into the EXE (which would require some preprocessing of manifests, I think) and thinking more seriously about how to roll this out safely.

Comment 12

[Sören Hentzschel]

Hi Nick,

do you have some news about this one? I can't speak for the international support, but in the German-speaking support alone this issue is reported every few weeks. It is very unsatisfying to have to tell users to copy a file from another directory so that Firefox will be able to display websites again. That's exactly the point: For affected users, Firefox is completely unusable. And I don't want to know how many users are lost who can't find their way to volunteer Firefox support. So this is a very serious issue.

Comment 13

[Nick Alexander :nalexander [he/him] (PTO August 22-September 12)]

(In reply to Sören Hentzschel from comment #12)

Hi Nick,

do you have some news about this one? I can't speak for the international support, but in the German-speaking support alone this issue is reported every few weeks. It is very unsatisfying to have to tell users to copy a file from another directory so that Firefox will be able to display websites again. That's exactly the point: For affected users, Firefox is completely unusable. And I don't want to know how many users are lost who can't find their way to volunteer Firefox support. So this is a very serious issue.

Sorry, I don't really have an update. I'll run this through the Desktop Integration triage system again and see if we can prioritize it. Leaving NI until that is done.

Comment 14

[Nick Rishel [:nrishel]]

To add a quick wrinkle to this, x64 builds will also need to ship the x86 redistributable due to NSIS dependencies.

Comment 15

[Mike Hommey [:glandium]]

This was fixed by a combination of bug 1832467 and bug 1880929.