Closed Bug 1624546 Opened 5 years ago Closed 1 year ago

Include vcruntime140_1.dll for VS2019 16.5 x64 from the redist directory in the package

Categories

(Firefox :: Installer, defect, P3)

Product:
Firefox
Component:
Installer
Type:
defect

Tracking

()

Status:
RESOLVED FIXED

People

(Reporter: iannbugzilla, Unassigned)

References

(Depends on 1 open bug)

Details

Attachments

(1 obsolete file)

At the moment the vcruntime140_1.dll for VS2019 16.5 x64 from the redist directory is not included in the package.

The way this works has always been a maintenance nightmare forever. I think what is really needed is to trust that the builder has the correct runtime routines in the redist dir and just to make sure any file matching any of the following patterns gets packaged:

vcruntime[0-9]+.dll
vcruntime[0-9]+[0-9]+.dll
msvcp[0-9]+.dll
msvcp[0-9]+[0-9]+.dll

Unless Microsoft changes their naming convention completely this would future proof the code here.

Sorry, currently don't have the spare cycles to look further into this.

Assignee: iann_bugzilla → nobody
Status: ASSIGNED → NEW

Because this bug's Severity has not been changed from the default since it was filed, and it's Priority is -- (Backlog,) indicating it has has not been previously triaged, the bug's Severity is being updated to -- (default, untriaged.)

Because this bug's Severity has not been changed from the default since it was filed, and it's Priority is -- (Backlog,) indicating it has has not been previously triaged, the bug's Severity is being updated to -- (default, untriaged.)

Because this bug's Severity has not been changed from the default since it was filed, and it's Priority is -- (Backlog,) indicating it has has not been previously triaged, the bug's Severity is being updated to -- (default, untriaged.)

Severity: normal → S3

The severity of these bugs was changed, mistakenly, from normal to S3.

Because these bugs have a priority of --, indicating that they have not been previously triaged, these bugs should be changed to Severity of --.

Severity: S3 → --

I see minimal recent activity so adding this here. Am currently stuck on this same bug + a few things:

  • Have already filed a bug report from the crash tool, not sure where that went.
  • Even a fresh download doesn't resolve it
  • From a brief look, am unclear how does this ticket show traction? Can't see a +1 to let your devs know what is and is not affecting people.
  • I landed here from https://support.mozilla.org/en-US/questions/1290656
  • Tried to copy the DLL file per the URL above, didn't help either.

FWIW: before the DLL file, the error was around:
--------------------------- firefox.exe - System Error ---------------------------
The code execution cannot proceed because VCRUNTIME140_1.dll was not found. Reinstalling the program may fix this problem.
--------------------------- OK ---------------------------

Once I copied the DLL file, didn't work. On uninstalling, now the error is :

firefox.exe - Entry Point Not Found

The procedure entry point __CxxFrameHandler4 could not be located in the dynamic link library C:\WINDOWS\SYSTEM32\MSVCP140.dll.

OK

Assignee: nobody → iann_bugzilla
Status: NEW → ASSIGNED

There's something interesting going on here. Right now, we ship builds with redist VC runtimes from two archives:

$ unzip -l vs2017_15.8.4.zip | grep redist
   329368  01-01-2010 00:00   vs2017_15.8.4/VC/redist/x64/Microsoft.VC141.CRT/concrt140.dll
   625808  01-01-2010 00:00   vs2017_15.8.4/VC/redist/x64/Microsoft.VC141.CRT/msvcp140.dll
    31896  01-01-2010 00:00   vs2017_15.8.4/VC/redist/x64/Microsoft.VC141.CRT/msvcp140_1.dll
   195248  01-01-2010 00:00   vs2017_15.8.4/VC/redist/x64/Microsoft.VC141.CRT/msvcp140_2.dll
   386720  01-01-2010 00:00   vs2017_15.8.4/VC/redist/x64/Microsoft.VC141.CRT/vccorlib140.dll
    87200  01-01-2010 00:00   vs2017_15.8.4/VC/redist/x64/Microsoft.VC141.CRT/vcruntime140.dll
   246576  01-01-2010 00:00   vs2017_15.8.4/VC/redist/x86/Microsoft.VC141.CRT/concrt140.dll
   453416  01-01-2010 00:00   vs2017_15.8.4/VC/redist/x86/Microsoft.VC141.CRT/msvcp140.dll
    28472  01-01-2010 00:00   vs2017_15.8.4/VC/redist/x86/Microsoft.VC141.CRT/msvcp140_1.dll
   154416  01-01-2010 00:00   vs2017_15.8.4/VC/redist/x86/Microsoft.VC141.CRT/msvcp140_2.dll
   269976  01-01-2010 00:00   vs2017_15.8.4/VC/redist/x86/Microsoft.VC141.CRT/vccorlib140.dll
    82752  01-01-2010 00:00   vs2017_15.8.4/VC/redist/x86/Microsoft.VC141.CRT/vcruntime140.dll
$ unzip -l vs2017_15.9.6.zip | grep redist
   359040  01-01-2010 00:00   vs2017_15.9.6/VC/redist/arm64/Microsoft.VC141.CRT/concrt140.dll
   660608  01-01-2010 00:00   vs2017_15.9.6/VC/redist/arm64/Microsoft.VC141.CRT/msvcp140.dll
    32384  01-01-2010 00:00   vs2017_15.9.6/VC/redist/arm64/Microsoft.VC141.CRT/msvcp140_1.dll
   189568  01-01-2010 00:00   vs2017_15.9.6/VC/redist/arm64/Microsoft.VC141.CRT/msvcp140_2.dll
   424576  01-01-2010 00:00   vs2017_15.9.6/VC/redist/arm64/Microsoft.VC141.CRT/vccorlib140.dll
    90752  01-01-2010 00:00   vs2017_15.9.6/VC/redist/arm64/Microsoft.VC141.CRT/vcruntime140.dll
   332336  01-01-2010 00:00   vs2017_15.9.6/VC/redist/x64/Microsoft.VC141.CRT/concrt140.dll
   627440  01-01-2010 00:00   vs2017_15.9.6/VC/redist/x64/Microsoft.VC141.CRT/msvcp140.dll
    30960  01-01-2010 00:00   vs2017_15.9.6/VC/redist/x64/Microsoft.VC141.CRT/msvcp140_1.dll
   205552  01-01-2010 00:00   vs2017_15.9.6/VC/redist/x64/Microsoft.VC141.CRT/msvcp140_2.dll
   366320  01-01-2010 00:00   vs2017_15.9.6/VC/redist/x64/Microsoft.VC141.CRT/vccorlib140.dll
    85232  01-01-2010 00:00   vs2017_15.9.6/VC/redist/x64/Microsoft.VC141.CRT/vcruntime140.dll
   249600  01-01-2010 00:00   vs2017_15.9.6/VC/redist/x86/Microsoft.VC141.CRT/concrt140.dll
   449280  01-01-2010 00:00   vs2017_15.9.6/VC/redist/x86/Microsoft.VC141.CRT/msvcp140.dll
    28440  01-01-2010 00:00   vs2017_15.9.6/VC/redist/x86/Microsoft.VC141.CRT/msvcp140_1.dll
   172824  01-01-2010 00:00   vs2017_15.9.6/VC/redist/x86/Microsoft.VC141.CRT/msvcp140_2.dll
   269568  01-01-2010 00:00   vs2017_15.9.6/VC/redist/x86/Microsoft.VC141.CRT/vccorlib140.dll
    80128  01-01-2010 00:00   vs2017_15.9.6/VC/redist/x86/Microsoft.VC141.CRT/vcruntime140.dll

Observe that there are no vcruntime_* DLLs at all! Now, this could be an error in our redistribution packaging, but in this case it doesn't look like it: the relevant patterns are here and they'll catch anything present in the source VC directory. (I haven't fetched the old VC versions to ensure there's no error here. I have looked at a more recent VC and I do see vcruntime140_1.dll.)

So I'm left wondering how we're ever seeing this in the wild. Would this be folks who install a (necessarily different) version of the VC CRT somehow? Maybe we somehow remove these unexpected files, leaving a partial VC CRT remaining?

As to the content of this ticket, there are two reasons to do this. The first is that at least some folks are trying to redistribute with VS2019, even though we don't do that yet in automation. (And I'm not 100% we support that at all.) The second is that as written, this is fragile.

mhowell: mkaply: this is not my area of expertise. Have you any theories as to how we might be witnessing this issue?

mhowell: mkaply: NI to ponder https://bugzilla.mozilla.org/show_bug.cgi?id=1624546#c9.

Flags: needinfo?(mozilla)
Flags: needinfo?(mhowell)

So I'm left wondering how we're ever seeing this in the wild. Would this be folks who install a (necessarily different) version of the VC CRT somehow? Maybe we somehow remove these unexpected files, leaving a partial VC CRT remaining?

I think that's really the thing we're struggling with. We know it happens, we just don't know what requires that DLL. Maybe there's someone at Msft we can reach out too.

Flags: needinfo?(mozilla)

Yeah, it would seem that what's happening is something is loading another copy of the CRT (presumably the copy in System32) and that one is trying to use the _1 file. But I have no idea either what would be causing the system CRT to be loaded (presumably some third-party module?) or why it would not then load its dependencies out of the same location.

Flags: needinfo?(mhowell)

Another possibility is that something is outright replacing the copy of vcruntime140.dll that we ship, but that seems pretty out there as an explanation.

If you install VS2017 or VS2019 the VS2015 to 2019 redist will be installed too. You won't see the error then. Same for x86 builds where this one does not exist. If you use a redist prior to VS2019 16.5 all good too. I no longer build with VS2017 so not sure if the requirement was added in later version too.

Edit Opps mixed it up. deleted the second part from this comment.

Interesting. That begs the question what API is in vcruntime140_1.dll that we sometimes use and sometimes don't...

I think it is just a new internal dependency and only Microsoft might know what changed unless you want to start digging. Other projects seem to have hit it too:
https://bugs.openjdk.java.net/browse/JDK-8242468

Not sure if this helps anyone else but this resolved my issue... (I was stuck on this issue for a few months)

Merely reinstalling the Microsoft Visual C++ Redistributable for Visual Studio 2015, 2017 and 2019 for my laptop architecture resolved the issue for me.

https://support.microsoft.com/en-au/help/2977003/the-latest-supported-visual-c-downloads

For FF Devs, in case I could help here by listing out folder contents etc... ( for e.g. if you want to see what are the current status of the DLL files etc. post this re-install) do let me know. I say that since I am still on the same laptop on which FF wasn't working for months.

See Also: → 1691782

I think I can reproduce the issue when an app uses merge modules (MSM) to install the redist, and it is then uninstalled. I roughly followed the steps to reproduce with LibreOffice in this thread, on Windows 10 20H2:

  1. Install Firefox 86.0
  2. Install vs2017 redist
  3. Install LibreOffice 7.0.0.3 msi
  4. Uninstall LibreOffice
  5. Firefox now hits a vcruntime140_1.dll error on start

It sounds like this is related to reference counting the DLLs, see this LibreOffice bug comment: There's an old runtime (e.g. 2017, pre-vcruntime140_1.dll), then a new runtime is installed via MSM which overwrites the DLLs and adds vcruntime140_1.dll, all the old DLLs go to refcount 2 but vcruntime140_1.dll is refcount 1. If this new runtime is removed, msiexec decrements the refcounts, leaving the DLLs that were present before it was installed (which have been updated), but not including vcruntime140_1.dll, even though the updated DLLs (e.g. concrt140.dll) now depend on it. The standalone 2015-2019 redist installer doesn't seem to make this same mistake, it raises everything to the same refcount.

If I subsequently install LibreOffice 6.4, Firefox starts, but can't start content processes. I'm not quite sure what's happening there.

Could this also happen when installing an application, which seems like it would be more common? Some installers may attempt to remove an existing version of the redist, and then fail to replace it, though I haven't been able to reproduce this with an installer yet. See this thread, the Microsoft folks seem to think that's what's going on there, with RemoveExistingProducts in the MSI. There are cases when an application tries to install the runtime it wants but then can't launch due to missing vcruntime140_1.dll, I've seen this mentioned with Sims 4, Star Wars Squadrons, and Snap Camera in a few places, though I wasn't able to reproduce it with Snap Camera 1.10.0, 1.11.0, or 1.12.0.

I made a local build with the attached patch, this seems to fix the issue. Weirdly it also fixes other apps on the system that can now find vcruntime140_1.dll in our install dir! Though this isn't an ideal solution, it would be an easier change than including the standalone runtime .exe installer.

Just experienced this bug on a laptop I'm building out. Only thing I can think of as a trigger is that I had just uninstalled all M365 apps and gone through a registry/file cleaning (CCleaner).

I fixed it by copying the file needed (vcruntime140_1.dll) from the Edge browser folder here - c:\Program Files (x86)\Microsoft\Edge\Application\89.0.774.45

I kinda like the idea of using Edge to fix Firefox but hope this bug gets fixed soon. :)

This bug also occurs after installing or uninstalling some autodesk programs.
I found the solution that can help us all:
https://knowledge.autodesk.com/support/autocad/troubleshooting/caas/sfdcarticles/sfdcarticles/How-to-remove-and-reinstall-Microsoft-Visual-C-Runtime-Libraries.html

The bug is caused by a conflict in the Visual C ++ libraries, uninstalling all versions and reinstalling only the latest version of each one of them, everything works correctly again.
Pay attention that for the libraries from 2015 to 2019 only the Visual C ++ version 2015-2019 is needed
Library download link:
https://docs.microsoft.com/en-us/archive/blogs/jagbal/where-can-i-download-the-old-visual-c-redistributables

I'm able to work around the repro from comment 18 by disabling "prefer system32" in the sandbox and launcher. I realized it when I saw that the launcher loads our copy of the redist, but the browser process loads from system32 (and fails). The parent process works after the first run because the launcher can see from the registry that the browser didn't come up successfully last time, so it just runs as the browser, by then it has already loaded the local runtime before it set "prefer system32" on itself. But then the sandboxing mitigations still prevent child processes from spawning successfully, so we get blank tabs.

I doubt that we're going to change this, but it's good to explain why a bad system redist install is affecting us at all (besides injection). Demonstration builds are on try.

(In reply to Adam Gashlin (he/him) [:agashlin] from comment #21)

I'm able to work around the repro from comment 18 by disabling "prefer system32" in the sandbox and launcher. I realized it when I saw that the launcher loads our copy of the redist, but the browser process loads from system32 (and fails). The parent process works after the first run because the launcher can see from the registry that the browser didn't come up successfully last time, so it just runs as the browser, by then it has already loaded the local runtime before it set "prefer system32" on itself. But then the sandboxing mitigations still prevent child processes from spawning successfully, so we get blank tabs.

I doubt that we're going to change this, but it's good to explain why a bad system redist install is affecting us at all (besides injection). Demonstration builds are on try.

Can we explicitly load our version?

Flags: needinfo?(agashlin)

I suspect it would be very difficult to make all uses late enough that we could explicitly load them first, but I don't know much about this. :toshi, does this seem feasible?

Flags: needinfo?(agashlin) → needinfo?(tkikuchi)

(In reply to Adam Gashlin (he/him) [:agashlin] from comment #23)

I suspect it would be very difficult to make all uses late enough that we could explicitly load them first, but I don't know much about this. :toshi, does this seem feasible?

I cannot come up with a way to explicitly load a module in the executable's directory, keeping the PreferSystem32Images policy. We can add a logic for the launcher process to check MSVCP140.dll's import table and the existence of vcruntime140_1.dll, and to decide not to turn on PreferSystem32Images, but it's really hacky.

One of the reasons why Firefox depends on MSVCP140.dll is std::ostringstream e.g. launcher process, chromium sandbox, and probably more. We could implement our own stream class to get rid of dependency, but it's not easy either. Or, if we can static-link msvcp140 into firefox.exe (by adding -MT option?), maybe we can pre-load our version of msvcp140 before loading system32's msvcp140.

Flags: needinfo?(tkikuchi)

A relative had this issue lately. Steps involved:

  • Firefox worked fine
  • Uninstalled the trial version of Office 365
  • Firefox opens a window but it's completely broken and empty. No interaction possible. Has to be killed from task manager.
  • Reinstalled Firefox
  • Now shows an error about vcruntime140_1.dll not found
  • Looked for this file on the computer. Found it and put it in the Firefox folder
  • Now runs fine.

Fortunately, he is really convinced about using Firefox and has good technical skills. But I fear that this is typically the kind of bug where users can easily give up :(

I was just hit by this by removing an old version of LibreOffice. Afterwards Firefox would load without any errors, but no content would be displayed. The fix was to reinstall vc_redist:

https://support.microsoft.com/en-us/topic/the-latest-supported-visual-c-downloads-2647da03-1eea-4433-9aff-95f26a218cc0

A relative had this problem also on windows 10 Home edition.
After installing her graphics tablet (drivers + software recommended by wacom). It is a one by wacom medium. After the install, the computer rebooted.
After the reboot, Firefox opens but displays nothing in the tabs, not even the about: ... works.
Reparation do not work. So I uninstalled and reinstalled. Same error as Mathieu Leplatre [:leplatrem].
I did the same thing and it work again.

It seems that installation and uninstallation can prevent Firefox to use vcruntime140_1.dll.
It should be added to firefox package.

(In reply to Toshihito Kikuchi [:toshi] from comment #24)

(In reply to Adam Gashlin (he/him) [:agashlin] from comment #23)

I suspect it would be very difficult to make all uses late enough that we could explicitly load them first, but I don't know much about this. :toshi, does this seem feasible?

I cannot come up with a way to explicitly load a module in the executable's directory, keeping the PreferSystem32Images policy. We can add a logic for the launcher process to check MSVCP140.dll's import table and the existence of vcruntime140_1.dll, and to decide not to turn on PreferSystem32Images, but it's really hacky.

We can create a private SxS assembly to load selected DLLs from our app directory even if the PreferSystem32Images mitigation is enabled. I fixed a similar issue about mozglue.dll (see bug 1496179).

See Also: → 1724313
Depends on: 1733734
See Also: → 1733734

(In reply to Masatoshi Kimura [:emk] from comment #28)

We can create a private SxS assembly to load selected DLLs from our app directory even if the PreferSystem32Images mitigation is enabled. I fixed a similar issue about mozglue.dll (see bug 1496179).

I filed bug 1733734 to investigate this.

The following patch is waiting for review from an inactive reviewer:

ID Title Author Reviewer Status
D68010 Bug 1624546 - Include vcruntime140_1.dll for VS2019 16.5 x64 from the redist directory in the package. r=dmajor iannbugzilla nalexander: Resigned from review

:iannbugzilla, could you please find another reviewer or abandon the patch if it is no longer relevant?

For more information, please visit auto_nag documentation.

Flags: needinfo?(iannbugzilla)
Attachment #9135362 - Attachment is obsolete: true

Patch abandoned.

Flags: needinfo?(iannbugzilla)
Assignee: iannbugzilla → nobody
Status: ASSIGNED → NEW

Firefox team -- FYI that users on the Mozilla VPN are reporting still seeing Firefox issues when uninstalling the VPN (or Firefox crashing when updating the VPN).

(In reply to Santiago from comment #32)

Firefox team -- FYI that users on the Mozilla VPN are reporting still seeing Firefox issues when uninstalling the VPN

This was reported in bug 1691782 comment 5.

See Also: → 1783888

This doesn't appear to be an issue with the Build System, moving over to Installer.

Component: General → Installer
Product: Firefox Build System → Firefox
Severity: -- → S3
Priority: -- → P3
Status: NEW → RESOLVED
Closed: 1 year ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: