Closed Bug 2078555 Opened 1 month ago Closed 1 month ago

Various startup crash signatures involving mozilla::nt::RtlAllocPolicy::free_ with mismatched firefox.exe and mozglue.dll versions

Categories

(Core :: DLL Services, defect)

defect

Tracking

()

RESOLVED FIXED
160 Branch
Tracking Status
firefox-esr153 --- fixed
firefox157 --- wontfix
firefox158 + fixed
firefox159 + fixed
firefox160 + fixed

People

(Reporter: yannis, Assigned: handyman)

References

(Blocks 2 open bugs)

Details

(Keywords: regression)

Crash Data

Attachments

(12 external files, 2 obsolete files)

The attached crash signatures look like the various ways that the problem described in bug 2076326 comment 2 manifests throughout the versions that have received the patch from bug 2057112. The root cause is that this patch results in an early fatal crash if there is a mismatch between firefox.exe and mozglue.dll where one version was compiled with the patch and the other without. The mismatch occurs through bugs like bug 2076326, though there are other ways for the updater to run into this kind of trouble. In particular bug 2060275 aimed to address a big chunk of these issues, but it is only present in 156+ release so far -- we were waiting to confirm a positive effect in release and no regression before uplifting to ESR, so ESR versions don't have it in at the moment. So ESR users are more at risk than release users for this bug.

One new concern is that 157.0.1 and corresponding ESR releases will have the backout for the patch from bug 2057112 in them. Updating from a version that had the patch to one that doesn't have it anymore is as problematic as the other way around, so the backout is a concern for users that already got the patch in their current install of Firefox (i.e. users that now run 157.0, 153.4.0esr or 140.16.0esr).

See Also: → 2078664

An interrupted update can leave firefox.exe, mozglue.dll and xul.dll at
different versions, and a ModuleLoadInfo passes between all three. The struct
now starts with a magic number and a layout version. Each binary that
receives one checks them before reading and drops the record otherwise, leaving
the object to the binary that built it.

Assignee: nobody → davidp99
Status: NEW → ASSIGNED

Part 1 lets a binary refuse a ModuleLoadInfo of another layout, but a binary
without that check still accepts one. So firefox.exe, mozglue.dll and xul.dll
now each export their layout version in a function, and a sender asks the
receiver for it before handing module load events across. mozglue.dll also
stops using an executable with a different layout to construct its
ModuleLoadInfos, since that happens by value and cannot be checked afterwards.
A binary without the export is treated as having a different layout.

Blocks: 1926447
Summary: Various crash signatures involving mozilla::nt::RtlAllocPolicy::free_ with mismatched firefox.exe and mozglue.dll versions → Various startup crash signatures involving mozilla::nt::RtlAllocPolicy::free_ with mismatched firefox.exe and mozglue.dll versions
Flags: needinfo?(davidp99)

An interrupted update can leave firefox.exe, mozglue.dll and xul.dll at
different versions, and a ModuleLoadInfo passes between all three. The struct
now starts with a magic number and a layout version. Each binary that
receives one checks them before reading and drops the record otherwise, leaving
the object to the binary that built it.

Original Revision: https://phabricator.services.mozilla.com/D331475

Attachment #9655560 - Flags: approval-mozilla-beta?

firefox-beta Uplift Approval Request

  • User impact if declined/Reason for urgency: Users with failed installs on Windows sometimes end up with mixed versions of our binaries -- firefox.exe, xul.dll and mozglue.dll should all be from the same version but may not be. Recent changes created a binary incompatibility for mixed versions, at their API layers. This patch avoids those crashes for users who get it, regardless of whether or not they got the recent binary incompatibility change. Without the patch, those users may see crashes again, since the binary incompatibility was backed out and that created a new binary incompatibility. These crashes are believed to break updates, so users have to perform a fresh install to get past them.
  • Code covered by automated testing?: yes
  • Fix verified in Nightly?: no
  • Needs manual QE testing?: no
  • Steps to reproduce for manual QE testing:
  • Risk associated with taking this patch: medium
  • Explanation of risk level: The code is not advanced but could break launch if it is buggy -- as the bug it fixes does. The bug it fixes has very bad consequences. If the patches were buggy, it would probably fail launch every time.
  • String changes made/needed?: None
  • Is Android affected?: no
Attachment #9655561 - Flags: approval-mozilla-beta?

Part 1 lets a binary refuse a ModuleLoadInfo of another layout, but a binary
without that check still accepts one. So firefox.exe, mozglue.dll and xul.dll
now each export their layout version in a function, and a sender asks the
receiver for it before handing module load events across. mozglue.dll also
stops using an executable with a different layout to construct its
ModuleLoadInfos, since that happens by value and cannot be checked afterwards.
A binary without the export is treated as having a different layout.

Original Revision: https://phabricator.services.mozilla.com/D331476

ModuleLoadInfo in mozilla-central is binary incompatible with ModuleLoadInfo
in beta, so bump the version number.

Attachment #9655572 - Attachment description: Bug 2078555: Part 3 - Bump ModuleLoadInfo version number r=#win-reviewers! → Bug 2078555: Part 3 - Bump the ModuleLoadInfo version number r=#win-reviewers!
Pushed by daparks@mozilla.com: https://github.com/mozilla-firefox/firefox/commit/bbe3d15ad379 https://hg.mozilla.org/integration/autoland/rev/cc13345e5282 Part 1 - Version ModuleLoadInfo where it crosses binaries r=win-reviewers,yjuglaret https://github.com/mozilla-firefox/firefox/commit/e032338eb9e6 https://hg.mozilla.org/integration/autoland/rev/65bc5f82cccb Part 2 - Don't send module load events if ModuleLoadInfo may have changed r=win-reviewers,yjuglaret
Status: ASSIGNED → RESOLVED
Closed: 1 month ago
Resolution: --- → FIXED
Target Milestone: --- → 159 Branch

Typo in copying from part 2 review.

Attachment #9655572 - Attachment description: Bug 2078555: Part 3 - Bump the ModuleLoadInfo version number r=#win-reviewers! → Bug 2078555: Part 4 - Bump ModuleLoadInfo version number r=#win-reviewers!
Attachment #9655644 - Attachment description: Bug 2078555: Part 3 - Fix conditional in LoaderObserver::OnForward r=me → Bug 2078555: Part 3 - Fix conditional in LoaderObserver::OnForward

The flags set by the bot don't reflect reality because the backouts also cause an issue here.

The patch landed in nightly and ESR is affected.

:handyman, please make an uplift decision for ESR:

  • For ESR, nominate the patch for the appropriate ESR uplift approval if the fix should be included in ESR, or set the ESR status flag(s) to wontfix if an uplift isn't needed. Each affected ESR release needs a decision.

See Requesting an Uplift for documentation on how to request an uplift.

For more information, please visit BugBot documentation.

Flags: needinfo?(davidp99)

An interrupted update can leave firefox.exe, mozglue.dll and xul.dll at
different versions, and a ModuleLoadInfo passes between all three. The struct
now starts with a magic number and a layout version. Each binary that
receives one checks them before reading and drops the record otherwise, leaving
the object to the binary that built it.

Original Revision: https://phabricator.services.mozilla.com/D331475

Attachment #9655815 - Flags: approval-mozilla-release?

Part 1 lets a binary refuse a ModuleLoadInfo of another layout, but a binary
without that check still accepts one. So firefox.exe, mozglue.dll and xul.dll
now each export their layout version in a function, and a sender asks the
receiver for it before handing module load events across. mozglue.dll also
stops using an executable with a different layout to construct its
ModuleLoadInfos, since that happens by value and cannot be checked afterwards.
A binary without the export is treated as having a different layout.

Original Revision: https://phabricator.services.mozilla.com/D331476

Attachment #9655816 - Flags: approval-mozilla-release?
Attachment #9655817 - Flags: approval-mozilla-release?

firefox-release Uplift Approval Request

  • User impact if declined/Reason for urgency: (Adding a patch that was missing from the previous uplift request) Users with failed installs on Windows sometimes end up with mixed versions of our binaries -- firefox.exe, xul.dll and mozglue.dll should all be from the same version but may not be. Recent changes created a binary incompatibility for mixed versions, at their API layers. This patch avoids those crashes for users who get it, regardless of whether or not they got the recent binary incompatibility change. Without the patch, those users may see crashes again, since the binary incompatibility was backed out and that created a new binary incompatibility. These crashes are believed to break updates, so users have to perform a fresh install to get past them.
  • Code covered by automated testing?: yes
  • Fix verified in Nightly?: no
  • Needs manual QE testing?: no
  • Steps to reproduce for manual QE testing:
  • Risk associated with taking this patch: medium
  • Explanation of risk level: The code is not advanced but could break launch if it is buggy -- as the bug it fixes does. The bug it fixes has very bad consequences. If the patches were buggy, it would probably fail launch every time.
  • String changes made/needed?: No
  • Is Android affected?: no

A patch has been attached on this bug, which was already closed. Filing a separate bug will ensure better tracking. If this was not by mistake and further action is needed, please alert the appropriate party. (Or: if the patch doesn't change behavior -- e.g. landing a test case, or fixing a typo -- then feel free to disregard this message)

Attachment #9655560 - Attachment is obsolete: true
Attachment #9655560 - Flags: approval-mozilla-beta?
Attachment #9655561 - Attachment is obsolete: true
Attachment #9655561 - Flags: approval-mozilla-beta?

I've confirmed, to the best extent possible with local builds, that David's patches achieve their purpose and do not break startup:

  • changing the layout like the original issue did, without David's patches, and putting the new mozglue.dll in an old Firefox build results in startup crashes like the ones we see in the crash signatures attached to this bug
  • same test with David's patches lets Firefox start normally (both in a no-version to version 1 transition, and in a version 1 to version 2 transition)

I will run the same test with an official build once the next Nightly is out.

Also, Thunderbird is more affected than Firefox, I assume because Thunderbird had been serving ESR as the default download until March 2025 and so they have a huge ESR population compared to Firefox. The tables in Bugzilla only shows the Firefox numbers, so be careful when reading them (I think the graph shows both). We're currently at around 812 crash reports from Thunderbird in crash-stats, around 286 for Firefox (query).

(In reply to Yannis Juglaret [:yannis] from comment #21)

I will run the same test with an official build once the next Nightly is out.

The idea works with close builds, unfortunately bug 2070220 is another breaking change for firefox.exe vs mozglue.dll cross-version compatibility (removes the MutexImpl export from mozglue.dll, and an old firefox.exe will fail to import it), so these changes are not enough for updates to 158. Similarly ESR153 mozglue.dll no longer exports things that ESR140 firefox.exe would need (e.g. gTwoCharEscapes). Maybe we have to accept that the fixes belong on the updater side (and some are landing these days...).

It seems a bit less than great that some of these patches landed on main for 159 and some after the version bump to 160. Can we please make sure everything gets uplifted where it needs to go? FWIW, this is why the leave-open keyword should be used when only partially landing stacks, especially near the end of a cycle.

Target Milestone: 159 Branch → 160 Branch

(In reply to Yannis Juglaret [:yannis] from comment #24)

The idea works with close builds, unfortunately bug 2070220 is another breaking change for firefox.exe vs mozglue.dll cross-version compatibility (removes the MutexImpl export from mozglue.dll, and an old firefox.exe will fail to import it), so these changes are not enough for updates to 158. Similarly ESR153 mozglue.dll no longer exports things that ESR140 firefox.exe would need (e.g. gTwoCharEscapes). Maybe we have to accept that the fixes belong on the updater side (and some are landing these days...).

To clarify the impact here: the patches cannot hurt if uplifted, but they won't be enough to fix the situation for people that would run into the updater issue during an update to 158 or an update from ESR140 to 153.5. These people will run into a "Couldn't load XPCOM" dialog instead of the startup crash, because of the mismatches in imports & exports described in comment 24. And that's an error we don't get any signal for at the moment, contrary to startup crashes, so we won't be able to evaluate the volume of people running into this. This is independent of the uplift decision: the "Couldn't load XPCOM dialog" is the error that triggers first here.

(In reply to Ryan VanderMeulen [:RyanVM] from comment #25)

It seems a bit less than great that some of these patches landed on main for 159 and some after the version bump to 160. Can we please make sure everything gets uplifted where it needs to go? FWIW, this is why the leave-open keyword should be used when only partially landing stacks, especially near the end of a cycle.

Good catch & yes. The 158 uplift request is correct (we don't want Part 4 there), but we need Part 4 in 159 indeed.

ModuleLoadInfo in mozilla-central is binary incompatible with ModuleLoadInfo
in beta, so bump the version number.

Original Revision: https://phabricator.services.mozilla.com/D331856

Attachment #9656580 - Flags: approval-mozilla-beta?

firefox-beta Uplift Approval Request

  • User impact if declined/Reason for urgency: Part 4 belongs in the 159 branch. This will protect people updating from release 158 to release 159 against the startup crash, if 158 gets its own uplift accepted (158 beta users updating to 159 are not at risk, because 158 beta didn't get Parts 1 to 3).
  • Code covered by automated testing?: yes
  • Fix verified in Nightly?: yes
  • Needs manual QE testing?: no
  • Steps to reproduce for manual QE testing:
  • Risk associated with taking this patch: low
  • Explanation of risk level: Restores the 159 branch to its desired state
  • String changes made/needed?: No
  • Is Android affected?: no
Attachment #9656580 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
Flags: needinfo?(davidp99)
Attachment #9655817 - Flags: approval-mozilla-release? → approval-mozilla-release+
Attachment #9655816 - Flags: approval-mozilla-release? → approval-mozilla-release+
Attachment #9655815 - Flags: approval-mozilla-release? → approval-mozilla-release+

An interrupted update can leave firefox.exe, mozglue.dll and xul.dll at
different versions, and a ModuleLoadInfo passes between all three. The struct
now starts with a magic number and a layout version. Each binary that
receives one checks them before reading and drops the record otherwise, leaving
the object to the binary that built it.

Original Revision: https://phabricator.services.mozilla.com/D331475

Attachment #9664863 - Flags: approval-mozilla-esr153?

Part 1 lets a binary refuse a ModuleLoadInfo of another layout, but a binary
without that check still accepts one. So firefox.exe, mozglue.dll and xul.dll
now each export their layout version in a function, and a sender asks the
receiver for it before handing module load events across. mozglue.dll also
stops using an executable with a different layout to construct its
ModuleLoadInfos, since that happens by value and cannot be checked afterwards.
A binary without the export is treated as having a different layout.

Original Revision: https://phabricator.services.mozilla.com/D331476

Attachment #9664864 - Flags: approval-mozilla-esr153?
Attachment #9664865 - Flags: approval-mozilla-esr153?

ModuleLoadInfo in mozilla-central is binary incompatible with ModuleLoadInfo
in beta, so bump the version number.

Original Revision: https://phabricator.services.mozilla.com/D331856

Attachment #9664866 - Flags: approval-mozilla-esr153?
Attachment #9664863 - Flags: approval-mozilla-esr153? → approval-mozilla-esr153+
Attachment #9664864 - Flags: approval-mozilla-esr153? → approval-mozilla-esr153+
Attachment #9664865 - Flags: approval-mozilla-esr153? → approval-mozilla-esr153+
Attachment #9664866 - Flags: approval-mozilla-esr153? → approval-mozilla-esr153+
QA Whiteboard: [qa-triage-done-c160/b159]
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: