Various startup crash signatures involving mozilla::nt::RtlAllocPolicy::free_ with mismatched firefox.exe and mozglue.dll versions
Categories
(Core :: DLL Services, defect)
Tracking
()
People
(Reporter: yannis, Assigned: handyman)
References
(Blocks 2 open bugs)
Details
(Keywords: regression)
Crash Data
Attachments
(12 external files, 2 obsolete files)
The attached crash signatures look like the various ways that the problem described in bug 2076326 comment 2 manifests throughout the versions that have received the patch from bug 2057112. The root cause is that this patch results in an early fatal crash if there is a mismatch between firefox.exe and mozglue.dll where one version was compiled with the patch and the other without. The mismatch occurs through bugs like bug 2076326, though there are other ways for the updater to run into this kind of trouble. In particular bug 2060275 aimed to address a big chunk of these issues, but it is only present in 156+ release so far -- we were waiting to confirm a positive effect in release and no regression before uplifting to ESR, so ESR versions don't have it in at the moment. So ESR users are more at risk than release users for this bug.
One new concern is that 157.0.1 and corresponding ESR releases will have the backout for the patch from bug 2057112 in them. Updating from a version that had the patch to one that doesn't have it anymore is as problematic as the other way around, so the backout is a concern for users that already got the patch in their current install of Firefox (i.e. users that now run 157.0, 153.4.0esr or 140.16.0esr).
Updated•1 month ago
|
Updated•1 month ago
|
| Assignee | ||
Comment 2•1 month ago
|
||
An interrupted update can leave firefox.exe, mozglue.dll and xul.dll at
different versions, and a ModuleLoadInfo passes between all three. The struct
now starts with a magic number and a layout version. Each binary that
receives one checks them before reading and drops the record otherwise, leaving
the object to the binary that built it.
Updated•1 month ago
|
| Assignee | ||
Comment 3•1 month ago
|
||
Part 1 lets a binary refuse a ModuleLoadInfo of another layout, but a binary
without that check still accepts one. So firefox.exe, mozglue.dll and xul.dll
now each export their layout version in a function, and a sender asks the
receiver for it before handing module load events across. mozglue.dll also
stops using an executable with a different layout to construct its
ModuleLoadInfos, since that happens by value and cannot be checked afterwards.
A binary without the export is treated as having a different layout.
| Reporter | ||
Updated•1 month ago
|
| Assignee | ||
Updated•1 month ago
|
| Assignee | ||
Comment 4•1 month ago
|
||
An interrupted update can leave firefox.exe, mozglue.dll and xul.dll at
different versions, and a ModuleLoadInfo passes between all three. The struct
now starts with a magic number and a layout version. Each binary that
receives one checks them before reading and drops the record otherwise, leaving
the object to the binary that built it.
Original Revision: https://phabricator.services.mozilla.com/D331475
Updated•1 month ago
|
Comment 5•1 month ago
|
||
firefox-beta Uplift Approval Request
- User impact if declined/Reason for urgency: Users with failed installs on Windows sometimes end up with mixed versions of our binaries -- firefox.exe, xul.dll and mozglue.dll should all be from the same version but may not be. Recent changes created a binary incompatibility for mixed versions, at their API layers. This patch avoids those crashes for users who get it, regardless of whether or not they got the recent binary incompatibility change. Without the patch, those users may see crashes again, since the binary incompatibility was backed out and that created a new binary incompatibility. These crashes are believed to break updates, so users have to perform a fresh install to get past them.
- Code covered by automated testing?: yes
- Fix verified in Nightly?: no
- Needs manual QE testing?: no
- Steps to reproduce for manual QE testing:
- Risk associated with taking this patch: medium
- Explanation of risk level: The code is not advanced but could break launch if it is buggy -- as the bug it fixes does. The bug it fixes has very bad consequences. If the patches were buggy, it would probably fail launch every time.
- String changes made/needed?: None
- Is Android affected?: no
| Assignee | ||
Comment 6•1 month ago
|
||
Part 1 lets a binary refuse a ModuleLoadInfo of another layout, but a binary
without that check still accepts one. So firefox.exe, mozglue.dll and xul.dll
now each export their layout version in a function, and a sender asks the
receiver for it before handing module load events across. mozglue.dll also
stops using an executable with a different layout to construct its
ModuleLoadInfos, since that happens by value and cannot be checked afterwards.
A binary without the export is treated as having a different layout.
Original Revision: https://phabricator.services.mozilla.com/D331476
| Assignee | ||
Comment 7•1 month ago
|
||
ModuleLoadInfo in mozilla-central is binary incompatible with ModuleLoadInfo
in beta, so bump the version number.
Updated•1 month ago
|
Comment 9•1 month ago
|
||
| bugherder | ||
https://hg.mozilla.org/mozilla-central/rev/cc13345e5282
https://hg.mozilla.org/mozilla-central/rev/65bc5f82cccb
| Assignee | ||
Comment 10•1 month ago
|
||
Typo in copying from part 2 review.
Updated•1 month ago
|
Updated•1 month ago
|
Comment 11•1 month ago
|
||
| Reporter | ||
Comment 12•1 month ago
|
||
The flags set by the bot don't reflect reality because the backouts also cause an issue here.
Comment 13•1 month ago
|
||
The patch landed in nightly and ESR is affected.
:handyman, please make an uplift decision for ESR:
- For ESR, nominate the patch for the appropriate ESR uplift approval if the fix should be included in ESR, or set the ESR status flag(s) to
wontfixif an uplift isn't needed. Each affected ESR release needs a decision.
See Requesting an Uplift for documentation on how to request an uplift.
For more information, please visit BugBot documentation.
| Reporter | ||
Updated•1 month ago
|
Comment 14•1 month ago
|
||
| bugherder | ||
Comment 15•1 month ago
|
||
| Reporter | ||
Comment 16•1 month ago
|
||
An interrupted update can leave firefox.exe, mozglue.dll and xul.dll at
different versions, and a ModuleLoadInfo passes between all three. The struct
now starts with a magic number and a layout version. Each binary that
receives one checks them before reading and drops the record otherwise, leaving
the object to the binary that built it.
Original Revision: https://phabricator.services.mozilla.com/D331475
Updated•1 month ago
|
| Reporter | ||
Comment 17•1 month ago
|
||
Part 1 lets a binary refuse a ModuleLoadInfo of another layout, but a binary
without that check still accepts one. So firefox.exe, mozglue.dll and xul.dll
now each export their layout version in a function, and a sender asks the
receiver for it before handing module load events across. mozglue.dll also
stops using an executable with a different layout to construct its
ModuleLoadInfos, since that happens by value and cannot be checked afterwards.
A binary without the export is treated as having a different layout.
Original Revision: https://phabricator.services.mozilla.com/D331476
Updated•1 month ago
|
| Reporter | ||
Comment 18•1 month ago
|
||
Typo in copying from part 2 review.
Original Revision: https://phabricator.services.mozilla.com/D331904
Updated•1 month ago
|
Comment 19•1 month ago
|
||
firefox-release Uplift Approval Request
- User impact if declined/Reason for urgency: (Adding a patch that was missing from the previous uplift request) Users with failed installs on Windows sometimes end up with mixed versions of our binaries -- firefox.exe, xul.dll and mozglue.dll should all be from the same version but may not be. Recent changes created a binary incompatibility for mixed versions, at their API layers. This patch avoids those crashes for users who get it, regardless of whether or not they got the recent binary incompatibility change. Without the patch, those users may see crashes again, since the binary incompatibility was backed out and that created a new binary incompatibility. These crashes are believed to break updates, so users have to perform a fresh install to get past them.
- Code covered by automated testing?: yes
- Fix verified in Nightly?: no
- Needs manual QE testing?: no
- Steps to reproduce for manual QE testing:
- Risk associated with taking this patch: medium
- Explanation of risk level: The code is not advanced but could break launch if it is buggy -- as the bug it fixes does. The bug it fixes has very bad consequences. If the patches were buggy, it would probably fail launch every time.
- String changes made/needed?: No
- Is Android affected?: no
Comment 20•1 month ago
|
||
A patch has been attached on this bug, which was already closed. Filing a separate bug will ensure better tracking. If this was not by mistake and further action is needed, please alert the appropriate party. (Or: if the patch doesn't change behavior -- e.g. landing a test case, or fixing a typo -- then feel free to disregard this message)
Updated•1 month ago
|
Updated•1 month ago
|
| Reporter | ||
Comment 21•1 month ago
|
||
I've confirmed, to the best extent possible with local builds, that David's patches achieve their purpose and do not break startup:
- changing the layout like the original issue did, without David's patches, and putting the new mozglue.dll in an old Firefox build results in startup crashes like the ones we see in the crash signatures attached to this bug
- same test with David's patches lets Firefox start normally (both in a no-version to version 1 transition, and in a version 1 to version 2 transition)
I will run the same test with an official build once the next Nightly is out.
| Reporter | ||
Comment 22•1 month ago
•
|
||
Also, Thunderbird is more affected than Firefox, I assume because Thunderbird had been serving ESR as the default download until March 2025 and so they have a huge ESR population compared to Firefox. The tables in Bugzilla only shows the Firefox numbers, so be careful when reading them (I think the graph shows both). We're currently at around 812 crash reports from Thunderbird in crash-stats, around 286 for Firefox (query).
Comment 23•1 month ago
|
||
| bugherder | ||
| Reporter | ||
Comment 24•1 month ago
•
|
||
(In reply to Yannis Juglaret [:yannis] from comment #21)
I will run the same test with an official build once the next Nightly is out.
The idea works with close builds, unfortunately bug 2070220 is another breaking change for firefox.exe vs mozglue.dll cross-version compatibility (removes the MutexImpl export from mozglue.dll, and an old firefox.exe will fail to import it), so these changes are not enough for updates to 158. Similarly ESR153 mozglue.dll no longer exports things that ESR140 firefox.exe would need (e.g. gTwoCharEscapes). Maybe we have to accept that the fixes belong on the updater side (and some are landing these days...).
Comment 25•1 month ago
|
||
It seems a bit less than great that some of these patches landed on main for 159 and some after the version bump to 160. Can we please make sure everything gets uplifted where it needs to go? FWIW, this is why the leave-open keyword should be used when only partially landing stacks, especially near the end of a cycle.
| Reporter | ||
Comment 26•1 month ago
•
|
||
(In reply to Yannis Juglaret [:yannis] from comment #24)
The idea works with close builds, unfortunately bug 2070220 is another breaking change for firefox.exe vs mozglue.dll cross-version compatibility (removes the
MutexImplexport frommozglue.dll, and an oldfirefox.exewill fail to import it), so these changes are not enough for updates to 158. Similarly ESR153 mozglue.dll no longer exports things that ESR140firefox.exewould need (e.g.gTwoCharEscapes). Maybe we have to accept that the fixes belong on the updater side (and some are landing these days...).
To clarify the impact here: the patches cannot hurt if uplifted, but they won't be enough to fix the situation for people that would run into the updater issue during an update to 158 or an update from ESR140 to 153.5. These people will run into a "Couldn't load XPCOM" dialog instead of the startup crash, because of the mismatches in imports & exports described in comment 24. And that's an error we don't get any signal for at the moment, contrary to startup crashes, so we won't be able to evaluate the volume of people running into this. This is independent of the uplift decision: the "Couldn't load XPCOM dialog" is the error that triggers first here.
(In reply to Ryan VanderMeulen [:RyanVM] from comment #25)
It seems a bit less than great that some of these patches landed on main for 159 and some after the version bump to 160. Can we please make sure everything gets uplifted where it needs to go? FWIW, this is why the
leave-openkeyword should be used when only partially landing stacks, especially near the end of a cycle.
Good catch & yes. The 158 uplift request is correct (we don't want Part 4 there), but we need Part 4 in 159 indeed.
| Reporter | ||
Comment 27•1 month ago
|
||
ModuleLoadInfo in mozilla-central is binary incompatible with ModuleLoadInfo
in beta, so bump the version number.
Original Revision: https://phabricator.services.mozilla.com/D331856
Updated•1 month ago
|
Comment 28•1 month ago
|
||
firefox-beta Uplift Approval Request
- User impact if declined/Reason for urgency: Part 4 belongs in the 159 branch. This will protect people updating from release 158 to release 159 against the startup crash, if 158 gets its own uplift accepted (158 beta users updating to 159 are not at risk, because 158 beta didn't get Parts 1 to 3).
- Code covered by automated testing?: yes
- Fix verified in Nightly?: yes
- Needs manual QE testing?: no
- Steps to reproduce for manual QE testing:
- Risk associated with taking this patch: low
- Explanation of risk level: Restores the 159 branch to its desired state
- String changes made/needed?: No
- Is Android affected?: no
Updated•1 month ago
|
Updated•1 month ago
|
Comment 29•1 month ago
|
||
| uplift | ||
| Assignee | ||
Updated•1 month ago
|
Updated•1 month ago
|
Updated•1 month ago
|
Updated•1 month ago
|
Updated•1 month ago
|
Updated•1 month ago
|
Comment 30•1 month ago
|
||
| uplift | ||
| Assignee | ||
Comment 31•1 month ago
|
||
An interrupted update can leave firefox.exe, mozglue.dll and xul.dll at
different versions, and a ModuleLoadInfo passes between all three. The struct
now starts with a magic number and a layout version. Each binary that
receives one checks them before reading and drops the record otherwise, leaving
the object to the binary that built it.
Original Revision: https://phabricator.services.mozilla.com/D331475
Updated•1 month ago
|
| Assignee | ||
Comment 32•1 month ago
|
||
Part 1 lets a binary refuse a ModuleLoadInfo of another layout, but a binary
without that check still accepts one. So firefox.exe, mozglue.dll and xul.dll
now each export their layout version in a function, and a sender asks the
receiver for it before handing module load events across. mozglue.dll also
stops using an executable with a different layout to construct its
ModuleLoadInfos, since that happens by value and cannot be checked afterwards.
A binary without the export is treated as having a different layout.
Original Revision: https://phabricator.services.mozilla.com/D331476
Updated•1 month ago
|
| Assignee | ||
Comment 33•1 month ago
|
||
Typo in copying from part 2 review.
Original Revision: https://phabricator.services.mozilla.com/D331904
Updated•1 month ago
|
| Assignee | ||
Comment 34•1 month ago
|
||
ModuleLoadInfo in mozilla-central is binary incompatible with ModuleLoadInfo
in beta, so bump the version number.
Original Revision: https://phabricator.services.mozilla.com/D331856
Updated•1 month ago
|
Updated•1 month ago
|
Updated•1 month ago
|
Updated•1 month ago
|
Updated•1 month ago
|
Updated•1 month ago
|
Comment 35•1 month ago
|
||
| uplift | ||
Updated•1 month ago
|
Description
•