Bugzilla

As initially shared by [:rkraesig], we seem to receive crash reports such as [this one](https://crash-stats.mozilla.org/report/index/92d1bb82-33fe-4e9c-8e2f-c96ac0230804), where the module version information is not coherent with the Timestamp and ImageSize.

In this example, the version information stored in the crash dump matches with Firefox 115.0.3esr, but for some modules such as `xul.dll` the Timestamp and ImageSize are from 115.1.0esr. The mismatch is likely related to the way the updater updates our `exe` and `dll` files: `xul.dll` was most likely already in 115.1.0esr version on disk at the moment we crashed, however the version loaded in the process was 115.0.3esr, as we can confirm by looking for the correct correct stack for the main thread in a debugger (see attachment).

On Windows, debuggers rely on Timestamp and ImageSize to collect a DLL from a debug server before using it to walk the stacks of threads, using what [Breakpad calls the `code_id`](https://searchfox.org/mozilla-central/source/toolkit/crashreporter/google-breakpad/src/processor/minidump.cc#2040-2049), which is derived from those. Because of the mismatch, Socorro (and debuggers) currently incorrectly uses the 115.1.0esr version of the mismatched DLLs to parse the crash dump from above. As a consequence, these kind of crashes can end up in the wrong signature, and with the wrong proto signature (see attachment).

There are multiple questions here that would require their own investigation, and possibly their own bug:

1. How common is this situation?
2. Is it possible to rewrite the updater and/or the crash reporter to avoid this kind of mismatch? Can we to preserve the information that the file on disk changed its version because it could be relevant for the investigation?
3. Can we add heuristics in the Socorro backend to detect this situation and try with the two versions of the DLL and try to guess which source of information seems more correct?
4. How can we adapt the Socorro webapp to warn about such mismatch?

Note that my own testing suggests that there could be situations where there is a mismatch and it is the `code_id` that is correct and not the version information. I obtained [this](https://crash-stats.mozilla.org/report/index/bce6770e-05a6-4613-914f-96f6e0231025#tab-modules) example crash report by forcing a crash after manually renaming `xul.dll` on disk while the 115.0.3esr process is running and putting version 115.1.0esr of the DLL in its place. In this report, the situation is somehow reversed: the version information says 115.1.0esr but the TimeStamp and Signature shown by my debugger are correctly from 115.0.3esr. I'm not sure that my test matches with how the updater works though, so maybe this situation does not happen with legit crashes.

As initially shared by [:rkraesig], we seem to receive crash reports such as [this one](https://crash-stats.mozilla.org/report/index/92d1bb82-33fe-4e9c-8e2f-c96ac0230804), where the module version information is not coherent with the Timestamp and ImageSize.

In this example, the version information stored in the crash dump matches with Firefox 115.0.3esr, but for some modules such as `xul.dll` the Timestamp and ImageSize are from 115.1.0esr. The mismatch is likely related to the way the updater updates our `exe` and `dll` files: `xul.dll` was most likely already in 115.1.0esr version on disk at the moment we crashed, however the version loaded in the process was 115.0.3esr, as we can confirm by looking for the correct correct stack for the main thread in a debugger (see attachment).

On Windows, debuggers rely on Timestamp and ImageSize to collect a DLL from a debug server before using it to walk the stacks of threads, using [what Breakpad calls the `code_id`, which is derived from those](https://searchfox.org/mozilla-central/source/toolkit/crashreporter/google-breakpad/src/processor/minidump.cc#2040-2049). Because of the mismatch, Socorro (and debuggers) currently incorrectly uses the 115.1.0esr version of the mismatched DLLs to parse the crash dump from above. As a consequence, these kind of crashes can end up in the wrong signature, and with the wrong proto signature (see attachment).

There are multiple questions here that would require their own investigation, and possibly their own bug:

1. How common is this situation?
2. Is it possible to rewrite the updater and/or the crash reporter to avoid this kind of mismatch? Can we to preserve the information that the file on disk changed its version because it could be relevant for the investigation?
3. Can we add heuristics in the Socorro backend to detect this situation and try with the two versions of the DLL and to guess which source of information seems more correct?
4. How can we adapt the Socorro webapp to warn about such mismatch?

Note that my own testing suggests that there could be situations where there is a mismatch and it is the `code_id` that is correct and not the version information. I obtained [this](https://crash-stats.mozilla.org/report/index/bce6770e-05a6-4613-914f-96f6e0231025#tab-modules) example crash report by forcing a crash after manually renaming `xul.dll` on disk while the 115.0.3esr process is running and putting version 115.1.0esr of the DLL in its place. In this report, the situation is somehow reversed: the version information says 115.1.0esr but the TimeStamp and Signature shown by my debugger are correctly from 115.0.3esr. I'm not sure that my test matches how the updater works though, so maybe this situation does not happen with legit crashes.

As initially shared by [:rkraesig], we seem to receive crash reports such as [this one](https://crash-stats.mozilla.org/report/index/92d1bb82-33fe-4e9c-8e2f-c96ac0230804), where the module version information is not coherent with the Timestamp and ImageSize.

In this example, the version information stored in the crash dump matches with Firefox 115.0.3esr, but for some modules such as `xul.dll` the Timestamp and ImageSize are from 115.1.0esr. The mismatch is likely related to the way the updater updates our `exe` and `dll` files: `xul.dll` was most likely already in 115.1.0esr version on disk at the moment we crashed, however the version loaded in the process was 115.0.3esr, as we can confirm by looking for the correct stack for the main thread in a debugger (see attachment).

On Windows, debuggers rely on Timestamp and ImageSize to collect a DLL from a debug server before using it to walk the stacks of threads, using [what Breakpad calls the `code_id`, which is derived from those](https://searchfox.org/mozilla-central/source/toolkit/crashreporter/google-breakpad/src/processor/minidump.cc#2040-2049). Because of the mismatch, Socorro (and debuggers) currently incorrectly uses the 115.1.0esr version of the mismatched DLLs to parse the crash dump from above. As a consequence, these kind of crashes can end up in the wrong signature, and with the wrong proto signature (see attachment).

There are multiple questions here that would require their own investigation, and possibly their own bug:

1. How common is this situation?
2. Is it possible to rewrite the updater and/or the crash reporter to avoid this kind of mismatch? Can we to preserve the information that the file on disk changed its version because it could be relevant for the investigation?
3. Can we add heuristics in the Socorro backend to detect this situation and try with the two versions of the DLL and to guess which source of information seems more correct?
4. How can we adapt the Socorro webapp to warn about such mismatch?

Note that my own testing suggests that there could be situations where there is a mismatch and it is the `code_id` that is correct and not the version information. I obtained [this](https://crash-stats.mozilla.org/report/index/bce6770e-05a6-4613-914f-96f6e0231025#tab-modules) example crash report by forcing a crash after manually renaming `xul.dll` on disk while the 115.0.3esr process is running and putting version 115.1.0esr of the DLL in its place. In this report, the situation is somehow reversed: the version information says 115.1.0esr but the TimeStamp and Signature shown by my debugger are correctly from 115.0.3esr. I'm not sure that my test matches how the updater works though, so maybe this situation does not happen with legit crashes.

As initially shared by [:rkraesig], we seem to receive crash reports such as [this one](https://crash-stats.mozilla.org/report/index/92d1bb82-33fe-4e9c-8e2f-c96ac0230804), where the module version information is not coherent with the Timestamp and ImageSize.

In this example, the version information stored in the crash dump matches with Firefox 115.0.3esr, but for some modules such as `xul.dll` the Timestamp and ImageSize are from 115.1.0esr. The mismatch is likely related to the way the updater updates our `exe` and `dll` files: `xul.dll` was most likely already in 115.1.0esr version on disk at the moment we crashed, however the version loaded in the process was 115.0.3esr, as we can confirm by looking for the correct stack for the main thread in a debugger (see attachment).

On Windows, debuggers rely on Timestamp and ImageSize to collect a DLL from a debug server before using it to walk the stacks of threads, using [what Breakpad calls the `code_id`, which is derived from those](https://searchfox.org/mozilla-central/source/toolkit/crashreporter/google-breakpad/src/processor/minidump.cc#2040-2049). Because of the mismatch, Socorro (and debuggers) currently incorrectly uses the 115.1.0esr version of the mismatched DLLs to parse the crash dump from above. As a consequence, these kind of crashes can end up in the wrong signature, and with the wrong proto signature (see attachment).

There are multiple questions here that would require their own investigation, and possibly their own bug:

1. How common is this situation?
2. Is it possible to rewrite the updater and/or the crash reporter to avoid this kind of mismatch? Can we preserve the information that the file on disk changed its version because it could be relevant for the investigation?
3. Can we add heuristics in the Socorro backend to detect this situation and try with the two versions of the DLL and to guess which source of information seems more correct?
4. How can we adapt the Socorro webapp to warn about such mismatch?

Note that my own testing suggests that there could be situations where there is a mismatch and it is the `code_id` that is correct and not the version information. I obtained [this](https://crash-stats.mozilla.org/report/index/bce6770e-05a6-4613-914f-96f6e0231025#tab-modules) example crash report by forcing a crash after manually renaming `xul.dll` on disk while the 115.0.3esr process is running and putting version 115.1.0esr of the DLL in its place. In this report, the situation is somehow reversed: the version information says 115.1.0esr but the TimeStamp and Signature shown by my debugger are correctly from 115.0.3esr. I'm not sure that my test matches how the updater works though, so maybe this situation does not happen with legit crashes.

As initially shared by [:rkraesig], we seem to receive crash reports such as [this one](https://crash-stats.mozilla.org/report/index/92d1bb82-33fe-4e9c-8e2f-c96ac0230804), where the module version information is not coherent with the Timestamp and ImageSize.

In this example, the version information stored in the crash dump matches with Firefox 115.0.3esr, but for some modules such as `xul.dll` the Timestamp and ImageSize are from 115.1.0esr. The mismatch is likely related to the way the updater updates our `exe` and `dll` files: `xul.dll` was most likely already in 115.1.0esr version on disk at the moment we crashed, however the version loaded in the process was 115.0.3esr, as we can confirm by looking for the correct stack for the main thread in a debugger (see attachment).

On Windows, debuggers rely on Timestamp and ImageSize to collect a DLL from a debug server before using it to walk the stacks of threads, using [what Breakpad calls the `code_id`, which is derived from those](https://searchfox.org/mozilla-central/source/toolkit/crashreporter/google-breakpad/src/processor/minidump.cc#2040-2049). Because of the mismatch, Socorro (and debuggers) currently incorrectly uses the 115.1.0esr version of the mismatched DLLs to parse the crash dump from above. As a consequence, these kind of crashes can end up in the wrong signature, and with the wrong proto signature (see attachment).

There are multiple questions here that would require their own investigation, and possibly their own bug:

1. How common is this situation?
2. Is it possible to rewrite the updater and/or the crash reporter to avoid this kind of mismatch? Can we preserve the information that the file on disk changed its version because it could be relevant for the investigation?
3. Can we add heuristics in the Socorro backend to detect this situation and try with the two versions of the DLL then guess which source of information seems more correct?
4. How can we adapt the Socorro webapp to warn about such mismatch?

Note that my own testing suggests that there could be situations where there is a mismatch and it is the `code_id` that is correct and not the version information. I obtained [this](https://crash-stats.mozilla.org/report/index/bce6770e-05a6-4613-914f-96f6e0231025#tab-modules) example crash report by forcing a crash after manually renaming `xul.dll` on disk while the 115.0.3esr process is running and putting version 115.1.0esr of the DLL in its place. In this report, the situation is somehow reversed: the version information says 115.1.0esr but the TimeStamp and Signature shown by my debugger are correctly from 115.0.3esr. I'm not sure that my test matches how the updater works though, so maybe this situation does not happen with legit crashes.

As initially shared by [:rkraesig], we seem to receive crash reports such as [this one](https://crash-stats.mozilla.org/report/index/92d1bb82-33fe-4e9c-8e2f-c96ac0230804), where the module version information is not coherent with the Timestamp and ImageSize.

In this example, the version information stored in the crash dump matches with Firefox 115.0.3esr, but for some modules such as `xul.dll` the Timestamp and ImageSize are from 115.1.0esr. The mismatch is likely related to the way the updater updates our `exe` and `dll` files: `xul.dll` was most likely already in 115.1.0esr version on disk at the moment we crashed, however the version loaded in the process was 115.0.3esr, as we can confirm by looking for the correct stack for the main thread in a debugger (see attachment).

On Windows, debuggers rely on Timestamp and ImageSize to collect a DLL from a debug server before using it to walk the stacks of threads, using [what Breakpad calls the `code_id`, which is derived from those](https://searchfox.org/mozilla-central/source/toolkit/crashreporter/google-breakpad/src/processor/minidump.cc#2040-2049). Because of the mismatch, Socorro (and debuggers) currently incorrectly uses the 115.1.0esr version of the mismatched DLLs to parse the crash dump from above. As a consequence, these kind of crashes can end up in the wrong signature, and with the wrong proto signature (see attachment).

There are multiple questions here that would require their own investigation, and possibly their own bug:

1. How common is this situation?
2. Is it possible to rewrite the updater and/or the crash reporter to avoid this kind of mismatch? Can we preserve the information that the file on disk changed its version because it could be relevant for the investigation?
3. Can we add heuristics in the Socorro backend to detect this situation and try with the two versions of the DLL then guess which source of information seems more correct?
4. How can we adapt the Socorro webapp to warn about such mismatch?

Note that my own testing suggests that there could be situations where there is a mismatch and it is the `code_id` that is correct and not the version information. I obtained [this](https://crash-stats.mozilla.org/report/index/bce6770e-05a6-4613-914f-96f6e0231025#tab-modules) example crash dump by forcing a crash after manually renaming `xul.dll` on disk while the 115.0.3esr process is running and putting version 115.1.0esr of the DLL in its place. In this dump, the situation is somehow reversed: the version information says 115.1.0esr but the TimeStamp and Signature shown by my debugger are correctly from 115.0.3esr. I'm not sure that my test matches how the updater works though, so maybe this situation does not happen with legit crashes.

As initially shared by [:rkraesig], we seem to receive crash reports such as [this one](https://crash-stats.mozilla.org/report/index/92d1bb82-33fe-4e9c-8e2f-c96ac0230804), where the module version information is not coherent with the Timestamp and ImageSize.

In this example, the version information stored in the crash dump matches with Firefox 115.0.3esr, but for some modules such as `xul.dll` the Timestamp and ImageSize are from 115.1.0esr. The mismatch is likely related to the way the updater updates our `exe` and `dll` files: `xul.dll` was most likely already in 115.1.0esr version on disk at the moment we crashed, however the version loaded in the process was 115.0.3esr, as we can confirm by looking for the correct stack for the main thread in a debugger (see attachment).

On Windows, debuggers rely on Timestamp and ImageSize to collect a DLL from a debug server before using it to walk the stacks of threads, using [what Breakpad calls the `code_id`, which is derived from those](https://searchfox.org/mozilla-central/source/toolkit/crashreporter/google-breakpad/src/processor/minidump.cc#2040-2049). Presumably because of this mismatch, Socorro (and debuggers) currently incorrectly uses the 115.1.0esr version of the mismatched DLLs to parse the crash dump from above. As a consequence, these kind of crashes can end up in the wrong signature, and with the wrong proto signature (see attachment).

There are multiple questions here that would require their own investigation, and possibly their own bug:

1. How common is this situation?
2. Is it possible to rewrite the updater and/or the crash reporter to avoid this kind of mismatch? Can we preserve the information that the file on disk changed its version because it could be relevant for the investigation?
3. Can we add heuristics in the Socorro backend to detect this situation and try with the two versions of the DLL then guess which source of information seems more correct?
4. How can we adapt the Socorro webapp to warn about such mismatch?

Note that my own testing suggests that there could be situations where there is a mismatch and it is the `code_id` that is correct and not the version information. I obtained [this](https://crash-stats.mozilla.org/report/index/bce6770e-05a6-4613-914f-96f6e0231025#tab-modules) example crash dump by forcing a crash after manually renaming `xul.dll` on disk while the 115.0.3esr process is running and putting version 115.1.0esr of the DLL in its place. In this dump, the situation is somehow reversed: the version information says 115.1.0esr but the TimeStamp and Signature shown by my debugger are correctly from 115.0.3esr. I'm not sure that my test matches how the updater works though, so maybe this situation does not happen with legit crashes.

[As initially discovered by [:rkraesig]](https://bugzilla.mozilla.org/show_bug.cgi?id=1518612#c13), we seem to receive crash reports such as [this one](https://crash-stats.mozilla.org/report/index/92d1bb82-33fe-4e9c-8e2f-c96ac0230804), where the module version information is not coherent with the Timestamp and ImageSize.

In this example, the version information stored in the crash dump matches with Firefox 115.0.3esr, but for some modules such as `xul.dll` the Timestamp and ImageSize are from 115.1.0esr. The mismatch is likely related to the way the updater updates our `exe` and `dll` files: `xul.dll` was most likely already in 115.1.0esr version on disk at the moment we crashed, however the version loaded in the process was 115.0.3esr, as we can confirm by looking for the correct stack for the main thread in a debugger (see attachment).

On Windows, debuggers rely on Timestamp and ImageSize to collect a DLL from a debug server before using it to walk the stacks of threads, using [what Breakpad calls the `code_id`, which is derived from those](https://searchfox.org/mozilla-central/source/toolkit/crashreporter/google-breakpad/src/processor/minidump.cc#2040-2049). Presumably because of this mismatch, Socorro (and debuggers) currently incorrectly uses the 115.1.0esr version of the mismatched DLLs to parse the crash dump from above. As a consequence, these kind of crashes can end up in the wrong signature, and with the wrong proto signature (see attachment).

There are multiple questions here that would require their own investigation, and possibly their own bug:

1. How common is this situation?
2. Is it possible to rewrite the updater and/or the crash reporter to avoid this kind of mismatch? Can we preserve the information that the file on disk changed its version because it could be relevant for the investigation?
3. Can we add heuristics in the Socorro backend to detect this situation and try with the two versions of the DLL then guess which source of information seems more correct?
4. How can we adapt the Socorro webapp to warn about such mismatch?

Note that my own testing suggests that there could be situations where there is a mismatch and it is the `code_id` that is correct and not the version information. I obtained [this](https://crash-stats.mozilla.org/report/index/bce6770e-05a6-4613-914f-96f6e0231025#tab-modules) example crash dump by forcing a crash after manually renaming `xul.dll` on disk while the 115.0.3esr process is running and putting version 115.1.0esr of the DLL in its place. In this dump, the situation is somehow reversed: the version information says 115.1.0esr but the TimeStamp and Signature shown by my debugger are correctly from 115.0.3esr. I'm not sure that my test matches how the updater works though, so maybe this situation does not happen with legit crashes.