Signs point to omni.ja truncation.
We looked at an instance of
@ ErrorLoadingSheet last week, it includes these annotations:
Error loading sheet: resource://gre/res/svg.css
NS_ERROR_FILE_CORRUPTION reason: nsJARInputStream: !mZs.next_in
Real location: jar:file:///C:/Program%20Files/Mozilla%20Firefox/omni.ja!/res/svg.css
GRE directory: C:\Program Files\Mozilla Firefox
Interesting files in the GRE directory:
C:\Program Files\Mozilla Firefox\browser\chrome.manifest (0 bytes, crc32 = 0x00000000)
C:\Program Files\Mozilla Firefox\browser\omni.ja (41437720 bytes, crc32 = 0x7059911f)
C:\Program Files\Mozilla Firefox\chrome.manifest (0 bytes, crc32 = 0x00000000)
C:\Program Files\Mozilla Firefox\omni.ja (17429973 bytes, crc32 = 0x326fbb3c)
Contents of chrome.manifest:
GRE omnijar URI string: jar:file:///C:/Program%20Files/Mozilla%20Firefox/omni.ja!/
Interesting files in the GRE omnijar:
chrome/chrome.manifest (2784 bytes, crc32 = 0x0a1409b4)
res/svg.css (2159 bytes, crc32 = 0xeae4bb67)
The released 65.0.2 Windows amd64 en-US
omni.ja should have that size, but the CRC should be
browser\omni.ja is also the right size but the CRC should be
a7d7a15d. The later CRCs (from the files in the GRE omnijar) are correct, but these come from the Zip central directory and thus don't reflect the actual data on disk.
It turns out that the reported CRCs can be explained if the files are truncated (specifically, filled with zeroes past a certain point, this can be checked with crctrunc):
0xb00000. Truncating the files in this way produces the same crash in my testing on 64 bit Windows 10.
I'm not sure what to blame this on. It would be interesting to see if the Android crashes look the same, as then there would be a totally different install/update process and filesystem responsible. One thing we could do to catch this earlier would be to check for the Zip "end of central directory" record at the end of the file.
I haven't yet had the opportunity to check more of these reports; this crash could still be a catch-all for any omnijar corruption. We may want to restore the annotations removed with https://phabricator.services.mozilla.com/D14050 , with some sort of support for Android (as Android crashes like edaf2b37-2f6c-4667-8bda-3e3ba0190325 don't report the CRC for