Crash in [@ qcms::transform_avx::qcms_transform_data_template_lut_avx]
Categories
(Core :: Graphics: Color Management, defect)
Tracking
()
People
(Reporter: yannis, Assigned: yannis)
References
(Blocks 1 open bug)
Details
(Keywords: csectype-bounds, sec-vector, Whiteboard: [adv-main131-][adv-esr128.3-][adv-esr115.16-])
Crash Data
Attachments
(6 files, 1 obsolete file)
|
5.98 KB,
image/png
|
Details | |
|
70.72 KB,
application/json
|
Details | |
|
48 bytes,
text/x-phabricator-request
|
tjr
:
sec-approval+
|
Details | Review |
|
48 bytes,
text/x-phabricator-request
|
phab-bot
:
approval-mozilla-beta+
|
Details | Review |
|
48 bytes,
text/x-phabricator-request
|
phab-bot
:
approval-mozilla-esr128+
|
Details | Review |
|
48 bytes,
text/x-phabricator-request
|
diannaS
:
approval-mozilla-esr115+
|
Details | Review |
Example crash: here
Top frames in call stack:
00 xul!qcms::transform_avx::qcms_transform_data_template_lut_avx+0x24f [/builds/worker/checkouts/gecko/gfx/qcms/src/transform_avx.rs @ 142]
01 xul!qcms::transform_avx::qcms_transform_data_rgba_out_lut_avx+0x27d [/builds/worker/checkouts/gecko/gfx/qcms/src/transform_avx.rs @ 217]
02 xul!mozilla::image::ColorManagementFilter<mozilla::image::SwizzleFilter<mozilla::image::DownscalingFilter<mozilla::image::SurfaceSink> > >::DoAdvanceRowFromBuffer+0x21 [/builds/worker/checkouts/gecko/image/SurfaceFilters.h @ 164]
03 xul!mozilla::image::SurfaceFilter::AdvanceRow+0x17 [/builds/worker/checkouts/gecko/image/SurfacePipe.h @ 141]
04 xul!mozilla::image::SurfaceFilter::WriteBuffer+0x23 [/builds/worker/checkouts/gecko/image/SurfacePipe.h @ 300]
05 xul!mozilla::image::SurfacePipe::WriteBuffer+0x2a [/builds/worker/checkouts/gecko/image/SurfacePipe.h @ 705]
06 xul!mozilla::image::nsPNGDecoder::WriteRow+0x59 [/builds/worker/checkouts/gecko/image/decoders/nsPNGDecoder.cpp @ 853]
07 xul!MOZ_PNG_push_have_row+0x6 [/builds/worker/checkouts/gecko/media/libpng/pngpread.c @ 1217]
08 xul!MOZ_PNG_push_proc_row+0x1c2 [/builds/worker/checkouts/gecko/media/libpng/pngpread.c @ 0]
09 xul!MOZ_PNG_proc_IDAT_data+0x2e1 [/builds/worker/checkouts/gecko/media/libpng/pngpread.c @ 887]
0a xul!MOZ_PNG_push_read_IDAT+0x7a4 [/builds/worker/checkouts/gecko/media/libpng/pngpread.c @ 763]
0b xul!MOZ_PNG_proc_some_data+0xbef [/builds/worker/checkouts/gecko/media/libpng/pngpread.c @ 115]
0c xul!MOZ_PNG_process_data+0xc6e [/builds/worker/checkouts/gecko/media/libpng/pngpread.c @ 46]
0d xul!mozilla::image::nsPNGDecoder::ReadPNGData+0xd70 [/builds/worker/checkouts/gecko/image/decoders/nsPNGDecoder.cpp @ 394]
Somehow I can reproduce this crash very reliably in Nightly and Release as follows (that's how I discovered it) [edit: see better STR in comment 1]:
- Install
TTD.exeas a standalone command line utility. - In Firefox, install and enable the 1password addon. I need to have logged into my account once through the addon.
- In Firefox, navigate to
about:configand setbrowser.launcherProcess.enabledtofalse. Close Firefox. - In an administrator command prompt, run Firefox through TTD, e.g.
TTD.exe -out C:\traces "C:\Program Files\Firefox Nightly\firefox.exe". - Do other things while waiting for Firefox to start (it will be slow because of TTD).
- Click the Firefox window and navigate to
lemonde.fr.
The navigation step may be optional, however the 1password addon seems required. Based on crashes from other users, maybe the Grammarly and/or Rakuten addons could also help reproduce the crash.
Because by definition TTD records a time-travel debugging trace, I have such a trace available that I can share privately to help the investigation.
| Assignee | ||
Comment 1•2 years ago
|
||
After all this does not appear to be related to addons. In my case the call stack is the following:
00 xul!qcms::transform_avx::qcms_transform_data_template_lut_avx+0x23b [/builds/worker/checkouts/gecko/gfx/qcms/src/transform_avx.rs @ 140]
01 xul!qcms::transform_avx::qcms_transform_data_bgra_out_lut_avx+0x269 [/builds/worker/checkouts/gecko/gfx/qcms/src/transform_avx.rs @ 227]
02 xul!mozilla::image::ColorManagementFilter<mozilla::image::SwizzleFilter<mozilla::image::SurfaceSink> >::DoAdvanceRowFromBuffer+0x21 [/builds/worker/checkouts/gecko/image/SurfaceFilters.h @ 164]
03 xul!mozilla::image::SurfaceFilter::AdvanceRow+0x17 [/builds/worker/checkouts/gecko/image/SurfacePipe.h @ 141]
04 xul!mozilla::image::SurfaceFilter::WriteBuffer+0x2a [/builds/worker/checkouts/gecko/image/SurfacePipe.h @ 300]
05 xul!mozilla::image::SurfacePipe::WriteBuffer+0x31 [/builds/worker/checkouts/gecko/image/SurfacePipe.h @ 705]
06 xul!mozilla::image::nsBMPDecoder::FinishRow+0x31 [/builds/worker/checkouts/gecko/image/decoders/nsBMPDecoder.cpp @ 418]
07 xul!mozilla::image::nsBMPDecoder::ReadPixelRow+0xa67 [/builds/worker/checkouts/gecko/image/decoders/nsBMPDecoder.cpp @ 1127]
08 xul!mozilla::image::nsBMPDecoder::DoDecode::<lambda_4>::operator()+0xaa4 [/builds/worker/checkouts/gecko/image/decoders/nsBMPDecoder.cpp @ 459]
09 xul!mozilla::image::StreamingLexer<mozilla::image::nsBMPDecoder::State,16>::BufferedRead+0x57 [/builds/worker/checkouts/gecko/image/StreamingLexer.h @ 604]
0a xul!mozilla::image::StreamingLexer<mozilla::image::nsBMPDecoder::State,16>::Lex+0x13d [/builds/worker/checkouts/gecko/image/StreamingLexer.h @ 468]
0b xul!mozilla::image::nsBMPDecoder::DoDecode+0x162 [/builds/worker/checkouts/gecko/image/decoders/nsBMPDecoder.cpp @ 431]
0c xul!mozilla::image::Decoder::Decode+0x13e [/builds/worker/checkouts/gecko/image/Decoder.cpp @ 180]
0d xul!mozilla::image::nsICODecoder::FlushContainedDecoder+0x38 [/builds/worker/checkouts/gecko/image/decoders/nsICODecoder.cpp @ 693]
0e xul!mozilla::image::nsICODecoder::ReadBIH+0x114 [/builds/worker/checkouts/gecko/image/decoders/nsICODecoder.cpp @ 429]
0f xul!mozilla::image::StreamingLexer<mozilla::image::ICOState,32>::Lex+0x199 [/builds/worker/checkouts/gecko/image/StreamingLexer.h @ 468]
10 xul!mozilla::image::nsICODecoder::DoDecode+0x1c3 [/builds/worker/checkouts/gecko/image/decoders/nsICODecoder.cpp @ 647]
11 xul!mozilla::image::Decoder::Decode+0x13e [/builds/worker/checkouts/gecko/image/Decoder.cpp @ 180]
12 xul!mozilla::image::DecodedSurfaceProvider::Run+0x4f [/builds/worker/checkouts/gecko/image/DecodedSurfaceProvider.cpp @ 128]
13 xul!mozilla::image::DecodePool::SyncRunIfPossible+0x2a [/builds/worker/checkouts/gecko/image/DecodePool.cpp @ 200]
14 xul!mozilla::image::LaunchDecodingTask+0x31a [/builds/worker/checkouts/gecko/image/RasterImage.cpp @ 1137]
15 xul!mozilla::image::RasterImage::Decode+0x3ee [/builds/worker/checkouts/gecko/image/RasterImage.cpp @ 1249]
16 xul!mozilla::image::RasterImage::LookupFrame+0x60a [/builds/worker/checkouts/gecko/image/RasterImage.cpp @ 385]
17 xul!mozilla::image::RasterImage::GetFrameAtSize+0x98 [/builds/worker/checkouts/gecko/image/RasterImage.cpp @ 581]
18 xul!mozilla::image::imgTools::EncodeScaledImage+0xdc [/builds/worker/checkouts/gecko/image/imgTools.cpp @ 517]
19 xul!nsFaviconService::OptimizeIconSizes+0x3a3 [/builds/worker/checkouts/gecko/toolkit/components/places/nsFaviconService.cpp @ 700]
1a xul!nsFaviconService::SetFaviconForPage+0x4a5 [/builds/worker/checkouts/gecko/toolkit/components/places/nsFaviconService.cpp @ 326]
Thanks to the time-travel trace I saw that this occurs with the favicon for www.google.com and so here are my refined STR now:
- Install
TTD.exeas a standalone command line utility. - Install Firefox, navigate to
google.com, make sure the favicon appears. - In Firefox, navigate to
about:configand setbrowser.launcherProcess.enabledtofalse. Close Firefox. - In an administrator command prompt, run Firefox through TTD, e.g.
TTD.exe -out C:\traces "C:\Program Files\Firefox Nightly\firefox.exe". - Once Firefox is started, navigate to
google.comand wait for the favicon to appear.
Expected behavior: the favicon appears and colors are correct.
What happens: we crash (with official Nightly and Release binaries) or the colors of the favicon are yellowish (in a custom build or with ESR 115 binaries).
| Assignee | ||
Comment 2•2 years ago
|
||
This is what the favicon looks like after following the STR in builds that don't crash (local builds, ESR 115).
| Assignee | ||
Updated•2 years ago
|
| Comment hidden (obsolete) |
| Comment hidden (obsolete) |
| Assignee | ||
Updated•2 years ago
|
| Assignee | ||
Comment 5•2 years ago
|
||
Adding about:support for completeness.
| Assignee | ||
Updated•2 years ago
|
| Assignee | ||
Comment 6•2 years ago
|
||
It looks like, on my machine, this is likely caused by a poorly handled instruction in TTD's CPU emulation. Here is what the critical point looks like in my time-travel trace:
Time Travel Position: 2FE1DB:F63
xul!qcms::transform_avx::qcms_transform_data_template_lut_avx+0x4b [inlined in xul!qcms::transform_avx::qcms_transform_data_rgba_out_lut_avx+0x82]:
00007ffd`87a101f2 c4e27d1a5120 vbroadcastf128 ymm2,oword ptr [rcx+20h] ds:000002e8`823b8960=000000003f80036537882400386dd000
0:028> r xmm2
xmm2= -1.#QNAN -1.#QNAN 1.4013e-045 4.48416e-044
0:028> df rcx+20 L4
000002e8`823b8960 5.6698918e-005 1.6229227e-005 1.0001036 0
0:028> t
Time Travel Position: 2FE1DB:F64
xul!qcms::transform_avx::qcms_transform_data_template_lut_avx+0x51 [inlined in xul!qcms::transform_avx::qcms_transform_data_rgba_out_lut_avx+0x88]:
00007ffd`87a101f8 4983f901 cmp r9,1
0:028> r xmm2
xmm2= -1.#QNAN -1.#QNAN 1.62292e-005 5.66989e-005
xmm2 holds -1.#QNAN -1.#QNAN 1.4013e-045 4.48416e-044 and then we broadcast from memory the values 5.6698918e-005 1.6229227e-005 1.0001036 0 but only the two first values get broadcasted, the third float stays at an invalid value of -1.#QNAN instead of taking the broadcasted value 1.0001036.
Here is the execution of the same instruction at the same point without TTD instrumentation:
Breakpoint 0 hit
xul!qcms::transform_avx::qcms_transform_data_template_lut_avx+0x4b [inlined in xul!qcms::transform_avx::qcms_transform_data_rgba_out_lut_avx+0x82]:
00007ffd`661c01f2 c4e27d1a5120 vbroadcastf128 ymm2,oword ptr [rcx+20h] ds:00000230`0d03d2a0=000000003f80036537882400386dd000
0:030> r xmm2
xmm2= -1.#QNAN -1.#QNAN 1.4013e-045 4.48416e-044
0:030> df rcx+20 L4
00000230`0d03d2a0 5.6698918e-005 1.6229227e-005 1.0001036 0
0:030> t
xul!qcms::transform_avx::qcms_transform_data_template_lut_avx+0x51 [inlined in xul!qcms::transform_avx::qcms_transform_data_rgba_out_lut_avx+0x88]:
00007ffd`661c01f8 4983f901 cmp r9,1
0:030> r xmm2
xmm2= 0 1.0001 1.62292e-005 5.66989e-005
I'll try to make a small reproducer and email Microsoft about it. However, I highly doubt that people in the wild are relying on TTD to run into this crash, so I'm marking as security and will explain in more detail why next week (potential out-of-bounds read).
Updated•2 years ago
|
Comment 7•2 years ago
|
||
For the in the wild crashes, only 12 of the 77 crashes that come up in the search do not have a non-zero bit flip confidence. So seems like mostly hardware type issues.
| Assignee | ||
Comment 8•2 years ago
|
||
I have confirmed the TTD CPU emulation bug and emailed our contacts at Microsoft about it. The effect of the TTD CPU emulation bug can be somewhat replicated by replacing:
let mat2: __m256 = _mm256_broadcast_ps(&*((*mat.offset(2isize)).as_ptr() as *const __m128));
With:
let mut mat2 = *mat.offset(2isize);
mat2[2] = f32::NAN;
let mat2: __m256 = _mm256_broadcast_ps(&*(mat2.as_ptr() as *const __m128));
The NaN value in mat2[2] propagates through the various computations until it eventually reaches _mm256_cvtps_epi32:
unsafe extern "C" fn qcms_transform_data_template_lut_avx<F: Format>(
transform: &qcms_transform,
mut src: *const u8,
mut dest: *mut u8,
mut length: usize,
) {
// ...
while length > 3 {
// ...
vec_b = _mm256_mul_ps(vec_b, mat2);
// ...
vec_r = _mm256_add_ps(vec_r, _mm256_add_ps(vec_g, vec_b));
vec_r = _mm256_max_ps(min, vec_r);
vec_r = _mm256_min_ps(max, vec_r);
result = _mm256_mul_ps(vec_r, scale);
_mm256_store_si256(output as *mut __m256i, _mm256_cvtps_epi32(result));
// ...
*dest.add(F::kBIndex) = *otdata_b.offset(*output.offset(2isize) as isize);
// ...
*dest.add(F::kBIndex + components as usize) =
*otdata_b.offset(*output.offset(6isize) as isize);
// ...
}
// ...
Then _mm256_cvtps_epi32 converts the NaN float value to the 32-bit integer 0x80000000, so this is what output[2] and output[6] will receive. We then use this 0x80000000 value as an index of the 8192-byte array otdata_b, resulting in an out-of-bounds read.
More generally stated, the max/min/mul sequence on vec_r is present to guarantee that output only contains values between 0 and 8191, but it fails to do that if the vec_r has somehow been contaminated by NaN values. I tested locally with all non-NaN f32 values and confirmed that the issue only exists with NaN values. I think such NaN contamination can only occur through transform.matrix and transform.input_gamma_table_{r,g,b}.
In order to fix this, which would let us use current versions TTD without crashing, we could simply swap the order of arguments either in the _mm256_min_ps or in the _mm256_max_ps operation. The description for these operations (max, min) says: "If only one value is a NaN (SNaN or QNaN) for this instruction, the second operand (source operand), either a NaN or a valid floating-point value, is written to the result.". Swapping in _mm256_max_ps would map NaN to 0, while swapping in _mm256_min_ps would map NaN to 8191.
| Assignee | ||
Comment 9•2 years ago
|
||
Microsoft TTD has a CPU emulation bug that makes Firefox crash when ran
under TTD. This patch mitigates the issue.
Updated•2 years ago
|
Updated•2 years ago
|
| Assignee | ||
Comment 11•2 years ago
|
||
[Tracking Requested - why for this release]: The bug is in pretty old code so all builds are affected. Not sure if Thunderbird uses that code.
| Assignee | ||
Comment 12•2 years ago
|
||
Microsoft TTD has a CPU emulation bug that makes Firefox crash when ran
under TTD. This patch mitigates the issue.
Original Revision: https://phabricator.services.mozilla.com/D220829
Updated•2 years ago
|
| Assignee | ||
Comment 13•2 years ago
|
||
Comment on attachment 9422091 [details]
Bug 1915813 - Fix a crash with Microsoft TTD. r=#gfx-reviewers
Security Approval Request
- How easily could an exploit be constructed based on the patch?: Even though this bug involves a memory corruption it looks unlikely that it can be used to construct an exploit. First, it is unclear how to trigger the corruption without the CPU emulation bug, if at all possible. Second, the memory corruption is an OOB read at a fixed large offset, and it seems unlikely that an attacker can reliably put an object that would be interesting for them to read as part of an exploit at the proper location.
- Do comments in the patch, the check-in comment, or tests included in the patch paint a bulls-eye on the security problem?: No
- Which branches (beta, release, and/or ESR) are affected by this flaw, and do the release status flags reflect this affected/unaffected state correctly?: beta, release, ESR
- If not all supported branches, which bug introduced the flaw?: None
- Do you have backports for the affected branches?: Yes
- If not, how different, hard to create, and risky will they be?:
- How likely is this patch to cause regressions; how much testing does it need?: Very unlikely to cause regressions, the patch just swaps argument order on an operation to handle NaN values differently, non-NaN values are treated the same no matter the order of arguments.
- Is the patch ready to land after security approval is given?: Yes
- Is Android affected?: Unknown
| Assignee | ||
Comment 14•2 years ago
•
|
||
I guess Android x86 could be affected by the OOB (assuming it is reachable without the CPU bug), not sure if that counts.
Comment 15•2 years ago
|
||
Comment on attachment 9422091 [details]
Bug 1915813 - Fix a crash with Microsoft TTD. r=#gfx-reviewers
Approved to land and request uplift
Updated•2 years ago
|
Comment 16•1 year ago
|
||
beta Uplift Approval Request
- User impact if declined: Crashing when using Firefox under Microsoft TTD and potential presence of an OOB read memory corruption.
- Code covered by automated testing: yes
- Fix verified in Nightly: no
- Needs manual QE test: no
- Steps to reproduce for manual QE testing: -
- Risk associated with taking this patch: Low
- Explanation of risk level: The patch just changes the order of arguments on an operation whose behavior only depends on argument order for NaN values. The patch does not affect the flow of computation for non-NaN values, and NaN values are unexpected and currently cause crashes.
- String changes made/needed: No
- Is Android affected?: no
Comment 17•1 year ago
|
||
Comment 18•1 year ago
|
||
| Assignee | ||
Comment 19•1 year ago
•
|
||
Fix verified in the latest Nightly. The CPU emulation bug still exists, so I still get a NaN value flowing around when it should not:
Time Travel Position: 2CBB3E:1C90
xul!qcms::transform_avx::qcms_transform_data_template_lut_avx+0x4b [inlined in xul!qcms::transform_avx::qcms_transform_data_rgba_out_lut_avx+0x82]:
00007ffb`514e9812 c4e27d1a5120 vbroadcastf128 ymm2,oword ptr [rcx+20h] ds:00000223`f603e960=000000003f80036537882400386dd000
0:027> rY
ymm2=0 0 0 0 -1.#QNAN -1.#QNAN 1.4013e-045 3.36312e-044
0:027> t
Time Travel Position: 2CBB3E:1C91
0:027> rY
ymm2=0 1.0001 1.62292e-005 5.66989e-005 -1.#QNAN -1.#QNAN 1.62292e-005 5.66989e-005
And the NaN value still propagates a bit, but eventually the vmaxps operation replaces it by zero:
Time Travel Position: 2CBB3E:1CAF
xul!qcms::transform_avx::qcms_transform_data_template_lut_avx+0x1a8 [inlined in xul!qcms::transform_avx::qcms_transform_data_rgba_out_lut_avx+0x1df]:
00007ffb`514e996f c5d45feb vmaxps ymm5,ymm5,ymm3
0:027> rY
ymm3=0 0 0 0 0 0 0 0
ymm5=0 0 0 0 -1.#QNAN -1.#QNAN 0 0
0:027> t
Time Travel Position: 2CBB3E:1CB0
0:027> rY
ymm5=0 0 0 0 0 0 0 0
And we end up converting 0 rather than NaN for the value that we use as an offset (as described in comment 8), so we don't crash.
Comment 20•1 year ago
|
||
beta Uplift Approval Request
- User impact if declined: Crashing when using Firefox under Microsoft TTD and potential presence of an OOB read memory corruption.
- Code covered by automated testing: yes
- Fix verified in Nightly: yes
- Needs manual QE test: no
- Steps to reproduce for manual QE testing: -
- Risk associated with taking this patch: Low
- Explanation of risk level: The patch just changes the order of arguments on an operation whose behavior only depends on argument order for NaN values. The patch does not affect the flow of computation for non-NaN values, and NaN values are unexpected and currently cause crashes.
- String changes made/needed: No
- Is Android affected?: no
Updated•1 year ago
|
Comment 21•1 year ago
|
||
| uplift | ||
Updated•1 year ago
|
Updated•1 year ago
|
| Assignee | ||
Comment 22•1 year ago
|
||
Microsoft TTD has a CPU emulation bug that makes Firefox crash when ran
under TTD. This patch mitigates the issue.
Original Revision: https://phabricator.services.mozilla.com/D220829
Updated•1 year ago
|
| Assignee | ||
Comment 23•1 year ago
|
||
Microsoft TTD has a CPU emulation bug that makes Firefox crash when ran
under TTD. This patch mitigates the issue.
Original Revision: https://phabricator.services.mozilla.com/D220829
Updated•1 year ago
|
Comment 24•1 year ago
|
||
esr128 Uplift Approval Request
- User impact if declined: Crashing when using Firefox under Microsoft TTD and potential presence of an OOB read memory corruption.
- Code covered by automated testing: yes
- Fix verified in Nightly: yes
- Needs manual QE test: no
- Steps to reproduce for manual QE testing: -
- Risk associated with taking this patch: Low
- Explanation of risk level: The patch just changes the order of arguments on an operation whose behavior only depends on argument order for NaN values. The patch does not affect the flow of computation for non-NaN values, and NaN values are unexpected and currently cause crashes.
- String changes made/needed: No
- Is Android affected?: no
Comment 25•1 year ago
|
||
esr115 Uplift Approval Request
- User impact if declined: Crashing when using Firefox under Microsoft TTD and potential presence of an OOB read memory corruption.
- Code covered by automated testing: yes
- Fix verified in Nightly: yes
- Needs manual QE test: no
- Steps to reproduce for manual QE testing: -
- Risk associated with taking this patch: Low
- Explanation of risk level: The patch just changes the order of arguments on an operation whose behavior only depends on argument order for NaN values. The patch does not affect the flow of computation for non-NaN values, and NaN values are unexpected and currently cause crashes.
- String changes made/needed: No
- Is Android affected?: no
Updated•1 year ago
|
Comment 26•1 year ago
|
||
Comment on attachment 9424364 [details]
Bug 1915813 - Fix a crash with Microsoft TTD.
Approved for 115.16esr
Comment 27•1 year ago
|
||
| uplift | ||
Updated•1 year ago
|
Updated•1 year ago
|
Comment 28•1 year ago
|
||
| uplift | ||
| Assignee | ||
Comment 29•1 year ago
|
||
Microsoft has also confirmed that the CPU emulation bug will be fixed in the next release of TTD.exe.
Updated•1 year ago
|
Updated•1 year ago
|
Updated•1 year ago
|
Updated•1 year ago
|
Description
•