1626898 - (nv50_validate_tic) PHC crash in [@ nv50_validate_tic] (buffer overflow?)

Status: RESOLVED FIXED

(Core :: Graphics: WebRender, defect, P2)

Description

[Julien Cristau [:jcristau]]

This bug is for crash report bp-f1db01f8-55fd-4487-ab73-2d29f0200330.

Top 10 frames of crashing thread:

0 libgallium_dri.so nv50_validate_tic ../src/gallium/drivers/nouveau/nv50/nv50_tex.c:324
1 libgallium_dri.so nv50_validate_textures ../src/gallium/drivers/nouveau/nv50/nv50_tex.c:339
2 libgallium_dri.so nv50_state_validate ../src/gallium/drivers/nouveau/nv50/nv50_state_validate.c:549
3 libgallium_dri.so nv50_state_validate_3d ../src/gallium/drivers/nouveau/nv50/nv50_state_validate.c:572
4 libgallium_dri.so nv50_draw_vbo ../src/gallium/drivers/nouveau/nv50/nv50_vbo.c:799
5 libgallium_dri.so cso_draw_arrays ../src/gallium/auxiliary/cso_cache/cso_context.c:1756
6 libgallium_dri.so st_pbo_draw ../src/mesa/state_tracker/st_pbo.c:282
7 libgallium_dri.so try_pbo_upload_common ../src/mesa/state_tracker/st_cb_texture.c:1322
8 libgallium_dri.so st_TexSubImage ../src/mesa/state_tracker/st_cb_texture.c:1561
9 libgallium_dri.so <name omitted> ../src/mesa/main/teximage.c:3584

Alloc stack from PHC:

0 firefox-bin replace_calloc(unsigned long, unsigned long)+0xe4
1 libgallium_dri.so nouveau_buffer_create+0x31  ../src/gallium/drivers/nouveau/nouveau_buffer.c:644
2 libgallium_dri.so st_bufferobj_data+0x27a  ../src/mesa/main/bufferobj.c:2087
3 libgallium_dri.so <name omitted>+0xbe  ../src/mesa/main/bufferobj.c:2087
4 libgallium_dri.so <name omitted>+0xf3  ../src/mesa/main/bufferobj.c:2130
5 libxul.so webrender::device::gl::Device::create_upload_buffer+0x4f
6 libxul.so webrender::renderer::Renderer::update_gpu_cache+0x3bd7
7 libxul.so webrender::renderer::Renderer::render_impl+0x69d7
8 libxul.so webrender::renderer::Renderer::render+0x46
9 libxul.so wr_renderer_render+0x68
10 libxul.so mozilla::wr::RendererOGL::UpdateAndRender(mozilla::Maybe<mozilla::gfx::IntSizeTyped<mozilla::gfx::UnknownUnits> > const&, mozilla::Maybe<mozilla::wr::ImageFormat> const&, mozilla::Maybe<mozilla::Range<unsigned char> > const&, bool, mozilla::wr::RendererStats*)+0x163
11 libxul.so mozilla::wr::RenderThread::UpdateAndRender(mozilla::wr::WrWindowId, mozilla::layers::BaseTransactionId<mozilla::VsyncIdType> const&, mozilla::TimeStamp const&, bool, mozilla::Maybe<mozilla::gfx::IntSizeTyped<mozilla::gfx::UnknownUnits> > const&, mozilla::Maybe<mozilla::wr::ImageFormat> const&, mozilla::Maybe<mozilla::Range<unsigned char> > const&, bool)+0x1fa
12 libxul.so mozilla::wr::RenderThread::HandleFrameOneDoc(mozilla::wr::WrWindowId, bool)+0x227
13 libxul.so mozilla::detail::RunnableMethodImpl<mozilla::wr::RenderThread*, void (mozilla::wr::RenderThread::*)(mozilla::wr::WrWindowId, bool), true, (mozilla::RunnableKind)0, mozilla::wr::WrWindowId, bool>::Run()+0x2f
14 libxul.so base::MessagePumpDefault::Run(base::MessagePump::Delegate*)+0x674
15 libxul.so MessageLoop::Run()+0x4f

I reported this to mesa, @karolherbst says:
"If I would make a guess, this issue is probably due to our multithreading issues we have. Essentially creating multiple OpenGL contexts within the same process leads to random crashes and is not something which is easily fixable as of right now.

Do you have the possibility to limit rendering to one thread? I know that this isn't a great solution here, but to actual fix this it requires a bigger rework of the driver and probably nothing we would backport to older versions of mesa anyway."

Filing this here in case we want to do something on our end.

Comment 1

[Jeff Muizelaar [:jrmuizel]]

The mesa issue link is broken. Julien can you provide the correct one?

We should probably just disable WebRender on nouveau because of this for now.

Comment 2

[Julien Cristau [:jcristau]]

The link is correct but I marked the issue confidential there out of caution...

Comment 3

[Jessie [:jbonisteel] pls NI]

Andrew - can you disable WR as per: https://bugzilla.mozilla.org/show_bug.cgi?id=1626898#c1

Comment 4

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1626898 - Remove NVIDIA Mesa drivers from Linux nightly WebRender allowlist.

Comment 5

[Andrew Osmond [:aosmond] (he/him)]

I don't think I need sec-approval for this to land, given it only ships on nightly configurations (hence no need to an uplift)?

Comment 6

[Julien Cristau [:jcristau]]

Correct.

Comment 7

[Julien Cristau [:jcristau]]

fixed on the mesa side:
https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/4424
https://gitlab.freedesktop.org/mesa/mesa/-/commit/1288ac7632b31a20497a0e75f374f66ce3d5bc3c

Comment 8

[Gabriele Svelto [:gsvelto]]

Note that we're seeing crashes all the way to Mesa 20.0.2.0. We probably need to block that version and every other one below.

https://crash-stats.mozilla.com/report/index/c63cd304-c9f9-4e5b-b53c-340de0200403

Comment 9

[Gabriele Svelto [:gsvelto]]

And BTW this should prevent a number of other fairly high volume Linux crashes such as nv30_fp_state_bind and nv30_draw_vbo.

Comment 10

[Andrew Osmond [:aosmond] (he/him)]

(In reply to Gabriele Svelto [:gsvelto] from comment #8)

Note that we're seeing crashes all the way to Mesa 20.0.2.0. We probably need to block that version and every other one below.

https://crash-stats.mozilla.com/report/index/c63cd304-c9f9-4e5b-b53c-340de0200403

I see that it is 20.1.0-devel in master. Do we know if they will do a point release with this fix, and thus I could just block up to 20.0.3.0, or need I block up to 20.1.0.0?

Comment 11

[Julien Cristau [:jcristau]]

The fix is marked cc: mesa-stable which I think means it might get backported to a point release. Maybe 20.0.5 (20.0.4 was just released today).

Comment 13

[Julien Cristau [:jcristau]]

mesa 20.0.5 was released this week with the fix.

Comment 15

[Gabriele Svelto [:gsvelto]]

The upstream bug in mesa has been fixed and given the volume went to zero it must have been rolled out to most major distros. For some reason the issue on mesa's issue tracker has been deleted. Either way I'm closing this as FIXED.