1692514 - Crash in [@ OOM | large | mozalloc_abort | webrender::renderer::upload::upload_to_texture_cache]

Status: RESOLVED INACTIVE

(Core :: Graphics: WebRender, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

11 crashes on 11 machines, all on Windows so far.

Crash report: https://crash-stats.mozilla.org/report/index/bcb23ef3-0d33-4515-b614-7ec090210206

MOZ_CRASH Reason: MOZ_CRASH()

Top 10 frames of crashing thread:

0 mozglue.dll mozalloc_abort memory/mozalloc/mozalloc_abort.cpp:33
1 mozglue.dll mozalloc_handle_oom memory/mozalloc/mozalloc_oom.cpp:51
2 xul.dll gkrust_shared::oom_hook::hook toolkit/library/rust/shared/lib.rs:134
3 xul.dll std::alloc::rust_oom ../e1884a8e3c3e813aada8254edfa120e85bf5ffca//library/std/src/alloc.rs:330
4 xul.dll alloc::alloc::__alloc_error_handler::__rg_oom ../e1884a8e3c3e813aada8254edfa120e85bf5ffca//library/alloc/src/alloc.rs:409
5 xul.dll _rust_alloc_error_handler 
6 xul.dll alloc::alloc::handle_alloc_error ../e1884a8e3c3e813aada8254edfa120e85bf5ffca//library/alloc/src/alloc.rs:363
7 xul.dll webrender::renderer::upload::upload_to_texture_cache gfx/wr/webrender/src/renderer/upload.rs:113
8 xul.dll webrender::renderer::Renderer::update_texture_cache gfx/wr/webrender/src/renderer/mod.rs:2396
9  @0xafb9bfe43f 

Comment 1

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

E.g. bp-6ed7fb3a-f03b-40be-92d0-bce320210211 has a better stacktrace.

Comment 2

[Nicolas Silva [:nical]]

This can happen when a page has to upload a large amount of texture data within a frame with the batched direct upload code path (Windows). For example lots of large blob images. We end up having to create a large amount of CPU-side staging buffers.

We could improve the situation by triggering the upload earlier. Instead of waiting for all updates to be written into staging buffers to kick the uploads, accumulate content into the staging buffer and trigger the immediate upload as soon as an allocation into the atlas fails, so that we can reuse the staging memory. THis would let us use a single staging memory buffer per texture type instead of some unbounded number depending on the amount of content to upload.

Unfortunately the upload code is written with very tight rust lifetime constraints which make it so that we can't use a new texture with the uploader as soon as we have started uploading because the texture needs to outlive the uploader, so it's tricky continue growing the number of textures in use while we have started the uploads. I think that doing this would require rewriting the upload code entirely, so that the texture uploader is responsible for managing the staging memory buffers and can start uploads earlier.

It's not so bad because the upload code is in a rough shape and a rewrite could take this critical part of webrender to a more maintainable state. On the other hand the crash volume is quite low so this isn't very high in my priority list.

Comment 3

[Wayne Mery (:wsmwk)]

There is a major spike in crashes for 94.0a1 starting with buildid 20210918213140 and continuing through yesterdays nightly build 20211003201113
https://crash-stats.mozilla.org/signature/?signature=OOM%20%7C%20large%20%7C%20mozalloc_abort%20%7C%20webrender%3A%3Arenderer%3A%3Aupload%3A%3Aupload_to_texture_cache&date=%3E%3D2021-09-27T04%3A10%3A00.000Z&date=%3C2021-10-04T04%3A10%3A00.000Z&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=address&_columns=install_time&_columns=startup_crash&_sort=-date&page=1#summary

Comment 4

[Nicolas Silva [:nical]]

I went through the commits in that period and I don't see suspicious changes in that area of the code so I suspect it's the OOM rate that increased and it consolidated here.

I'm working on some texture uploads changes that will hopefully decrease the need for allocating large chunks of memory here. It'll probably land in a couple of weeks.

Comment 5

[Jeff Muizelaar [:jrmuizel]]

Here's a crash report with a memory report:
https://crash-stats.mozilla.org/report/index/ecb273fb-0b37-48be-9f7d-6dab20211104#tab-bugzilla

Comment 6

[GerardoPcp04]

More crashes with Firefox 96.0.3
https://support.mozilla.org/es/questions/1366897
https://crash-stats.mozilla.com/report/index/bp-278c00d3-61cb-4560-8a34-ad6c10220204
https://crash-stats.mozilla.com/report/index/bp-5286e4cb-1a3f-40ff-b0d2-ad8530220204

Comment 7

[Wayne Mery (:wsmwk)]

Crashes pretty died out in mid-September.