1442921 - Crash in mozalloc_abort | abort | webrender::renderer::Renderer::draw_tile_frame

Status: RESOLVED FIXED

(Core :: Graphics: WebRender, defect, P1)

Description

[Darkspirit]

bp-dd89a5ef-6190-4988-8fc9-3fec40180303
> assertion failed: `(left == right)`
> |[G0][GFX1-]: WR: Attempting to allocate a texture of size 22140x2048 above the limit, trimming (t=432.375)

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I see the same and it results in a perma-crash for me each time when I want to sign into Treeherder via Auth0:

Crash report: bp-dc4ac6da-e209-46d4-99d8-f44b70180305.

Top 10 frames of crashing thread:
0 libmozglue.dylib mozalloc_abort memory/mozalloc/mozalloc_abort.cpp:33
1 libmozglue.dylib abort memory/mozalloc/mozalloc_abort.cpp:80
2 XUL std::panicking::rust_panic src/libpanic_abort/lib.rs:59
3 XUL std::panicking::rust_panic_with_hook::h6e16f84ca8c4a724 src/libstd/panicking.rs:593
4 XUL std::panicking::begin_panic::h950ea52f07a191e2 src/libstd/panicking.rs:538
5 XUL std::panicking::begin_panic_fmt::he83be180ce68059a src/libstd/panicking.rs:522
6 XUL webrender::renderer::Renderer::draw_tile_frame::h34631d473819e941 src/libcore/fmt/mod.rs
7 XUL webrender::renderer::Renderer::render_impl::h0ed5dd32021869b8 gfx/webrender/src/renderer.rs:2872
8 XUL webrender_bindings::bindings::wr_renderer_render gfx/webrender/src/renderer.rs:2771
9 XUL mozilla::wr::RendererOGL::UpdateAndRender gfx/webrender_bindings/RendererOGL.cpp:136

Here the steps to reproduce the crash:

1) Open https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound
2) Click Login/Register at the top right
3) Select Auth0 (in that case my mozilla.com address)

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Hm, now after upgrading the Nightly to Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:60.0) Gecko/20100101 Firefox/60.0 ID:20180304220118 everything seems to work fine. Was there a webrender push in the meantime which fixed that crash?

Comment 3

[Kartikaya Gupta (email:[e-mail])]

(In reply to Henrik Skupin (:whimboo) from comment #2)
> Hm, now after upgrading the Nightly to Mozilla/5.0 (Macintosh; Intel Mac OS
> X 10.13; rv:60.0) Gecko/20100101 Firefox/60.0 ID:20180304220118 everything
> seems to work fine. Was there a webrender push in the meantime which fixed
> that crash?

There was a WR update that landed on March 2nd, bug 1440664. And then there was an out-of-band fix in bug 1442608 which landed on March 3rd. It's possible the latter might have fixed this, particularly if the STR involved the first paint of a new window (which I think the Auth0 login case does, but not sure).

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

It's actually not fixed. I hit it again with Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:60.0) Gecko/20100101 Firefox/60.0 ID:20180304220118.

Crash report: bp-54a6b7d3-b5c9-49ef-bf24-d9c700180305

Comment 5

[Taegeon Lee]

while using Firefox nightly (20180306100123) which webrender enabled, the rendering process is changed to 'Basic' with this crash.

Crash report: bp-3a2b544a-54ac-47f6-8a76-4c5840180306

Comment 6

[Andrew McCreight [:mccr8]]

#6 top crash for the March 7 Windows Nightly.

Comment 7

[Glenn Watson [:gw]]

I'm currently unable to reproduce this (on Linux). I was able to log in to Treeherder without problems - although, I do use Duo security for a 2FA, which may be different than in the original report?

I notice above it was mentioned that it's fixed for some users - are the steps to repro still the same as the original report?

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I have seen it when using containers and accidentally tried to login to Treeherder in a container which was never used with auth0 before. Glenn, did you try with a fresh profile? Maybe that gives better results.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Happened again with latest Nightly build: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:61.0) Gecko/20100101 Firefox/61.0 ID:20180312220041

Comment 10

[Glenn Watson [:gw]]

I tried with a fresh profile, and still can't reproduce this yet.

Comment 11

[Glenn Watson [:gw]]

Are there any other known sites that reproduce this issue other than treeherder?

Comment 12

[Sotaro Ikeda [:sotaro]]

Bug 1441392's crash stack has Renderer::draw_tile_frame(). It could be reproduced on my linux pc.

Comment 13

[Sotaro Ikeda [:sotaro]]

(In reply to Sotaro Ikeda [:sotaro] from comment #12)
> Bug 1441392's crash stack has Renderer::draw_tile_frame(). It could be
> reproduced on my linux pc.

I meant attachment 8954253 [details] in Bug 1441392 caused crash by hitting assertion. And the crash stack had Renderer::draw_tile_frame().

Comment 14

[Glenn Watson [:gw]]

That bug you linked to does reproduce on my machine, thanks Sotaro!

It would be somewhat surprising if the real treeherder page above is trying to open a window / popup that is larger than the maximum texture size, but perhaps there is an initial very large window size supplied by Gecko?

Anyway, I'll add a workaround in WR to handle the linked bug, and we'll see if that also resolves this bug.

Comment 15

[Ludovic Hirlimann [:Usul]]

(In reply to Glenn Watson [:gw] from comment #11)
> Are there any other known sites that reproduce this issue other than
> treeherder?

I can repro from soccoro's web interface when you try to file a bug in say CORE from soccoro. I fails with this stack https://crash-stats.mozilla.com/report/index/ffe5c6ba-f9d5-4ef9-ac49-3c3d80180315#tab-bugzilla and

[ludo@Oulanl ~]$ bin/firefox/firefox
WebRender - OpenGL version new 4.5 (Core Profile) Mesa 17.3.6
WebRender - OpenGL version new 4.5 (Core Profile) Mesa 17.3.6
ERROR:<unknown>: Attempting to allocate a texture of size 18824x2048 above the limit, trimming
thread '<unnamed>' panicked at 'assertion failed: `(left == right)`
  left: `16384×2048`,
 right: `18824×2048`', gfx/webrender/src/tiling.rs:222:9
stack backtrace:
   0:     0x7f55f515c263 - std::sys::unix::backtrace::tracing::imp::unwind_backtrace::hb98fbe643b37b8bb
   1:     0x7f55f5156c0c - std::panicking::default_hook::{{closure}}::h83c090f00cd2917d
   2:     0x7f55f515659d - std::panicking::default_hook::hf9722061a353cd29
   3:     0x7f55f515615c - std::panicking::rust_panic_with_hook::h574be4fada9826dc
   4:     0x7f55f5158045 - std::panicking::begin_panic::h87949d2c83be3718
   5:     0x7f55f5157fd9 - std::panicking::begin_panic_fmt::h351d9e48d73bd160
   6:     0x7f55f512a509 - webrender::renderer::Renderer::draw_tile_frame::hb68db5e58463a1ef
   7:     0x7f55f511a392 - webrender::renderer::Renderer::render_impl::h6b04bb72ad93d195
   8:     0x7f55f4bfe01d - wr_renderer_render
   9:     0x7f55f234dab9 - _ZN7mozilla2wr11RendererOGL15UpdateAndRenderEb
  10:     0x7f55f234db77 - _ZN7mozilla2wr12RenderThread15UpdateAndRenderENS0_10WrWindowIdEb
  11:     0x7f55f234ede7 - _ZN7mozilla2wr12RenderThread13NewFrameReadyENS0_10WrWindowIdE
  12:     0x7f55f234cee6 - _ZN7mozilla6detail18RunnableMethodImplIPNS_2wr12RenderThreadEMS3_FvNS2_10WrWindowIdEELb1ELNS_12RunnableKindE0EJS5_EE3RunEv
  13:     0x7f55f3ee7824 - _ZN4base18MessagePumpDefault3RunEPNS_11MessagePump8DelegateE
  14:     0x7f55f4695bf6 - _ZN11MessageLoop3RunEv
  15:     0x7f55f46978ba - _ZN4base6Thread10ThreadMainEv
  16:     0x7f55f4695a19 - _ZL10ThreadFuncPv
  17:     0x7f560267761a - start_thread
  18:     0x7f56016a5c2e - __GI___clone
  19:                0x0 - <unknown>
Redirecting call to abort() to mozalloc_abort

ExceptionHandler::GenerateDump cloned child ExceptionHandler::WaitForContinueSignal waiting for continue signal...
3429
ExceptionHandler::SendContinueSignalToChild sent continue signal to child
[ludo@Oulanl ~]$ Failed to open curl lib from binary, use libcurl.so instead

Comment 16

[Darkspirit]

Debian Testing (KDE, Radeon RX480, 2560x1440)

Instant crash.
mozregression --launch 7faa8401a434 --pref gfx.webrender.all:true startup.homepage_welcome_url:"https://bugzilla.mozilla.org/attachment.cgi?id=8933009" -B debug
> 0:46.65 INFO: [GFX1-]: WR: Attempting to allocate a texture of size 202x50004 above the limit, trimming
> 0:46.66 INFO: [GFX1-]: WR: Attempting to allocate a texture of size 2048x50004 above the limit, trimming
> 0:46.66 INFO: thread '<unnamed>' panicked at 'assertion failed: `(left == right)`
> 0:46.66 INFO:   left: `2048×16384`,
> 0:46.66 INFO:  right: `2048×50004`', gfx/webrender/src/tiling.rs:223:9

-----

mozregression --find-fix --bad 20180308100121 --good 2018-03-14 --pref gfx.webrender.all:true startup.homepage_welcome_url:"https://bugzilla.mozilla.org/attachment.cgi?id=8933009" -B debug
> 9:52.11 INFO: First good revision: 8a988d1a6f23f900b90c7843e2c61db14a57e56c
> 9:52.11 INFO: Last bad revision: 5110a382f6a073a9fa220936442125c961bc57af
> 9:52.11 INFO: Pushlog:
> https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=5110a382f6a073a9fa220936442125c961bc57af&tochange=8a988d1a6f23f900b90c7843e2c61db14a57e56c

> 8a988d1a6f23	Kartikaya Gupta — Bug 1443807 - Update reftest annotations for changes in WR PR 2487. r=jrmuizel
> 1bc8b8422379	Kartikaya Gupta — Bug 1443807 - Update webrender to commit 5cb71f0f23719795e7c89417d91a7abad8ac20e9. r=jrmuizel

> WR @ f3af2632723f62eb7973a87bfaece43a675cefb5
mozregression --repo try --launch c2415020c1acc328f0dab4b0f72fafae312ea996 --pref gfx.webrender.all:true startup.homepage_welcome_url:"https://bugzilla.mozilla.org/attachment.cgi?id=8933009" -B debug
good

> WR @ 7e459e819cbfb6ed81a86bf94a5384af472583e0
mozregression --repo try --launch 4ce5c83364d2a96c74a233e2241c65373e741ece --pref gfx.webrender.all:true startup.homepage_welcome_url:"https://bugzilla.mozilla.org/attachment.cgi?id=8933009" -B debug
good

Fix range: https://github.com/servo/webrender/compare/0da6c839b3a0e165f1115fb9fe286be7540c24ed...f3af2632723f62eb7973a87bfaece43a675cefb5
= servo/webrender#2487 (Fixed the crash at least for me.)

That might explain why crash numbers went down. So it's fixed for me, but not for everyone. 

> https://hg.mozilla.org/mozilla-central/rev/1bc8b8422379
"first release with" bug 1443807: 2018-03-10_100108

Last crashes so far:
Linux: 2018-03-15_223216
Mac:   2018-03-12_220041
Win:   2018-03-10_220052

Comment 17

[Darkspirit]

 servo/webrender#2524 is included in bug 1444946.

Comment 18

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Jan, I assume you wanted to set fixed and not disabled for both 60 and 61?

Comment 19

[Darkspirit]

 servo/webrender#2524 says "potentially", so we could wait some days to be sure.

Comment 20

[Thomas Daede [:TD-Linux]]

Another site that crashes for me is https://www.bloomberg.com/graphics/2018-tesla-tracker/

I think the update should have made it into the most recent nightly, but I'll test again tomorrow.

Comment 21

[Darkspirit]

(In reply to Thomas Daede [:TD-Linux] from comment #20)
> https://www.bloomberg.com/graphics/2018-tesla-tracker/

Nightly 61 x64 20180317111042 de_DE 97160a734959af73cc97af0bf8d198e301ebedae @ Debian Testing (KDE, Radeon RX480, 2560x1440)
main profile: gpu process, webrender, omtp, etc.

My build contains bug 1444946. I got a fallback to OpenGL compositing just by zooming in on that website.
bp-8d387fad-f6d7-4e0f-85e4-ce3bf0180317 17.03.18 23:29
bp-db23ea3a-8df7-43a7-9d15-097240180317 17.03.18 23:29

Comment 22

[Jan Alexander Steffens [:heftig]]

I believe I'm still affected by this as well.

In a fresh profile with gfx.webrender.all, opening this website causes a crash:
https://www.fimfiction.net/story/395229/the-nameless-queen

Nightly 61 x64 20180318002633 en_US efce78e62b6dac195e7cb4898684da54155d1661
Arch Linux (GNOME, Geforce 980M, 3840x2160, UI scale 2)

Built from https://hg.mozilla.org/mozilla-unified/rev/efce78e62b6dac195e7cb4898684da54155d1661

Report: bp-a0156d33-a27d-4707-ac2d-41be10180319

Log:

    WebRender - OpenGL version new 3.2.0 NVIDIA 390.42
    ERROR:<unknown>: Attempting to allocate a texture of size 2048x21435 above the limit, trimming
    thread '<unnamed>' panicked at 'assertion failed: `(left == right)`
      left: `2048×16384`,
     right: `2048×21435`', gfx/webrender/src/tiling.rs:222:9

Comment 23

[Jan Alexander Steffens [:heftig]]

On my other machine I can also reproduce it, but only when launching Firefox with GDK_SCALE=2.

Nightly 61 x64 20180318002633 en_US efce78e62b6dac195e7cb4898684da54155d1661
Arch Linux (GNOME, Intel Haswell GT3e, 1920x1080)

Report: bp-d3aaf803-19db-48a3-b8b4-130650180319

Log:
    WebRender - OpenGL version new 4.5 (Core Profile) Mesa 17.3.6
    ERROR:<unknown>: Attempting to allocate a texture of size 2048x33664 above the limit, trimming
    thread '<unnamed>' panicked at 'assertion failed: `(left == right)`
      left: `2048×16384`,
     right: `2048×33664`', gfx/webrender/src/tiling.rs:222:9

Comment 24

[Chris Peterson [:cpeterson]]

I hit this crash every time I log into the Magic Leap SDK website with Nightly 61 (2018-03-19) on macOS 10.13.3.

STR:
1. Load https://creator.magicleap.com/home
2. Register an email address.
3. Click through the EULA pages.

RESULT:
Crash with assertion failed: `(left == right)`

bp-cbc911c4-28f5-41e3-b1cb-076060180319

Comment 25

[Jan Alexander Steffens [:heftig]]

The issue is that a box shadow that is too thin (400x25473) for the minimal shadow rect (blur 60 + corners 200 = 580x580) BoxShadowStretchMode::Stretch, so it falls back to Simple mode (clip.rs:188), which needs to draw the entire height.

Comment 26

[Glenn Watson [:gw]]

https://github.com/servo/webrender/pull/2563 has landed in WR now, so will hopefully be included in the next WR update in Gecko.

Comment 27

[Jan Alexander Steffens [:heftig]]

Crashes are fixed for me.

Comment 28

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Thanks! I also cannot reproduce the crash with auth0 anymore.