1516011 - Google Slides: Crash in wr_moz2d_render_cb (ExternalSourceSurfaceCreation PLAY failure)

Status: RESOLVED FIXED

(Core :: Graphics: WebRender, defect, P3)

Description

[Panos Astithas (he/him) [:past] (please ni?)]

This bug was filed from the Socorro interface and is
report bp-200a75db-d14f-4eb6-80a2-8fe600181221.
=============================================================

Top 10 frames of crashing thread:

0 XUL wr_moz2d_render_cb gfx/webrender_bindings/Moz2DImageRenderer.cpp:435
1 XUL webrender_bindings::moz2d_renderer::rasterize_blob gfx/webrender_bindings/src/moz2d_renderer.rs:524
2 XUL <webrender_bindings::moz2d_renderer::Moz2dBlobRasterizer as webrender_api::image::AsyncBlobImageRasterizer>::rasterize libcore/ops/function.rs:156
3 XUL webrender::scene_builder::Transaction::rasterize_blobs gfx/wr/webrender/src/scene_builder.rs:96
4 XUL std::sys_common::backtrace::__rust_begin_short_backtrace gfx/wr/webrender/src/scene_builder.rs:779
5 XUL <F as alloc::boxed::FnBox<A>>::call_box libstd/thread/mod.rs:409
6 XUL std::sys::unix::thread::Thread::new::thread_start src/liballoc/boxed.rs:682
7 libsystem_pthread.dylib libsystem_pthread.dylib@0x3304 
8 libsystem_pthread.dylib libsystem_pthread.dylib@0x626e 
9 libsystem_pthread.dylib libsystem_pthread.dylib@0x2414 

=============================================================

Other similar crashes:
bp-6a18798d-aa31-414e-8697-0c54c0181219
bp-041299dc-349e-4998-9dd4-338c90181212

Comment 1

[Darkspirit]

> MOZ_RELEASE_ASSERT(false)

Comment 2

[Darkspirit]

> GraphicsCriticalError 	|[0][GFX1-]: Replay failure: ExternalSourceSurfaceCreation PLAY (t=14976.7)

Comment 3

[Jeff Muizelaar [:jrmuizel]]

The only crashes I see are on MacOS

https://crash-stats.mozilla.com/signature/?product=Firefox&signature=wr_moz2d_render_cb&date=%3E%3D2018-10-10T08%3A17%3A00.000Z&date=%3C2019-01-07T07%3A17%3A49.000Z&_columns=date&_columns=product&_columns=version&_columns=build_id&_columns=platform&_columns=reason&_columns=address&_columns=install_time&_sort=-date&page=1#summary

Why did you mark this bug as OS = All?

Comment 4

[Darkspirit]

Because of the other signature with the same GraphicsCriticalError.
bp-5a6a5539-b81f-4eb0-91ee-c99990190103
bp-c1c008f2-1477-4b53-aecf-bdc2e0190103

Comment 5

[Jeff Muizelaar [:jrmuizel]]

Thanks. It looks like this happens quite a bit on Google slides.

Comment 6

[Jeff Muizelaar [:jrmuizel]]

Here's an example url: https://docs.google.com/presentation/d/1xluJDxyqncGGn3He5QUvO27oBmZQLyeUuzWiaSnxM3U/edit#slide=id.g447595bf74_93_0

Comment 7

[Jeff Muizelaar [:jrmuizel]]

Some more urls:
https://www.r6counters.com/
https://www.yr.no/kart/#lat=65.00146&lon=19.54158&zoom=5&laga=vind&proj=3575
http://preview.ideditor.com/master/#background=Bing&disable_features=boundaries&id=w150664283&map=16.93/22.41255/114.22643&offset=1.06,0.82
http://www.kicker.de/news/fussball/bundesliga/spieltag/1-bundesliga/2018-19/17/4243414/livematch_eintracht-frankfurt-32_bayern-muenchen-14.html
https://coub.com/hot

Comment 8

[Jeff Muizelaar [:jrmuizel]]

I'm assigning to Andrew as this is his code. I expect this will be difficult to debug without a STR

Comment 9

[Andrew Osmond [:aosmond] (he/him)]

It is hard to filter the signatures on the gfx logs, but I suspect recycling caused this. Animated image inside a blob might have some extra edge cases.

Comment 10

[Andrew Osmond [:aosmond] (he/him)]

I can reproduce on giphy.com in a local build by forcing images to go through fallback. I need to scroll around a lot to trigger though.

Comment 11

[Andrew Osmond [:aosmond] (he/him)]

So if I fix this check:

https://searchfox.org/mozilla-central/rev/76fe4bb385348d3f45bbebcf69ba8c7283dfcec7/gfx/layers/wr/WebRenderCommandBuilder.cpp#186

to take into account recycled surfaces, it appears to fix the crashes. I'm a little worried about some state inconsistencies though, because recycling adds new races where the blob image might display the wrong data...

Comment 12

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1516011 - Part 1. Ensure wrapped shared surfaces are retained for blob rasterization.

References to shared surfaces are already kept alive for the blob in the
content process, and it also ensures an image key is created to ensure
any release of the surface is delayed until the next epoch. Wrapped
shared surfaces (when used in an animation which is recycling its
surfaces) did not get an image key created which this patch corrects.

Comment 13

[Andrew Osmond [:aosmond] (he/him)]

Attachment: Bug 1516011 - Part 2. Deny recycling for frames used in blob recordings.

Given the crash resolved in part 1, it is possible for the blob
rasterizer in the compositor process to still be using surfaces after
the animation has advanced to the next frame. With recycling this can be
problematic as the recycled surface will be reused for a future frame.
In an ideal world, the blob recording would use the animation's image
key instead, but the rasterizer doesn't have easy access to the mapping
table. As such, for any frames used in a blob recording, we now
explicitly mark them as non-recyclable and we will be forced to allocate
a new frame instead.

Depends on D16191

Comment 14

[Andrew Osmond [:aosmond] (he/him)]

try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=5c4c03d21e87de7900a2e276fd69d21b6b86f649

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/c74c81fe8dcb
Part 1. Ensure wrapped shared surfaces are retained for blob rasterization. r=jrmuizel
https://hg.mozilla.org/integration/mozilla-inbound/rev/3c5fcd8a2f4a
Part 2. Deny recycling for frames used in blob recordings. r=tnikkel

Comment 16

[Dorel Luca [:dluca]]

Backed out 2 changesets (bug 1516011) for build bustge in /build/build/src/gfx/layers/wr/WebRenderCommandBuilder.cpp

Log:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=221383927&repo=mozilla-inbound&lineNumber=19480

Push with failures:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=3c5fcd8a2f4a860769a82132eb0ca97c577a441d

Backout:
https://hg.mozilla.org/integration/mozilla-inbound/rev/6b946ad790d09f81ed5c37db0bea635275826fcc

Comment 17

[Andrew Osmond [:aosmond] (he/him)]

Hopefully fixed the build issue:

try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=114557f5b60c3c911d78d97626412c5c26bcab5e

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/7ed5d7408250
Part 1. Ensure wrapped shared surfaces are retained for blob rasterization. r=jrmuizel
https://hg.mozilla.org/integration/mozilla-inbound/rev/c5f982e02892
Part 2. Deny recycling for frames used in blob recordings. r=tnikkel

Comment 19

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/7ed5d7408250
https://hg.mozilla.org/mozilla-central/rev/c5f982e02892

Comment 20

[Andrew Osmond [:aosmond] (he/him)]

I see https://crash-stats.mozilla.com/report/index/9f210bca-5925-4eb8-8e08-8fdd50190115#tab-details which includes my fix. I am now puzzled as to the root cause.