1493616 - blob invalidation: Crash in mozilla::layers::DIGroup::PaintItemRange

Status: VERIFIED FIXED

(Core :: Graphics: WebRender, defect, P1)

Description

[Calixte Denizet (:calixte)]

This bug was filed from the Socorro interface and is
report bp-a1e2e77f-a0f7-4b0b-9bb1-6ad320180920.
=============================================================

Top 10 frames of crashing thread:

0 xul.dll void mozilla::layers::DIGroup::PaintItemRange gfx/layers/wr/WebRenderCommandBuilder.cpp:763
1 xul.dll void mozilla::layers::DIGroup::EndGroup gfx/layers/wr/WebRenderCommandBuilder.cpp:674
2 xul.dll void mozilla::layers::Grouper::ConstructGroups gfx/layers/wr/WebRenderCommandBuilder.cpp:1020
3 xul.dll void mozilla::layers::WebRenderCommandBuilder::DoGroupingForDisplayList gfx/layers/wr/WebRenderCommandBuilder.cpp:1224
4 xul.dll nsDisplaySVGWrapper::CreateWebRenderCommands layout/painting/nsDisplayList.cpp:10352
5 xul.dll mozilla::layers::WebRenderCommandBuilder::CreateWebRenderCommandsFromDisplayList gfx/layers/wr/WebRenderCommandBuilder.cpp:1412
6 xul.dll nsDisplayTransform::CreateWebRenderCommands layout/painting/nsDisplayList.cpp:8771
7 xul.dll mozilla::layers::WebRenderCommandBuilder::CreateWebRenderCommandsFromDisplayList gfx/layers/wr/WebRenderCommandBuilder.cpp:1412
8 xul.dll void mozilla::layers::WebRenderCommandBuilder::BuildWebRenderCommands gfx/layers/wr/WebRenderCommandBuilder.cpp:1277
9 xul.dll mozilla::layers::WebRenderLayerManager::EndTransactionWithoutLayer gfx/layers/wr/WebRenderLayerManager.cpp:291

=============================================================

There are 60 crashes (from 52 installations) in nightly 64 starting with buildid 20180920100522.
:darkspirit, could you investigate please ?

Comment 1

[Darkspirit]

> MOZ_RELEASE_ASSERT(!data->mInvalid)

Changes between 20180919220108 and 20180920100522:
https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=9812141ec782&tochange=08592337ced1

Comment 2

[Darkspirit]

Changes between 2018-09-18 and 2018-09-22:
https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=85b4d2bf888afa32b67638602a3338d0a6935ff9&tochange=221c18ebe962f68358b4cba927df9099ea935b40

Ctrl+F Jeff Muizelaar. Suspicious sounding titles:
20180922100157 Bug 1491590. Make sure that the paint rect is set to the bounds. r=mstange
20180919123806 Bug 1491395. Replace BorderWidths with LayoutSideOffsets.

Comment 3

[Jeff Muizelaar [:jrmuizel]]

Bug 1447880 is the more likely cause.

Comment 4

[Darkspirit]

Could you use your superpowers and look if some crash reports contain an URL?

Comment 5

[Darkspirit]

"it's mainly pornhub"

We'll need to find STR on:
https://www.pornhub.com/view_video.php?viewkey=323852368
https://www.fanatical.com/en/bundle

Comment 6

[Darkspirit]

Bug 1396642 caused a behavior change in WR bug 1494934.
> dc6c04a63309	Robert Longson — Bug 1396642 - support smaller viewBox coordinates at the expense of larger ones r=dholbert
It lies within the possible regression range.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/c97cf45fc0e4
Add some logging to help debug this crash.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/742336dae392
Fix up logging to only trigger appropriately.

Comment 9

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/c97cf45fc0e4
https://hg.mozilla.org/mozilla-central/rev/742336dae392

Comment 10

[Jeff Muizelaar [:jrmuizel]]

The crashing item seems to be a nsDisplayTransform

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/a91bad559883
blob-inval: Log whether we have a 3d transform

Comment 12

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/a91bad559883

Comment 13

[Jeff Muizelaar [:jrmuizel]]

https://multimedia.scmp.com/news/world/article/2165980/flight-paths/index.html is a crashing URL and it seems most likely to be reproducible of the ones I've seen. 

Note: this may have been fixed by bug 1496188 so builds after 20181004100222 may not reproduce anymore.

Comment 14

[Darkspirit]

https://addons.mozilla.org/firefox/addon/bugzilla-crash-stop
Last crash with build 20181003100127 so far. No crashes with 20181003223518 and 20181004100222.

Changes between 20181003100127 and 20181003223518:
https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=5675805eb31d&tochange=317b91c1fbd5

Comment 15

[Jeff Muizelaar [:jrmuizel]]

And here are crashes with build 20181004100222 https://crash-stats.mozilla.com/search/?build_id=20181004100222&release_channel=nightly&signature=%3Dmozilla%3A%3Alayers%3A%3ADIGroup%3A%3APaintItemRange&product=Firefox&version=64.0a1&date=%3E%3D2018-09-26T20%3A00%3A00.000Z&date=%3C2018-10-05T00%3A05%3A43.000Z&_sort=-date&_facets=url&_facets=user_comments&_facets=install_time&_facets=version&_facets=address&_facets=moz_crash_reason&_facets=reason&_facets=build_id&_facets=platform_pretty_version&_facets=signature&_facets=useragent_locale&_columns=date&_columns=signature&_columns=product&_columns=version&_columns=build_id&_columns=platform#crash-reports

Comment 16

[Jeff Muizelaar [:jrmuizel]]

That search might get time zone confused. Here's an actual crash: https://crash-stats.mozilla.com/report/index/43d89dad-7869-4428-a751-332540181004

Comment 17

[Jeff Muizelaar [:jrmuizel]]

So looking at the call stack we see nsDisplayMasksAndClipPaths::PaintWithContentsPaintCallback(nsDisplayListBuilder*, gfxContext*, std::function<void > const&) so bug 1496188 exposed this.

Comment 18

[Jeff Muizelaar [:jrmuizel]]

Trying out a possible theory: https://treeherder.mozilla.org/#/jobs?repo=try&revision=356994d14b8827cef8fbdbb84c1fdd50cfa42c01

Comment 19

[Jeff Muizelaar [:jrmuizel]]

Attachment: Bug 1493616. Clear mInvalid more agressively so that we don't reuse it across paints

Comment 20

[Jeff Muizelaar [:jrmuizel]]

And a try for that change: https://treeherder.mozilla.org/#/jobs?repo=try&revision=0dbc9d2b5c360e9cf33d1f316a735502d05e9b71

Comment 21

[Jeff Muizelaar [:jrmuizel]]

Try better: https://treeherder.mozilla.org/#/jobs?repo=try&revision=7a13fa509085e079e46f4c0f4758e344d5c41544

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f936a4baa698
Clear mInvalid more agressively so that we don't reuse it across paints r=mstange

Comment 23

[Andrei Ciure[:aciure]]

Backed out changeset f936a4baa698 (bug 1493616) for build bustages WebRenderCommandBuilder.cpp

push that caused the backout: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=testfailed,busted,exception,retry,usercancel,runnable&selectedJob=203786255&revision=f936a4baa698ac02c3ae215f61dee51c595cb37b

failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception&classifiedState=unclassified&fromchange=b26a70a0fe8f22ee4a5118c7c563ab98a115e692&selectedJob=203786257&searchStr=linux%2Cx64%2Cdebug%2Cbuild-linux64%2Fdebug%2C%28b%29

backout: https://hg.mozilla.org/integration/autoland/rev/d4cd818673005e926f505b82eccb89b0b46045f7

Comment 24

[Jeff Muizelaar [:jrmuizel]]

Forgot to update the patch. Sorry.

Comment 25

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/497bb152e567
Clear mInvalid more agressively so that we don't reuse it across paints r=mstange

Comment 26

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/497bb152e567

Comment 27

[Jeff Muizelaar [:jrmuizel]]

Still crashing https://crash-stats.mozilla.com/report/index/2a3b153d-c3a8-4688-9d17-54c0b0181006

Comment 28

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/ce95adbcf2e4
Add some data gathering to try to figure out the crash.

Comment 29

[Stefan Hindli [:stefan_hindli]]

https://hg.mozilla.org/mozilla-central/rev/ce95adbcf2e4

Comment 30

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/b3158d143d83
blob-inval: Log the some more data about the broken situation.

Comment 31

[Dorel Luca [:dluca]]

https://hg.mozilla.org/mozilla-central/rev/b3158d143d83

Comment 32

[Jeff Muizelaar [:jrmuizel]]

It looks like this is probably a rounding or off by one issue: "CGC--0-0-281-31,0-0-280-31-ib"

Comment 33

[Jeff Muizelaar [:jrmuizel]]

Attachment: Bug 1493616. Make sure we invalidate the entire area of the item

Comment 34

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6a8e29086733
Make sure we invalidate the entire area of the item r=mstange

Comment 35

[Daniel Varga [:dvarga]]

https://hg.mozilla.org/mozilla-central/rev/6a8e29086733

Comment 36

[Maire Reavy [:mreavy]]

Jeff -- I believe it's now safe to mark this resolved.  Just needinfo'ing you for a sanity-check.

Comment 37

[Jeff Muizelaar [:jrmuizel]]

Yes. Probably. We should double check that the crashes go away, but they should.

Comment 38

[Ryan VanderMeulen [:RyanVM]]

No crashes on Nightly since this landed.