717658 - Crash with abort message "###!!! ABORT: creating ThebesLayer 'back buffer' failed! type=3000"

Status: RESOLVED WORKSFORME

(Core :: Graphics, defect)

Description

[Scoobidiver (away)]

It's #4 top crasher in Fennec 10.0b3.
It's probably the new form of bug 711706 for Fennec 10.
App notes let think it's a graphics bug.

Signature 	mozalloc_abort | __swrite | libxul.so@0xead678 More Reports Search
UUID	e49ccbed-097a-4b5c-8273-478b72120111
Date Processed	2012-01-11 14:01:06.729007
Process Type	content
Uptime	117
Install Age	4.8 minutes since version was first installed.
Install Time	2012-01-11 21:56:32
Product	Fennec
Version	10.0
Build ID	20120104111157
Release Channel	beta
OS	Linux
OS Version	0.0.0 Linux 2.6.35.13-g84f8edd #1 SMP PREEMPT Fri Aug 5 01:14:20 CST 2011 armv7l
Build Architecture	arm
Build Architecture Info	
Crash Reason	SIGSEGV
Crash Address	0x0
App Notes 	
xpcom_runtime_abort(###!!! ABORT: creating ThebesLayer 'back buffer' failed!: file /builds/slave/rel-m-beta-lnx-andrd-bld/build/gfx/layers/basic/BasicLayers.cpp, line 2383)	
EMCheckCompatibility	True

Frame 	Module 	Signature [Expand] 	Source
0 	libmozalloc.so 	mozalloc_abort 	memory/mozalloc/mozalloc_abort.cpp:66
1 	libc.so 	__swrite 	
2 	libxul.so 	libxul.so@0xead678 	
3 	libxul.so 	nsHTMLEditRules::MoveNodeSmart 	editor/libeditor/html/nsHTMLEditRules.cpp:2869
4 	libxul.so 	nsBlockFrame::GetBaseline 	layout/generic/nsBlockFrame.cpp:550

Comment 1

[Scoobidiver (away)]

With combined signatures, it's #6 top crasher in 10.0b4.

Comment 2

[Benoit Jacob [:bjacob] (mostly away)]

This is an out-of-memory error.

I agree we should trust the assertion message from the AppNotes, more than the stacks.

What this strongly suggests is that some code around BasicLayers/ThebesLayer needs to improve its handling of OOM conditions.

This assertion message points to this line:
http://hg.mozilla.org/mozilla-central/file/79e5d0b77d10/gfx/layers/basic/BasicLayers.cpp#l2402

  2398   // XXX error handling
  2399   if (!BasicManager()->AllocBuffer(gfxIntSize(aSize.width, aSize.height),
  2400                                    aType,
  2401                                    &mBackBuffer)) {
* 2402       NS_RUNTIMEABORT("creating ThebesLayer 'back buffer' failed!");
  2403   }

Comment 3

[Benoit Jacob [:bjacob] (mostly away)]

CC'ing romaxa for http://hg.mozilla.org/mozilla-central/rev/b083bc8b79ab

Comment 4

[Benoit Jacob [:bjacob] (mostly away)]

OTOH, it doesn't seem easy/possible to recover from this allocation failure, so maybe I'm wrong and the real problem is with what causes us to get there.

Maybe a good first step would be to print the AllocBuffer parameters as part of this assert message, so we see if there's an obvious problem with them (e.g. absurdly large size).

Comment 5

[Benoit Jacob [:bjacob] (mostly away)]

Attachment: dump sizes and contenttype

This will get us more information in those assert messages.

Comment 6

[Scoobidiver (away)]

(In reply to Benoit Jacob [:bjacob] from comment #5)
> This will get us more information in those assert messages.
First feedback in two months when 12.0 becomes Beta. I think it should land in Aurora and even Beta if the code is not shared with desktop.

Comment 7

[Benoit Jacob [:bjacob] (mostly away)]

http://hg.mozilla.org/integration/mozilla-inbound/rev/bdc686b071bb

So hopefully we'll start getting more informative crash reports in 2 days.

Comment 8

[Scoobidiver (away)]

(In reply to Benoit Jacob [:bjacob] from comment #7)
> So hopefully we'll start getting more informative crash reports in 2 days.
There are about 500-1000 Aurora and Nightly testers on Fennec which is not enough to make this crash appear in these versions. So the first results will be in 12.0b1.

Comment 9

[Benoit Jacob [:bjacob] (mostly away)]

Comment on attachment 589563 [details] [diff] [review]
dump sizes and contenttype

(In reply to Scoobidiver from comment #8)
> (In reply to Benoit Jacob [:bjacob] from comment #7)
> > So hopefully we'll start getting more informative crash reports in 2 days.
> There are about 500-1000 Aurora and Nightly testers on Fennec which is not
> enough to make this crash appear in these versions. So the first results
> will be in 12.0b1.

In that case, let's land it on aurora now so that it will be uplifted into 11.0b1, so we win 6 weeks.

Comment 10

[:Ms2ger (he/him; ⌚ UTC+1/+2)]

https://hg.mozilla.org/mozilla-central/rev/bdc686b071bb

Comment 11

[Alex Keybl [:akeybl]]

Comment on attachment 589563 [details] [diff] [review]
dump sizes and contenttype

[Triage Comment]
Extra debug info in support of a top crasher - approved for Aurora.

Comment 12

[Benoit Jacob [:bjacob] (mostly away)]

http://hg.mozilla.org/releases/mozilla-aurora/rev/b5ed79ddf350

Comment 13

[Scoobidiver (away)]

Here are the first crashes in Fennec 11.0b3 with the extra debug info:
https://crash-stats.mozilla.com/report/list?product=Fennec&signature=mozalloc_abort%20|%20__swrite%20|%20libxul.so%400xf7a311%20|%20mozilla%3A%3Adom%3A%3AindexedDB%3A%3AIDBObjectStore%3A%3AAppendIndexUpdateInfo

Type is always set to 3000.

Comment 14

[Benoit Jacob [:bjacob] (mostly away)]

Sorry I forgot a bit about this one. The good news is that the extra debug info is very conclusive:

for example in:
https://crash-stats.mozilla.com/report/index/5043a48c-2296-4006-8eb3-98d292120224

We get:
###!!! ABORT: creating ThebesLayer 'back buffer' failed! width=1092240, height=546120, type=3000: file /builds/slave/rel-m-beta-andrd-xul-bld/build/gfx/layers/basic/BasicLayers.cpp, line 2389

This width and height are hugem and explain the OOM condition. The next question is why do we have these huge dimensions here.

Comment 15

[Benoit Jacob [:bjacob] (mostly away)]

Notice that in the above exampe, we have width == 2*height.

Comment 16

[Benoit Jacob [:bjacob] (mostly away)]

The next thing we notice is that each crash report has different huge values here. This might be an uninitialized value, but that's not very clear: these values seem to be concentrated around the 1e+6 order of magnitude, which is uncharacteristic of uninitialized values.

Also, the 'rule' that width == 2*height seems to always hold, within +/- 1 precision on height.

Comment 17

[Benoit Jacob [:bjacob] (mostly away)]

Here is the data for the 3 days from Feb. 23 to Feb. 25. That's 30 crashes in 3 days so 10/day.

The 'rule' about width == 2*height is not always true. Sometimes it's height==2*width and sometimes it's neither. Sometimes, only one of the two dimensions is huge.

 width=1600, height=40218, type=3000
 width=1600, height=40218, type=3000
 width=450116, height=225058, type=3000
 width=48150, height=32101, type=3000
 width=48150, height=32101, type=3000
 width=1246128, height=1869192, type=3000
 width=1041525, height=2603811, type=3000
 width=1155584, height=770390, type=3000
 width=470, height=56416, type=3000
 width=470, height=56416, type=3000
 width=1004, height=35322, type=3000
 width=1004, height=35321, type=3000
 width=1004, height=35475, type=3000
 width=1027186, height=2567963, type=3000
 width=519, height=34436, type=3000
 width=182532, height=60845, type=3000
 width=182532, height=60845, type=3000
 width=1420, height=38674, type=3000
 width=555, height=33554, type=3000
 width=1420, height=38674, type=3000
 width=2054371, height=1027186, type=3000
 width=1092240, height=546120, type=3000
 width=3617588, height=1808795, type=3000
 width=468, height=36617, type=3000
 width=232, height=139023, type=3000
 width=1239, height=37521, type=3000
 width=20583, height=13722, type=3000
 width=29848, height=59694, type=3000
 width=20445, height=13630, type=3000
 width=25852, height=51703, type=3000

Comment 18

[Benoit Jacob [:bjacob] (mostly away)]

To 1% accuracy, the width/height ratio is always very close to a simple fraction:

1/25
1/25
2/1
3/2
3/2
2/3
2/5
3/2
1/120
1/120
1/35
1/35
1/35
2/5
3/200
3/1
3/1
1/27
1/60
1/27
2/1
2/1
2/1
1/78
1/600
1/30
3/2
1/2
3/2
1/2

Comment 19

[Benoit Jacob [:bjacob] (mostly away)]

Someone who knows ThebesBasicLayers should have a look at that. The stacks point to

  http://hg.mozilla.org/releases/mozilla-beta/annotate/2427757522ff/gfx/layers/basic/BasicLayers.cpp#l733

calling

  http://hg.mozilla.org/releases/mozilla-beta/annotate/2427757522ff/gfx/layers/ThebesLayerBuffer.cpp#l273

but that doesn't really tell us where the weird sizes come from. They seem to be read from the mBufferRect member, but I don't know where that was set.

Comment 20

[Benoit Jacob [:bjacob] (mostly away)]

CC'ing people recommended by hg annotate.

Comment 21

[Benoit Jacob [:bjacob] (mostly away)]

(In reply to Benoit Jacob [:bjacob] from comment #18)

The fractions in comment 18 are further non-random in that they are made of small prime numbers only: mostly 2,3,5, one occurence 7 and one occurence of 13.

25  = 5*5
120 = 2*2*2*3*5
35  = 5*7
200 = 2*2*2*5*5
27  = 3*3*3
60  = 2*2*3*5
78  = 2*3*13
600 = 2*2*2*3*5*5
30  = 2*3*5

Comment 22

[Robert O'Callahan (:roc) (email my personal email if necessary)]

I wonder if the front-end code is setting the displayport to a very large value?

Comment 24

[Benoit Jacob [:bjacob] (mostly away)]

People of the Layout work week, here is a very useful bug to look at while you're in the same room!

Comment 25

[Martijn Wargers (dead)]

Fennec crashes consistently on this url for me: https://developer.mozilla.org/nl/demos/detail/css3-piano/launch
And I've seen it with other css3 demo-sites too.

It crashes on the Samsung Galaxy Nexus, Android 4.0.2 and the HTC Desire HD, Android 2.3.5.

Comment 26

[Martijn Wargers (dead)]

Also crashing here: file:///C:/Users/mw22/Desktop/tests/performance/overflowscrollscrolling.htm
After tapping on the link.

Comment 27

[Benoit Jacob [:bjacob] (mostly away)]

Bingo, I can reproduce with the link in comment 25 on a Nexus S, Android 2.3.

Comment 28

[Benoit Jacob [:bjacob] (mostly away)]

Attachment: Call stack and last messages before crash

Here's what I got by reproducing in moz-gdb.

The last messages before the crash say:

adb| State - 17760257
adb| Got a document start
adb| leaving void mozilla::AndroidBridge::HandleGeckoMessage(const nsAString_...
adb| ###!!! ASSERTION: Wrong bounds: 'bounds.IsEqualInterior(aChildren.GetBou...
adb| ###!!! ASSERTION: Residual translation out of range: '-0.5 <= mResidualT...
adb| ShmemAndroid::ShareToProcess(): Too many open files (24)
adb| ###!!! ABORT: creating ThebesLayer 'back buffer' failed! width=44, heigh...

Don't the "wrong bounds" and "out of range" messages sound interesting in connection with this apparently layout-related bug? See also the strange "too many open files" message, which I can't make sense of.

Comment 29

[Benoit Jacob [:bjacob] (mostly away)]

Another strange thing: this time the sizes are small: width=44, height=9!

Comment 30

[Benoit Jacob [:bjacob] (mostly away)]

Attachment: Call stack and last messages before crash, another run

I got this crash again, this time after a few device orientation changes. This time, the warning messages preceding the crash are quite different:

adb| ###!!! ASSERTION: RenderTo with uninitialised buffer!: 'Initialised()', ...
adb| ###!!! ASSERTION: RenderTo with uninitialised buffer!: 'Initialised()', ...
adb| Compositor: Composite took 2055 ms.
adb| ShmemAndroid::ShareToProcess(): Too many open files (24)

The only common message between these two runs seems to be the 'Too many open files' message, immediately before the crash. Can anyone make sense of it?

Comment 31

[Benoit Jacob [:bjacob] (mostly away)]

Last spam, I promise: all these assertions sound really scary. Why aren't they fatal? Wouldn't it make this bug easier to debug?

Also, how do I run with XPCOM_DEBUG_BREAK=abort on Android with moz-gdb? I tried 

  set environment XPCOM_DEBUG_BREAK=abort

in moz-gdb, but that didn't seem to make a difference.

Comment 32

[Chris Lord [:cwiiis]]

(In reply to Benoit Jacob [:bjacob] from comment #31)
> Last spam, I promise: all these assertions sound really scary. Why aren't
> they fatal? Wouldn't it make this bug easier to debug?
> 
> Also, how do I run with XPCOM_DEBUG_BREAK=abort on Android with moz-gdb? I
> tried 
> 
>   set environment XPCOM_DEBUG_BREAK=abort
> 
> in moz-gdb, but that didn't seem to make a difference.

When I want to run with environment variables, I launch the app first and attach to it with moz-gdb.

To launch with environment variables, you can do:

adb shell am start -a android.activity.MAIN -n org.mozilla.fennec_$USER/org.mozilla.fennec_$USER.App --es env0 VAR1=VAL1 --es env1 VAR2=VAL2


With regards to the too many files, it sounds like we're opening too many shared memory handles and not closing them (leak)? I'm not familiar with how shmem works on android at the moment, so this comment might be nonsense.

Comment 33

[Brad Lassey [:blassey] (use needinfo?)]

re-nom'ing because I don't see this in the top crashes anymore

Comment 34

[Martijn Wargers (dead)]

Sorry, comment 26 should have pointed to http://people.mozilla.com/~mwargers/tests/performance/overflowscrollscrolling.htm
That link doesn't seem to crash anymore on the Samsung Galaxy Nexus or the Samsung Galaxy SII.

The link from comment 25 does close Fennec after zooming in and changing orientation, but that seems more like an Android close because of OOM, not an OOM crash from within Fennec.

Comment 35

[Robert Kaiser]

It looks like the "mozalloc_abort | __swrite | libxul.so@0xeb14f4 | nsHTMLEditRules::GetChildNodesForOperation" signature has been introduced into the stable XUL releases with 10.0.4esr - apparently it wasn't there for 10.0.3esr

Comment 36

[Scoobidiver (away)]

All stacks are buggy (libeditor component), only the abort message is right (graphics component).

Comment 37

[Marco Castelluccio [:marco]]

I'm working on bug 703484 and this abort is showing also on Windows (with omtc enabled), it is ShareToProcess that is failing here.
I don't know if the cause is the same.

Comment 38

[Benoit Jacob [:bjacob] (mostly away)]

The steps-to-reproduce in comment 25 now give me a OOM SIGKILL instead of the above-discussed assertion failures.

Shortly before death, I see this in logcat:

07-05 17:07:40.812 I/GeckoScreenshot( 1334): rect: 30.000000, 0.000000, 980.000000, 1408.750000
07-05 17:07:40.816 I/GeckoScreenshot( 1334): rect: -441.000000, -3592.166748, 1755.650146, 1409.000000
07-05 17:07:45.738 I/ActivityManager(  107): Process android.process.acore (pid 1259) has died.
07-05 17:07:45.867 I/ActivityManager(  107): Process android.process.media (pid 1311) has died.
07-05 17:07:45.867 I/ActivityManager(  107): Low Memory: No more background processes.
07-05 17:07:46.195 I/ActivityManager(  107): Process com.android.launcher (pid 1226) has died.

So apparently we're taking an absurdly large screenshot and immediately then we start running out of memory. The too-large screenshot size seems to be another symptom of the same underlying hypothetical layout bug that would produce such absurdly large sizes.

Comment 39

[Benoit Jacob [:bjacob] (mostly away)]

Here's the code printing this rect message:

http://mxr.mozilla.org/mozilla-central/source/widget/android/nsAppShell.cpp#131

119     virtual nsresult HandleEvent(nsIDOMEvent* aEvent) {
120         nsCOMPtr<nsIDOMNotifyPaintEvent> paintEvent = do_QueryInterface(aEvent);
121         if (!paintEvent)
122             return NS_OK;
123 
124         nsCOMPtr<nsIDOMClientRect> rect;
125         paintEvent->GetBoundingClientRect(getter_AddRefs(rect));
126         float top, left, bottom, right;
127         rect->GetTop(&top);
128         rect->GetLeft(&left);
129         rect->GetRight(&right);
130         rect->GetBottom(&bottom);
131         __android_log_print(ANDROID_LOG_INFO, "GeckoScreenshot", "rect: %f, %f, %f, %f", top, left, right, bottom);
132         AndroidBridge::NotifyPaintedRect(top, left, bottom, right);
133         return NS_OK;
134     }

So apparently this is the rect of a nsIDOMNotifyPaintEvent.

So are (-441.000000, -3592.166748, 1755.650146, 1409.000000) reasonable coords for a NotifyPaintEvent?

Comment 40

[Robert O'Callahan (:roc) (email my personal email if necessary)]

Yes, if the document is that large and it all gets invalidated.

Comment 41

[Scoobidiver (away)]

XUL Fennec is no longer maintained.

Comment 42

[Timothy Nikkel (:tnikkel)]

The code mentioned in the bug title no longer exists.

All of the crash signatures linked to this bug have 0 crashes except for [@ mozalloc_abort ] . That is a very generic signature, and none of the crashes for it have any useful stack at all, so we don't have much info on them. I'm going to remove [@ mozalloc_abort ] from the signatures of this bug since it doesn't seem relevant to this bug and then close this bug since the code it was originally reported for no longer exists.