1021149 - [OMTC] crash in mozalloc_abort(char const* const) | mozilla::ipc::MessageChannel::DebugAbort(char const*, int, char const*, char const*, bool) | mozilla::ipc::MessageChannel::~MessageChannel() | mozilla::layers::PImageBridgeParent::~PImageBridgeParent()

Status: VERIFIED FIXED

(Core :: Graphics: Layers, defect)

Description

[juan becerra [:juanb]]

This bug was filed from the Socorro interface and is 
report bp-6bceca79-3a53-403e-ad69-a95142140521.
=============================================================

This signature appeared starting 5/21 and it is happening mostly on Win7, followed by Win8.1 and Win8. There are no comments in the reports. Currently in the top 25 crashers list on nightly.

More reports can be found here: https://crash-stats.mozilla.com/report/list?product=Firefox&signature=mozalloc_abort%28char+const%2A+const%29+%7C+NS_DebugBreak+%7C+mozilla%3A%3Aipc%3A%3AMessageChannel%3A%3ADebugAbort%28char+const%2A%2C+int%2C+char+const%2A%2C+char+const%2A%2C+bool%29+%7C+mozilla%3A%3Aipc%3A%3AMessageChannel%3A%3A%7EMessageChannel%28%29+%7C+mozilla%3A%3Alayers%3A%3APImageBridgeParent%3A%3A%7EPImageBridgeParent%28%29

0 	mozalloc.dll 	mozalloc_abort(char const * const) 	memory/mozalloc/mozalloc_abort.cpp
1 	xul.dll 	NS_DebugBreak 	xpcom/base/nsDebugImpl.cpp
2 	xul.dll 	mozilla::ipc::MessageChannel::DebugAbort(char const *,int,char const *,char const *,bool) 	ipc/glue/MessageChannel.cpp
3 	xul.dll 	mozilla::ipc::MessageChannel::~MessageChannel() 	ipc/glue/MessageChannel.cpp
4 	xul.dll 	mozilla::layers::PImageBridgeParent::~PImageBridgeParent() 	obj-firefox/ipc/ipdl/PImageBridgeParent.cpp
5 	xul.dll 	mozilla::layers::ImageBridgeParent::~ImageBridgeParent() 	gfx/layers/ipc/ImageBridgeParent.cpp
6 	xul.dll 	mozilla::layers::ImageBridgeParent::`vector deleting destructor'(unsigned int) 	
7 	xul.dll 	mozilla::AtomicRefCountedWithFinalize<mozilla::layers::ISurfaceAllocator>::Release() 	obj-firefox/dist/include/mozilla/layers/AtomicRefCountedWithFinalize.h
8 	xul.dll 	mozilla::layers::DeleteImageBridgeSync 	gfx/layers/ipc/ImageBridgeChild.cpp
9 	xul.dll 	RunnableFunction<void (*)(mozilla::layers::ImageBridgeChild *,mozilla::layers::ImageBridgeParent *),Tuple2<mozilla::layers::ImageBridgeChild *,mozilla::layers::ImageBridgeParent *> >::Run() 	ipc/chromium/src/base/task.h
10 	xul.dll 	MessageLoop::RunTask(Task *) 	ipc/chromium/src/base/message_loop.cc
11 	xul.dll 	MessageLoop::DeferOrRunPendingTask(MessageLoop::PendingTask const &) 	ipc/chromium/src/base/message_loop.cc
12 	xul.dll 	MessageLoop::DoWork() 	ipc/chromium/src/base/message_loop.cc
13 	xul.dll 	base::MessagePumpDefault::Run(base::MessagePump::Delegate *) 	ipc/chromium/src/base/message_pump_default.cc
14 	xul.dll 	MessageLoop::RunHandler() 	ipc/chromium/src/base/message_loop.cc
15 	xul.dll 	MessageLoop::Run() 	ipc/chromium/src/base/message_loop.cc
16 	xul.dll 	base::Thread::ThreadMain() 	ipc/chromium/src/base/thread.cc
17 	xul.dll 	`anonymous namespace'::ThreadFunc(void *) 	ipc/chromium/src/base/platform_thread_win.cc
18 	kernel32.dll 	kernel32.dll@0xb729

Comment 2

[(Away)]

Over in bug 924622 comment 35, Benjamin says:

> See also bug 959080 where this is seen on mac as well, so it seems to affect every platform with OMTC enabled.

Given the timing, I guess OMTC on Windows is now part of this club.

Comment 3

[(Away)]

I guess I should add, the abort message is:
###!!! ABORT: mismatched CxxStackFrame ctor/dtors: file c:\builds\moz2_slave\m-cen-w32-ntly-000000000000000\build\ipc\glue\MessageChannel.cpp, line 1732

Comment 4

[Cosmin Malutan, [:cosmin-malutan]]

This crashed in mozmill testrun too, after:
http://hg.mozilla.org/qa/mozmill-tests/file/default/firefox/tests/functional/testToolbar/testHomeButton.js
https://crash-stats.mozilla.com/report/index/c76a6fa7-e34f-43f1-9a68-3061e2140610

Comment 5

[Cosmin Malutan, [:cosmin-malutan]]

Attachment: jenkins_log.txt

This is the jenkins log, though I couldn't reproduce the crash, I have seen the same log we had before the crash:
> ###!!! [Child][DispatchAsyncMessage] Error: Route error: message sent to unknown actor ID

Comment 6

[Ioana Chiorean]

I got this on Nexus 4 (android 4.4 ) after playing a video fro like 5s

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Looking at the following URL shows that the crashes have been started on May 21st:

https://crash-stats.mozilla.com/report/list?product=Firefox&range_unit=days&range_value=28&signature=mozalloc_abort%28char+const*+const%29+|+NS_DebugBreak+|+mozilla%3A%3Aipc%3A%3AMessageChannel%3A%3ADebugAbort%28char+const*%2C+int%2C+char+const*%2C+char+const*%2C+bool%29+|+mozilla%3A%3Aipc%3A%3AMessageChannel%3A%3A~MessageChannel%28%29+|+mozilla%3A%3Alayers%3A%3APImageBridgeParent%3A%3A~PImageBridgeParent%28%29#tab-reports

Maybe the pushlog helps us a bit here:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=cb9f34f73ebe&tochange=9d8d16695f6a

(In reply to David Major [:dmajor] (UTC+12) from comment #2)
> > See also bug 959080 where this is seen on mac as well, so it seems to affect every platform with OMTC enabled.
> 
> Given the timing, I guess OMTC on Windows is now part of this club.

Well, good tip that is the problem. Bug 899785 landed that day on mozilla-central.

Comment 8

[Daniel Gherasim]

(In reply to Henrik Skupin (:whimboo) from comment #7)
> Looking at the following URL shows that the crashes have been started on May
> 21st:
> 
> https://crash-stats.mozilla.com/report/
> list?product=Firefox&range_unit=days&range_value=28&signature=mozalloc_abort%
> 28char+const*+const%29+|+NS_DebugBreak+|+mozilla%3A%3Aipc%3A%3AMessageChannel
> %3A%3ADebugAbort%28char+const*%2C+int%2C+char+const*%2C+char+const*%2C+bool%2
> 9+|+mozilla%3A%3Aipc%3A%3AMessageChannel%3A%3A~MessageChannel%28%29+|+mozilla
> %3A%3Alayers%3A%3APImageBridgeParent%3A%3A~PImageBridgeParent%28%29#tab-
> reports
> 
> Maybe the pushlog helps us a bit here:
> http://hg.mozilla.org/mozilla-central/
> pushloghtml?fromchange=cb9f34f73ebe&tochange=9d8d16695f6a
> 
> (In reply to David Major [:dmajor] (UTC+12) from comment #2)
> > > See also bug 959080 where this is seen on mac as well, so it seems to affect every platform with OMTC enabled.
> > 
> > Given the timing, I guess OMTC on Windows is now part of this club.
> 
> Well, good tip that is the problem. Bug 899785 landed that day on
> mozilla-central.

So we get crashes about 25-30% of the time when running our testruns on XP nodes.

I'v done the regression range & it looks like this is the one:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=a6a457fe2a2a&tochange=7a8d540c1cb2

*** 1402826524 - 15.06.2014   13:26:00
*** 500 testruns - NO CRASH AT ALL

*** 1402849754 - 15.06.2014   17:14:00
*** CRASHES 17/100 CRASHES

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to daniel.gherasim from comment #8)
> I'v done the regression range & it looks like this is the one:
> http://hg.mozilla.org/mozilla-central/
> pushloghtml?fromchange=a6a457fe2a2a&tochange=7a8d540c1cb2

Thanks Daniel. But this is not fully done yet. We have too many changesets here, so please continue to find the single changeset, which causes this crash to appear.

Comment 10

[Daniel Gherasim]

(In reply to Henrik Skupin (:whimboo) from comment #9)
> (In reply to daniel.gherasim from comment #8)
> > I'v done the regression range & it looks like this is the one:
> > http://hg.mozilla.org/mozilla-central/
> > pushloghtml?fromchange=a6a457fe2a2a&tochange=7a8d540c1cb2
> 
> Thanks Daniel. But this is not fully done yet. We have too many changesets
> here, so please continue to find the single changeset, which causes this
> crash to appear.

Done:
http://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=f903c0f48969&tochange=bb0c777636e9

Comment 11

[Daniel Gherasim]

So we're seeing crashes a lot on XP after bug 774388 landed when running our tests.
Jacob, any idea what cause this ?

Comment 12

[Benoit Jacob [:bjacob] (mostly away)]

(In reply to David Major [:dmajor] (Away until June 23) from comment #3)
> I guess I should add, the abort message is:
> ###!!! ABORT: mismatched CxxStackFrame ctor/dtors: file
> c:\builds\moz2_slave\m-cen-w32-ntly-
> 000000000000000\build\ipc\glue\MessageChannel.cpp, line 1732

This assert, and the call stack in comment 0, are the same as Bug 924622, so this looks like the same bug, and the changeset identified in comment 10 would have merely changed the resulting signature (?). Do you know specifically if the frequency of this crash increased with this changeset, compared to what was seen on bug 924622 ?

Comment 13

[Benoit Jacob [:bjacob] (mostly away)]

Oh or is the new thing here that this is seen on Windows whereas bug 924622 was on Android?

Comment 14

[Benoit Jacob [:bjacob] (mostly away)]

Here's the question: is the changeset identified in comment 10 the first changeset where windows is crashy, or the first changeset where windows crashes *with that particular signature* ?

I'm asking because it is not surprising that OMTC-on-windows would make until-now-Android-only OMTC crashes now also happen on Windows. So I suppose that two different things could have happened here: 1) OMTC-on-windows made bug 924622 now also affect Windows, and 2) the changeset identified in comment 10 could have changed its signature. Is that theory validated or invalidated by your experimental data?

Comment 15

[Daniel Gherasim]

Hi Jacob, this happens on Windowx XP with this signature: 
[@ EMPTY: no crashing thread identified; ERROR_NO_THREAD_LIST].

I ran 500 tests with the build before that changeset & it didn't crash at all. 
With the build after changeset was applied crashed 15/100.

*** 1402708216 DATE
*** 500 RUNS / NO CRASH
*** URL

*** 1402711754 DATE
*** 100 RUNS / 15 crashes
*** URL

Running a simple dummy test will cause the crash (open/close firefox).

Details from 'about:support' on the machine I ran the tests (VM XP x86):
Graphics
Adapter Drivers    vga framebuf vga256 vga64k
Adapter RAM    Unknown
Device ID    0x0000
Direct2D Enabled    Blocked for your graphics card because of unresolved driver issues.
DirectWrite Enabled    false (0.0.0.0)
GPU #2 Active    false
GPU Accelerated Windows    0/1 Basic Blocked for your graphics card because of unresolved driver issues.
Vendor ID    0x0000
WebGL Renderer    Blocked for your graphics card because of unresolved driver issues.
windowLayerManagerRemote    false
AzureCanvasBackend    skia
AzureContentBackend    cairo
AzureFallbackCanvasBackend    cairo
AzureSkiaAccelerated    0

Comment 16

[Andrei Eftimie]

Interesting. I've run the same test on a dedicated XP machine, and I got no crashes in 300 runs.

> Graphics
> Adapter Description ATI Radeon 3000 Graphics
> Adapter Drivers ati2dvag
> Adapter RAM Unknown
> Device ID 0x9616
> DirectWrite Enabled false (0.0.0.0)
> Driver Date 7-3-2012
> Driver Version  8.970.100.3000
> GPU #2 Active false
> GPU Accelerated Windows 1/1 Direct3D 9 (OMTC)
> Vendor ID 0x1002
> WebGL Renderer  Google Inc. -- ANGLE (ATI Radeon 3000 Graphics Direct3D9 vs_3_0 ps_3_0)
> windowLayerManagerRemote  true
> AzureCanvasBackend  skia
> AzureContentBackend cairo
> AzureFallbackCanvasBackend  cairo
> AzureSkiaAccelerated  0

Comment 17

[Benoit Jacob [:bjacob] (mostly away)]

Thanks for the testing; I'm currently looking deep into this crash; if after a few days I can't fix it, I'll back out the patch that you've identified to be regressing this.

Comment 18

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Maybe this crash only happens on virtual machines run under VMware? Looks like the graphics driver cannot be correctly identified. Just a note here, for all of the VMs we do not have 3D acceleration enabled.

Comment 19

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Daniel or Andrei, could one of you please check if the crash also appears when you turn on/off 3D graphics support in VMware for this machine? Also are the VMware tools up-to-date in the VM? If not may an update of them fix it?

Comment 20

[Daniel Gherasim]

(In reply to Henrik Skupin (:whimboo) from comment #18)
> Maybe this crash only happens on virtual machines run under VMware? Looks
> like the graphics driver cannot be correctly identified. Just a note here,
> for all of the VMs we do not have 3D acceleration enabled.

I'm using Virtual Box for the local machine on which firefox crashes.

(In reply to Henrik Skupin (:whimboo) from comment #19)
> Daniel or Andrei, could one of you please check if the crash also appears
> when you turn on/off 3D graphics support in VMware for this machine? Also
> are the VMware tools up-to-date in the VM? If not may an update of them fix
> it?

I enabled 3d support on my local VM, it shows that it's enabled in dxdiag but in about:support I got the same configuration. Crash still reproduces.

Current configuration:
> Adapter Description    VirtualBox Graphics Adapter
> Adapter Drivers    VBoxDisp
> Adapter RAM    Unknown
> Device ID    0xbeef
> Direct2D Enabled    Blocked for your graphics card because of unresolved driver issues.
> DirectWrite Enabled    false (0.0.0.0)
> Driver Date    12-18-2013
> Driver Version    4.3.6.0
> GPU #2 Active    false
> GPU Accelerated Windows    0/1 Basic Blocked for your graphics card because of unresolved driver issues.
> Vendor ID    0x80ee
> WebGL Renderer    Blocked for your graphics card because of unresolved driver issues.
> windowLayerManagerRemote    false
> AzureCanvasBackend    skia
> AzureContentBackend    cairo
> AzureFallbackCanvasBackend    cairo
> AzureSkiaAccelerated    0

Comment 21

[Andrei Eftimie]

This has a really high failure rate on our CI Windows XP machines, rendering tests on these machines uneffective.

Comment 22

[Henrik Skupin [:whimboo][⌚️UTC+2]]

qablocker is not the keyword you want to use. What you want is actually the qa-automation-blocked whiteboard entry.

Comment 23

[Benoit Jacob [:bjacob] (mostly away)]

Note taken; I am still working full time on resolving this class of issues; this green try push gives hope that fixes could land soon: https://tbpl.mozilla.org/?tree=Try&rev=ef203ea5b300

Comment 24

[Andrei Eftimie]

(In reply to Benoit Jacob [:bjacob] from comment #23)
> Note taken; I am still working full time on resolving this class of issues;
> this green try push gives hope that fixes could land soon:
> https://tbpl.mozilla.org/?tree=Try&rev=ef203ea5b300

I've tested this try build and it seems to work fine, I haven't had any crash with it.

Comment 25

[Andrei Eftimie]

I spoke too soon. I did got a crash, twice (much more infrequent than the original)

But this is another another crash:
> Firefox 33.0a1 Crash Report [@ EMPTY: no crashing thread identified; ERROR_NO_MINIDUMP_HEADER ]
which appears to be bug 711568.

This with the try build mentioned in comment 23.
Seems to be totally unrelated...

Comment 26

[Benoit Jacob [:bjacob] (mostly away)]

My patches have finally landed and stuck, on mozilla-central. So you could use current TBPL builds or wait for tomorrow's Nightly.

Comment 27

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Benoit Jacob [:bjacob] from comment #26)
> My patches have finally landed and stuck, on mozilla-central. So you could
> use current TBPL builds or wait for tomorrow's Nightly.

On which bug did that happen? I don't see a comment here or on any of the dependent bugs.

Comment 28

[Benoit Jacob [:bjacob] (mostly away)]

But 774388 - which the present bug blocks. Sorry about insufficient communication :-)

Comment 29

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Thanks for the info. Adding it accordingly.

Comment 30

[Andrei Eftimie]

We haven't seen any crashes since bug 774388 landed on our Mozmill daily testruns.
This appears to be fixed (for our testcases at least).

Comment 31

[Benoit Jacob [:bjacob] (mostly away)]

Great. Yes, that is what I expected. Let's mark this bug accordingly then. I'm defaulting to FIXED but maybe that should be VERIFIED?

Comment 32

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Last reported crash for Firefox 33.0a1 was with 20140707030202. That's the date when the fix on the other bug has been landed. Interesting is 32.0a2 which had the last crash with 20140625004001. Maybe that is something different, given the low volume in crashes too.

Flagging as verified fixed, but not sure what to do with status-firefox32.

Comment 33

[Lawrence Mandel [:lmandel] (use needinfo)]

Marking status32 as unaffected as OMTC has been disabled on 32.