883059 - crash in mozilla::a11y::EventQueue::PushEvent

Status: VERIFIED FIXED

(Core :: Disability Access APIs, defect)

Description

[Scoobidiver (away)]

There are a few crashes.

Signature 	nsTArray_Impl<nsRefPtr<mozilla::a11y::AccEvent>, nsTArrayInfallibleAllocator>::AppendElements<mozilla::a11y::AccEvent*>(mozilla::a11y::AccEvent* const*, unsigned int) More Reports Search
UUID	43938546-8d1a-46af-baff-8b04f2130610
Date Processed	2013-06-10 14:38:28
Uptime	827
Install Age	13.8 minutes since version was first installed.
Install Time	2013-06-10 14:24:36
Product	MetroFirefox
Version	24.0a1
Build ID	20130610031147
Release Channel	nightly
OS	Windows NT
OS Version	6.2.9200
Build Architecture	x86
Build Architecture Info	GenuineIntel family 6 model 58 stepping 9
Crash Reason	EXCEPTION_ACCESS_VIOLATION_READ
Crash Address	0x8
App Notes 	
AdapterVendorID: 0x8086, AdapterDeviceID: 0x0166, AdapterSubsysID: 05541028, AdapterDriverVersion: 9.17.10.2849
Has dual GPUs. GPU #2: AdapterVendorID2: 0x10de, AdapterDeviceID2: 0x1140, AdapterSubsysID2: 05541028, AdapterDriverVersion2: 9.18.13.1422D2D! D2D+ DWrite? DWrite+ D3D10 Layers! D3D10 Layers+ 
Processor Notes 	sp-processor04_phx1_mozilla_com_3661:2012; non-integer value of "SecondsSinceLastCrash"; WARNING: raw_crash missing Add-ons
EMCheckCompatibility	True
Adapter Vendor ID	0x8086
Adapter Device ID	0x0166
Total Virtual Memory	4294836224
Available Virtual Memory	3675074560
System Memory Use Percentage	55
Available Page File	3118219264
Available Physical Memory	1838673920
Accessibility	Active

Frame 	Module 	Signature 	Source
0 	xul.dll 	nsTArray_Impl<nsRefPtr<mozilla::a11y::AccEvent>,nsTArrayInfallibleAllocator>::Ap 	obj-firefox/dist/include/nsTArray.h:1044
1 	xul.dll 	mozilla::a11y::EventQueue::PushEvent 	accessible/src/base/EventQueue.cpp:30
2 	xul.dll 	mozilla::a11y::DocAccessible::FireDelayedEvent 	accessible/src/generic/DocAccessible-inl.h:30
3 	xul.dll 	mozilla::a11y::DocAccessible::Observe 	accessible/src/generic/DocAccessible.cpp:808
4 	xul.dll 	nsCommandManager::CommandStatusChanged 	embedding/components/commandhandler/src/nsCommandManager.cpp:105
5 	xul.dll 	nsComposerCommandsUpdater::UpdateOneCommand 	editor/composer/src/nsComposerCommandsUpdater.cpp:334
6 	xul.dll 	nsComposerCommandsUpdater::NotifyDocumentCreated 	editor/composer/src/nsComposerCommandsUpdater.cpp:54
7 	xul.dll 	nsEditor::NotifyDocumentListeners 	editor/libeditor/base/nsEditor.cpp:2547
8 	xul.dll 	nsEditor::PostCreate 	editor/libeditor/base/nsEditor.cpp:297
9 	xul.dll 	nsEditingSession::SetupEditorOnWindow 	editor/composer/src/nsEditingSession.cpp:476
10 	xul.dll 	nsEditingSession::MakeWindowEditable 	editor/composer/src/nsEditingSession.cpp:168
11 	xul.dll 	nsHTMLDocument::EditingStateChanged 	content/html/document/src/nsHTMLDocument.cpp:2864
12 	xul.dll 	nsHTMLDocument::DeferredContentEditableCountChange 	content/html/document/src/nsHTMLDocument.cpp:2580
13 	xul.dll 	DeferredContentEditableCountChangeEvent::Run 	content/html/document/src/nsHTMLDocument.cpp:2545
14 	xul.dll 	nsContentUtils::AddScriptRunner 	content/base/src/nsContentUtils.cpp:4827
15 	xul.dll 	nsHTMLDocument::ChangeContentEditableCount 	content/html/document/src/nsHTMLDocument.cpp:2565
16 	xul.dll 	nsGenericHTMLElement::ChangeEditableState 	content/html/content/src/nsGenericHTMLElement.cpp:3102
17 	xul.dll 	nsGenericHTMLElement::SetAttr 	content/html/content/src/nsGenericHTMLElement.cpp:980
18 	xul.dll 	nsGenericHTMLElement::SetContentEditable 	content/html/content/src/nsGenericHTMLElement.h:206
19 	xul.dll 	mozilla::dom::HTMLElementBinding::set_contentEditable 	obj-firefox/dom/bindings/HTMLElementBinding.cpp:854
20 	xul.dll 	mozilla::dom::HTMLElementBinding::genericSetter 	obj-firefox/dom/bindings/HTMLElementBinding.cpp:5071
21 	mozjs.dll 	js::Invoke 	js/src/vm/Interpreter.cpp:389
22 	mozjs.dll 	js::Invoke 	js/src/vm/Interpreter.cpp:435
...

More reports at:
https://crash-stats.mozilla.com/report/list?signature=nsTArray_Impl%3CnsRefPtr%3Cmozilla%3A%3Aa11y%3A%3AAccEvent%3E%2C+nsTArrayInfallibleAllocator%3E%3A%3AAppendElements%3Cmozilla%3A%3Aa11y%3A%3AAccEvent*%3E%28mozilla%3A%3Aa11y%3A%3AAccEvent*+const*%2C+unsigned+int%29

Comment 1

[Trevor Saunders (:tbsaunde)]

I haven't looked at how it happens yet, but I'm pretty sure this is what happens when you try to Call QueueEvent() on a null NotificationController* how we got to that I'm not clear yet.  But on a 32 bit machine mEvents should have an offset of 0x8 because of vtable + pointer to doc.

Comment 2

[alexander :surkov (:asurkov)]

should we assert and null check?

Comment 3

[Martijn Wargers (dead)]

I can reproduce this crash on Windows 8 with a metro enabled Firefox with the following steps:
- Go to http://people.mozilla.org/~mwargers/tests/contenteditable/contenteditable_focus_blur_parentframe.htm
- Let it open for 5 seconds or so.
- Go back in history, then load another page.

Result:
This crash

Comment 4

[Tracy Walker [:tracy]]

This crash has returned in volume on Nightly builds since 20131018.  Comments indicate webmail, outlook email and google hangouts chat. It is correlated to oleacc.dll, the Active Accessibility Core component in Windows. Currently #12 crasher on trunk.

Comment 5

[Trevor Saunders (:tbsaunde)]

I'm not seeing how mNotificationController can be null.  Martijn can you still reproduce on windows?  I can't on linux, surkov want to try on windows?

Comment 6

[alexander :surkov (:asurkov)]

windows 7 doesn't make me crash

Comment 7

[Tracy Walker [:tracy]]

this signature has risen up to #7 on topcrash list

Comment 8

[Martijn Wargers (dead)]

(In reply to Trevor Saunders (:tbsaunde) from comment #5)
> I'm not seeing how mNotificationController can be null.  Martijn can you
> still reproduce on windows?  I can't on linux, surkov want to try on windows?

I was testing this on a borrowed Windows 8 tablet/laptop (which was in tablet mode). I'm not able to test this anymore. Juan might be able test.

Comment 9

[alexander :surkov (:asurkov)]

Trev, can we get a fix? If it's a null check (+ assertion) and if not a root problem fix then it will be better than keeping crashing.

Comment 10

[Trevor Saunders (:tbsaunde)]

(In reply to alexander :surkov from comment #9)
> Trev, can we get a fix? If it's a null check (+ assertion) and if not a root
> problem fix then it will be better than keeping crashing.

I'm hoping its just a dup of bug 931399, so I want to see if that turns out to be the case before thinking about what else it might be.

Comment 11

[Tracy Walker [:tracy]]

This is a top crash on Fx 27 at the moment.  The fix for 931399 may have actually aggravated this crash.

Comment 12

[bhavana bajaj [:bajaj]]

Tracking it as this is a top-crash and passing onto Trevor as this may be a fallout from 931399.

Can we backout 931399 to avoid this top-crash ?

Comment 13

[Trevor Saunders (:tbsaunde)]

(In reply to [:tracy] Tracy Walker - QA Mentor from comment #11)
> This is a top crash on Fx 27 at the moment.  The fix for 931399 may have
> actually aggravated this crash.

that's weird, but possible I guess.

Can we get a test case?

(In reply to bhavana bajaj [:bajaj] from comment #12)
> Tracking it as this is a top-crash and passing onto Trevor as this may be a
> fallout from 931399.
> 
> Can we backout 931399 to avoid this top-crash ?

maybe? I think we may have tests that crash without it at this point, but I can push it to try over the weekend and hopefully the tree will open so I can land it if backing out is an option.

Comment 14

[Trevor Saunders (:tbsaunde)]

I'm not particularly comfortable with it, but I backed bug 931399 out in https://hg.mozilla.org/integration/mozilla-inbound/rev/5d40a8f1b00b since mochitest-a11y seems to still pass at least on linux.

Comment 15

[bhavana bajaj [:bajaj]]

:tracy, is the crash volume down now based on the backout in comment #14 ?

Comment 16

[Tracy Walker [:tracy]]

mozilla::a11y::EventQueue::PushEvent is nowhere to be found.  but [@ nsTArray_Impl<nsRefPtr<mozilla::a11y::AccEvent>, nsTArrayInfallibleAllocator>::AppendElements<mozilla::a11y::AccEvent*>(mozilla::a11y::AccEvent* const*, unsigned int)] is still a top crasher at #6.

Our speculation that 931399 had an effect here was mistaken.

Comment 17

[Trevor Saunders (:tbsaunde)]

(In reply to [:tracy] Tracy Walker - QA Mentor from comment #16)
> mozilla::a11y::EventQueue::PushEvent is nowhere to be found.  but [@
> nsTArray_Impl<nsRefPtr<mozilla::a11y::AccEvent>,
> nsTArrayInfallibleAllocator>::AppendElements<mozilla::a11y::
> AccEvent*>(mozilla::a11y::AccEvent* const*, unsigned int)] is still a top
> crasher at #6.
> 
> Our speculation that 931399 had an effect here was mistaken.

its good to confirm that, I'll check it back in then.  Do we have any str for this bug since its not a dup of 931399?

Comment 18

[Tracy Walker [:tracy]]

the only STR's we had here were from comment #3.  I can't reproduce with those steps, atm. (having difficulty getting lastest nightly build into metro mode.  Will try again once that is resolved.

Comment 19

[Tracy Walker [:tracy]]

It turns out, once desktop/metro profile sharing was turned, you can no longer have multiple profiles. 

Anyway, figured that out, got into metro mode and still can not reproduce this.

Comment 20

[Trevor Saunders (:tbsaunde)]

(In reply to [:tracy] Tracy Walker - QA Mentor from comment #19)
> It turns out, once desktop/metro profile sharing was turned, you can no
> longer have multiple profiles. 
> 
> Anyway, figured that out, got into metro mode and still can not reproduce
> this.

ok :( I'm pretty sure a null check will cause a leak so I don't know there's much we can do here until we have some idea how to reproduce.

Comment 21

[Richard Newman [:rnewman]]

My girlfriend is able to reliably reproduce this on 27:

https://crash-stats.mozilla.com/report/index/464a4e56-4db7-4816-a8c7-880a52131216

Windows 7 tablet. Windows in en_US.

STR: open Outlook Web Access. Click on an email in the inbox. Message window pops up for the reply. Immediately crashes.

I'm asking for about:support content right now.

Comment 22

[Richard Newman [:rnewman]]

Attachment: about:support for an affected user

Comment 23

[Richard Newman [:rnewman]]

(In reply to Trevor Saunders (:tbsaunde) from comment #20)

> ok :( I'm pretty sure a null check will cause a leak so I don't know there's
> much we can do here until we have some idea how to reproduce.

Personal opinion: a leak is way less serious than repeatably crashing every time a user tries to visit their webmail.

Perhaps land a band-aid on all channels but Nightly, then continue to debug?

Is there anything I can do to debug, given that I have some ability (at a distance right now) to repro?

Comment 24

[Trevor Saunders (:tbsaunde)]

assuming http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/[e-mail]-df08ed89ae2a builds and crashes I'd like to know what output it dumps to standard out if you run it in aterminal (however you see standard out on windows Alex?)

fwiw I think Alex tried to and failed to reprudce this with the outlook web thing, but I'll try on linux just for fun, I just need to setup an account or something I guess.

Comment 25

[Richard Newman [:rnewman]]

(In reply to Trevor Saunders (:tbsaunde) from comment #24)
> I'd like to know what
> output it dumps to standard out if you run it in aterminal (however you see
> standard out on windows Alex?)

That needs some additional help, I think -- running with -console starts a new temporary console which doesn't show any of the usual Firefox log output, and of course it disappears when the process goes away. I don't do much dev on Windows, and the affected machine doesn't have dev tools installed (nor space for them).

Can you make a build that logs whatever you're trying to capture to a file?


> fwiw I think Alex tried to and failed to reprudce this with the outlook web
> thing, but I'll try on linux just for fun, I just need to setup an account
> or something I guess.

I doubt it'll be worth your time -- 100% of the thousands of crashes in Socorro with this signature are on Windows, and Tracy suggested that it's an interaction with MSAA (which, judging by Googling, is a contributor to crashes in other software, too).

Comment 26

[alexander :surkov (:asurkov)]

(In reply to Trevor Saunders (:tbsaunde) from comment #24)
> assuming
> http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/trev.
> [e-mail]-df08ed89ae2a builds and crashes I'd like to know what
> output it dumps to standard out if you run it in aterminal (however you see
> standard out on windows Alex?)

what standard out should I check and what's for? 

(In reply to Richard Newman [:rnewman] from comment #23)

> Is there anything I can do to debug, given that I have some ability (at a
> distance right now) to repro?

if could check the state of the document in DocAccessible::FireDelayedEvent (it looks like it's defunct at this point), see Accessible::mStateFlags. If so then I'd say that editable state change notification should be ignored

Comment 27

[Trevor Saunders (:tbsaunde)]

(In reply to alexander :surkov from comment #26)
> (In reply to Trevor Saunders (:tbsaunde) from comment #24)
> > assuming
> > http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/trev.
> > [e-mail]-df08ed89ae2a builds and crashes I'd like to know what
> > output it dumps to standard out if you run it in aterminal (however you see
> > standard out on windows Alex?)
> 
> what standard out should I check and what's for? 

I was just asking how you do it because I have no clue how it works in windows.

> (In reply to Richard Newman [:rnewman] from comment #23)
> 
> > Is there anything I can do to debug, given that I have some ability (at a
> > distance right now) to repro?
> 
> if could check the state of the document in DocAccessible::FireDelayedEvent
> (it looks like it's defunct at this point), see Accessible::mStateFlags. If
> so then I'd say that editable state change notification should be ignored

I'd rather localize the hack to DocAccessible::Observe()

Comment 28

[Trevor Saunders (:tbsaunde)]

(In reply to Richard Newman [:rnewman] from comment #25)
> (In reply to Trevor Saunders (:tbsaunde) from comment #24)
> > I'd like to know what
> > output it dumps to standard out if you run it in aterminal (however you see
> > standard out on windows Alex?)
> 
> That needs some additional help, I think -- running with -console starts a
> new temporary console which doesn't show any of the usual Firefox log
> output, and of course it disappears when the process goes away. I don't do
> much dev on Windows, and the affected machine doesn't have dev tools
> installed (nor space for them).
> 
> Can you make a build that logs whatever you're trying to capture to a file?
> 
> 
> > fwiw I think Alex tried to and failed to reprudce this with the outlook web
> > thing, but I'll try on linux just for fun, I just need to setup an account
> > or something I guess.
> 
> I doubt it'll be worth your time -- 100% of the thousands of crashes in
> Socorro with this signature are on Windows, and Tracy suggested that it's an
> interaction with MSAA (which, judging by Googling, is a contributor to
> crashes in other software, too).

Well, msaa may well be what's causing accessibility to be active, but I really don't see how this could be windows specific.

Comment 29

[Trevor Saunders (:tbsaunde)]

ok, lets try this, I believe the log will go to $CWD/firefox-crash-log does that work for you?  # Uncomment to resolve bug

Comment 30

[Trevor Saunders (:tbsaunde)]

of course the try link would help https://tbpl.mozilla.org/?tree=Try&rev=a3ca41cced3a

Comment 31

[alexander :surkov (:asurkov)]

(In reply to Trevor Saunders (:tbsaunde) from comment #27)
> (In reply to alexander :surkov from comment #26)
> > (In reply to Trevor Saunders (:tbsaunde) from comment #24)
> > > assuming
> > > http://ftp.mozilla.org/pub/mozilla.org/firefox/try-builds/trev.
> > > [e-mail]-df08ed89ae2a builds and crashes I'd like to know what
> > > output it dumps to standard out if you run it in aterminal (however you see
> > > standard out on windows Alex?)
> > 
> > what standard out should I check and what's for? 
> 
> I was just asking how you do it because I have no clue how it works in
> windows.

oh, just clicked on message, then on forward button (reply is not active since those automatic messages)

> > (In reply to Richard Newman [:rnewman] from comment #23)
> > 
> > > Is there anything I can do to debug, given that I have some ability (at a
> > > distance right now) to repro?
> > 
> > if could check the state of the document in DocAccessible::FireDelayedEvent
> > (it looks like it's defunct at this point), see Accessible::mStateFlags. If
> > so then I'd say that editable state change notification should be ignored
> 
> I'd rather localize the hack to DocAccessible::Observe()

if you talk about the fix then yes, I meant the steps to check for the person who can reproduce

Comment 32

[u279076]

7 Day Ranking:
> Firefox 26: N/A
> Firefox 27: #5 @ 1.75% (+0.08%)
> Firefox 28: N/A
> Firefox 29: N/A

Current rankings show this to be a top-crash in Firefox 27 but it seems to have all but disappeared in Aurora, Nightly, and Release.

Comment 33

[JonasK]

i can confirm that:
on Win7 x64 Firefox beta 27 crashed on me every time i opened an email composer in Outlook Web Access on a corporate Exchange Server.

for example: 
https://crash-stats.mozilla.com/report/index/66f3f9ca-1f03-4ed6-bdcf-844a12131218
https://crash-stats.mozilla.com/report/index/7f15612d-43a7-4440-9d4a-531032131213

from a quick check Firefox 28 Aurora does NOT crash doing the same operations.

on a sidenote: i have a WinXP machine and 27 did NOT crash there. Don't know if this helps though.

Comment 34

[alexander :surkov (:asurkov)]

Trev, what about to add IsDefunct check into Observe special for Firefox 27?

Comment 35

[Trevor Saunders (:tbsaunde)]

Attachment: bug 883059 - check if the document is defunct before handling observer notifications

If this is beta only I'm going to guess bug 931399 actually did fix this (or atleast I'm going to hope that), and I don't really care enough to get this debugged so lets take the IsDefunct() check on beta only

Comment 36

[alexander :surkov (:asurkov)]

Comment on attachment 8350640 [details] [diff] [review]
bug 883059 - check if the document is defunct before handling observer notifications

Review of attachment 8350640 [details] [diff] [review]:
-----------------------------------------------------------------

r=me, thanks! (is it supposed to be landed into Firefox 27 only)?

Comment 37

[Trevor Saunders (:tbsaunde)]

Comment on attachment 8350640 [details] [diff] [review]
bug 883059 - check if the document is defunct before handling observer notifications

[Approval Request Comment]
Bug caused by (feature/regressing bug #): unknown
User impact if declined: crashes
Testing completed (on m-c, etc.): locally tested it builds and passed tests, haven't yet tested it fixes crash but it really really should
Risk to taking this patch (and alternatives if risky): basically non, we could possible fail to tell people documents have entered content editable mode as they go away, but almost certainly effects nothing.
String or IDL/UUID changes made by this patch:none

Comment 38

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 8350640 [details] [diff] [review]
bug 883059 - check if the document is defunct before handling observer notifications

Before approving this to land on FF27 (beta) - are we sure that this won't be needed on central/aurora?  Those are marked as affected.

Comment 39

[Trevor Saunders (:tbsaunde)]

(In reply to Lukas Blakk [:lsblakk] from comment #38)
> Comment on attachment 8350640 [details] [diff] [review]
> bug 883059 - check if the document is defunct before handling observer
> notifications
> 
> Before approving this to land on FF27 (beta) - are we sure that this won't
> be needed on central/aurora?  Those are marked as affected.


seems they actually aren't see comments 32 / 33

Comment 40

[Lukas Blakk [:lsblakk] use ?needinfo]

Comment on attachment 8350640 [details] [diff] [review]
bug 883059 - check if the document is defunct before handling observer notifications

OK we can go ahead with a direct-to-Beta uplift.

Comment 41

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/5a92c34bbba2

Comment 42

[JonasK]

i verified the latest Firefox Beta 27 20140106141415 does not crash anymore :). So at least for me this is fixed.

Comment 43

[u279076]

(In reply to JonasK from comment #42)
> i verified the latest Firefox Beta 27 20140106141415 does not crash anymore

Thanks!

Comment 44

[Tracy Walker [:tracy]]

Crash stats support verification: as of 27.0b4 there this crash is no longer occurring.