1144016 - [FFOS7715 v2.1][Message] monkey test crash 0 libxul.so!nsTArray_base<nsTArrayInfallibleAllocator, nsTArray_CopyWithMemutils>::Length() const [nsTArray.h : 329 + 0x0]

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: RIL, defect)

Description

[Wei Gao (Spreadtrum)]

0  libxul.so!nsTArray_base<nsTArrayInfallibleAllocator, nsTArray_CopyWithMemutils>::Length() const [nsTArray.h : 329 + 0x0]
1  libxul.so!mozilla::dom::mobilemessage::MobileMessageCursorCallback::NotifyCursorResult(nsISupports**, unsigned int) [MobileMessageCursorCallback.cpp : 158 + 0x7]
2  libxul.so!mozilla::dom::mobilemessage::MobileMessageCursorChild::DoNotifyResult(nsTArray<mozilla::dom::mobilemessage::MobileMessageData> const&) [SmsChild.cpp : 354 + 0xb]
3  libxul.so!mozilla::dom::mobilemessage::MobileMessageCursorChild::RecvNotifyResult(mozilla::dom::mobilemessage::MobileMessageCursorData const&) [SmsChild.cpp : 298 + 0x3]
4  libxul.so!mozilla::dom::mobilemessage::PMobileMessageCursorChild::OnMessageReceived(IPC::Message const&) [PMobileMessageCursorChild.cpp : 179 + 0x9]
5  libxul.so!mozilla::dom::PContentChild::OnMessageReceived(IPC::Message const&) [PContentChild.cpp : 3982 + 0x7]
6  libxul.so!mozilla::ipc::MessageChannel::DispatchAsyncMessage(IPC::Message const&) [MessageChannel.cpp : 1233 + 0x5]
7  libxul.so!mozilla::ipc::MessageChannel::OnMaybeDequeueOne() [MessageChannel.cpp : 1098 + 0x3]
8  libxul.so!RunnableMethod<FdWatcher, void (FdWatcher::*)(), Tuple0>::Run() [tuple.h : 383 + 0x13]
9  libxul.so!mozilla::ipc::MessageChannel::DequeueTask::Run() [MessageChannel.h : 411 + 0x9]
10  libxul.so!MessageLoop::RunTask(Task*) [message_loop.cc : 362 + 0x5]
11  libxul.so!MessageLoop::DeferOrRunPendingTask(MessageLoop::PendingTask const&) [message_loop.cc : 370 + 0x5]
12  libxul.so!MessageLoop::DoWork() [message_loop.cc : 448 + 0x3]
13  libxul.so!mozilla::ipc::DoWorkRunnable::Run() [MessagePump.cpp : 233 + 0x7]
14  libxul.so!nsThread::ProcessNextEvent(bool, bool*) [nsThread.cpp : 823 + 0x5]
15  libxul.so!NS_ProcessNextEvent(nsIThread*, bool) [nsThreadUtils.cpp : 265 + 0xb]
16  libxul.so!mozilla::ipc::MessagePump::Run(base::MessagePump::Delegate*) [MessagePump.cpp : 99 + 0x7]
17  libxul.so!MessageLoop::RunInternal() [message_loop.cc : 234 + 0x5]
18  libxul.so!MessageLoop::Run() [message_loop.cc : 227 + 0x5]
19  libxul.so!nsBaseAppShell::Run() [nsBaseAppShell.cpp : 164 + 0x7]
20  libxul.so!XRE_RunAppShell [nsEmbedFunctions.cpp : 702 + 0x5]
21  libxul.so!MessageLoop::RunInternal() [message_loop.cc : 234 + 0x5]
22  libxul.so!MessageLoop::Run() [message_loop.cc : 227 + 0x5]
23  libxul.so!XRE_InitChildProcess [nsEmbedFunctions.cpp : 539 + 0x5]
24  libxul.so!content_process_main(int, char**) [plugin-container.cpp : 150 + 0x7]
25  libxul.so!mozilla::ipc::ProcLoaderLoadRunner::DoWork() [ProcessUtils_linux.cpp : 415 + 0x7]
26  libxul.so!XRE_ProcLoaderServiceRun [ProcessUtils_linux.cpp : 574 + 0x7]
27  b2g!main [B2GLoader.cpp : 219 + 0x13]
28  libc.so!__libc_init [libc_init_dynamic.cpp : 112 + 0x7]
29  b2g + 0x2be2
30  linker!set_soinfo_pool_protection [linker.cpp : 291 + 0xb]
31  0xbedcdb5c

Comment 1

[Wei Gao (Spreadtrum)]

Hi Vance:

  We got this crash when run monkey test, could you please help with it. thanks.

Comment 2

[Wei Gao (Spreadtrum)]

It seems when getting the length of pending, the crash occurs.

nsTArray<nsCOMPtr<nsISupports>>& pending = mDOMCursor->mPendingResults;
pending.Length()

Comment 3

[Julien Wajsberg [:julienw]]

From the stack, the issue seems to be in the Mobile Messaging code.

Bevis, maybe you could have a look or find someone who could ?

Comment 4

[Bevis Tseng[:bevis][:btseng](Exited)]

The call stack looks similar to the crash in bug 1119608 comment 19.
Need to review the implementation in MobileMessage to see if there is any potential memory corruption issue. :(

Comment 5

[Wei Gao (Spreadtrum)]

I have uploaded logs at:

ftp://ftp.spreadtrum.com/7715/1144016/1144016.tar.bz2

username:mouzhi
password:mouZHI$$61

Thanks.

Comment 6

[Wei Gao (Spreadtrum)]

Dear Vance

Do you know who can help to analyze this problem?
Thanks a great.

Comment 7

[Bevis Tseng[:bevis][:btseng](Exited)]

Sorry, I am still blocked on other blocker issue.

It will be nice to that someone can help on this potential memory corruption issue.

Comment 8

[Wei Gao (Spreadtrum)]

(In reply to Bevis Tseng[:bevistseng][:btseng] from comment #7)
> Sorry, I am still blocked on other blocker issue.
> 
> It will be nice to that someone can help on this potential memory corruption
> issue.

Ok, thanks for your attention.
By the way, could you give me some advice about under what conditions the memory corruption issue may happen?
And do we have some similar bugs and the corresponding bug number?
Thanks so much.

Comment 9

[Bevis Tseng[:bevis][:btseng](Exited)]

(In reply to Wei Gao (Spreadtrum) from comment #8)
> (In reply to Bevis Tseng[:bevistseng][:btseng] from comment #7)
> > Sorry, I am still blocked on other blocker issue.
> > 
> > It will be nice to that someone can help on this potential memory corruption
> > issue.
> 
> Ok, thanks for your attention.
> By the way, could you give me some advice about under what conditions the
> memory corruption issue may happen?
> And do we have some similar bugs and the corresponding bug number?
> Thanks so much.

This could be something related to improper cycle-collection of the created objects in MobileMessageManager::GetThreads() and MobileMessageManager::GetMessages() and their corresponding callback in MobileMessageCursorCallback.

I'll take some time to review these next week to see if there is anything abnormal.

[1] http://hg.mozilla.org/releases/mozilla-b2g34_v2_1s/file/0d285c6423fd/dom/mobilemessage/MobileMessageManager.cpp
[2] http://hg.mozilla.org/releases/mozilla-b2g34_v2_1s/file/0d285c6423fd/dom/mobilemessage/MobileMessageCursorCallback.cpp

Comment 10

[Bevis Tseng[:bevis][:btseng](Exited)]

Attachment: Debugging Memory Corruption.

Hi Wei Gao,

After reviewing of the reference couting of these objects related to the
mobilemessage query, I did't find any suspicious point of how the crash 
could be happened in comment 0.

In this debugging patch, I've assigned each instance a unique |ObjectId| 
and more debugging messages for further analysis to see if there is any
object de-referenced unexpectedly when we still needs it.

I've done some test locally with this debugging patch.
However, I didn't see anything abnormal so far.

Would you please have a build with these debugging info for further analysis?
(Note: Both crash dump & adb logcat are required if it happens again.)

Thanks!
--
Here is a summary of how these objects are used for thread/message queries
in IPC mode for your reference:
1. |MobileMessageCursorCallback|: 
   - Created in either MobileMessageManager::GetMessages() or GetThreads().
   - This callback will be reference-counted by 
     |MobileMessageCursorChild::mCursorCallback|.
   - Contains a data member |mDOMCursor| to a reference-counted 
     |MobileMessageCursor| instance.
2. |MobileMessageCursor|
   - Created and returned in either MobileMessageManager::GetMessages() 
     or GetThreads().
   - referenced-counted by both |MobileMessageCursorCallback::mDOMCursor|
     and the pages in SMS app which invoke GetMessages()/GetThreads().
   - A subclass of |DOMCursor| to override DOMCursor::Continue() called 
     by SMS app to fetch the next queried result with cursor.continue().
   - inherits a data member |mCallback| from |DOMCuror| which references
     a |MobileMessageCursorChild| object as an implementation of 
     |nsICursorContinueCallback|.
3. |MobileMessageCursorChild|
   - Created by |SmsIPCService::CreateMessageCursor()/CreateThreadCursor()|
     where SmsIPCService here represents an IPC proxy of 
     |nsIMobileMessageDatabaseService|.
   - An IPC-version of |nsICursorContinueCallback| which delivers |Continue|
     requests and the corresponding response by invoking 
     |MobileMessageCursorCallback::NotifyCursorResult()/NotifyCursorDone()
     /NotifyCursorError()| through IPC channel.
   - Contains a data member |mCursorCallback| to a reference-counted 
     |MobileMessageCursorCallback| instance.
   - Reference-counted by both |MobileMessageCursor::mCallback| and the IPC channel.
4. When MobileMessageManager::GetMessages(),
   - The objects mentioned above will be created.
   - When DOMCursor::Continue() is invoked by SMS App, 
     a. MobileMessageCursorChild::HandleContinue() will send the Continue()
        request to parent process.
     b. When result is retrieved, MobileMessageCursorChild::DoNotifyResult
        will notify the result back to SMS App by 
        MobileMessageCursorCallback::NotifyCursorResult().

Comment 11

[Wei Gao (Spreadtrum)]

(In reply to Bevis Tseng[:bevistseng][:btseng] from comment #10)

Dear Bevis

Thanks for your debugging patch and detailed explanation.
I am very willing to have a build with these debugging info for further analysis.
I will also carefully study your summary of how these objects are used. That's too helpful for me.
Keep my feedback flags.
Many thanks.

Comment 12

[Wei Gao (Spreadtrum)]

I will reply to feedback as soon as possible.

Comment 13

[Wei Gao (Spreadtrum)]

Dear Bevis

I have modified and merged into our local code from your patch line by line. 
But your patch had conflict when applied into 2.1s. Maybe you are looking the code 2.2? I am uncertain of it.
I will ask our QA to try to reproduce this issue with your patch.
Thanks.

Comment 14

[Bevis Tseng[:bevis][:btseng](Exited)]

(In reply to Wei Gao (Spreadtrum) from comment #13)
> Dear Bevis
> 
> I have modified and merged into our local code from your patch line by line. 
> But your patch had conflict when applied into 2.1s. Maybe you are looking
> the code 2.2? I am uncertain of it.
> I will ask our QA to try to reproduce this issue with your patch.
> Thanks.

Yes, the patch is based on mozilla-central but there is no difference to 2.1s for querying threads/messages.

Comment 15

[Wei Gao (Spreadtrum)]

Ok, I get it.
Thanks very much.

Comment 16

[Wei Gao (Spreadtrum)]

Hi Bevis

We reproduced this crash issue yesterday.
The attachment is log.

ftp://ftp.spreadtrum.com/7715/1144016/1144016.tar.bz2

username:mouzhi
password:mouZHI$$61

Please help to check it.
Thanks so much.

Comment 17

[Wei Gao (Spreadtrum)]

Sorry, the log name in Comment16 is mistake.

The real log is:
ftp://ftp.spreadtrum.com/7715/1144016/sms_crash_log_3.14.tar.bz2

Backtrace:
ftp://ftp.spreadtrum.com/7715/1144016/sms_crash_backtrace_3.14.parse

Thanks.

Comment 18

[Bevis Tseng[:bevis][:btseng](Exited)]

Hi Wei Gao,

Unfortunately,
After comparing the timetamp of the main log (0-main-17-28-46.log) & sms_crash_backtrace_3.14.parse,
I found that 
1. the main log was started at 03-24 17:28:45 and the timestamp of the crash is at 24 14:22:41 which is quite earlier then the one in main log and I found that all the requests are done correctly in the main log.
2. the process id of sms app is quite large '16889', that implies this is not the first process the sms app was launched at. (previous one might be crashed.)

Would you please help to have the main log that captures the crash for further analysis?

Thanks!

Comment 19

[Wei Gao (Spreadtrum)]

(In reply to Bevis Tseng[:bevistseng][:btseng] from comment #18)
> Would you please help to have the main log that captures the crash for
> further analysis?
> 
> Thanks!

Yes, I will go on to have a try.
Thanks for your help.

Comment 20

[Wei Gao (Spreadtrum)]

Keep on reproducing...

Comment 21

[Wei Gao (Spreadtrum)]

Hi Bevis

This issue was reproduced yesterday.
Could you help to check this log?

The log is:
ftp://ftp.spreadtrum.com/7715/1144016/1144016_4.22.tar.bz2

username:mouzhi
password:mouZHI$$61

Thanks so much.

Comment 22

[Bevis Tseng[:bevis][:btseng](Exited)]

Attachment: [v2.1] Prevent Further Notification from MobileMessageCursorCallback if Parent window is detached.

Hi Wei Gao,

Thanks for providing the log.
I've found something suspicious and the problem might be similar to Bug 1119608.
The error might be happened at [1].

Would you please apply this patch and give it another trial?

[1]https://dxr.mozilla.org/mozilla-central/source/dom/mobilemessage/MobileMessageCursorCallback.cpp#84

--
04-20 23:45:14.600  2124  2124 I Gecko   : BVS, MobileMessageCursorCallback::NotifyCursorResult() start, mObjectId: 50
04-20 23:45:14.600  2124  2124 I Gecko   : BVS, MobileMessageCursorCallback::NotifyCursorError() from NotifyCursorResult()
04-20 23:45:14.600  2124  2124 I Gecko   : BVS, MobileMessageCursorCallback::NotifyCursorError(), mObjectId: 50
04-20 23:45:14.600  2124  2124 I Gecko   : BVS, MobileMessageCursorCallback::NotifyCursorResult() end, mObjectId: 50

Comment 23

[Wei Gao (Spreadtrum)]

(In reply to Bevis Tseng[:bevistseng][:btseng] from comment #22)
> Created attachment 8596428 [details] [diff] [review]
> [v2.1] Prevent Further Notification from MobileMessageCursorCallback if
> Parent window is detached.
> 
> Hi Wei Gao,
> 
> Thanks for providing the log.
> I've found something suspicious and the problem might be similar to Bug
> 1119608.
> The error might be happened at [1].
> 
> Would you please apply this patch and give it another trial?

Yes, I am glad to do that.
Thank you for your great help.
Keep my feedback flag.

Comment 24

[Wei Gao (Spreadtrum)]

Comment on attachment 8596428 [details] [diff] [review]
[v2.1] Prevent Further Notification from MobileMessageCursorCallback if Parent window is detached.

Review of attachment 8596428 [details] [diff] [review]:
-----------------------------------------------------------------

Dear Bevis

I think the patch can resolve this problem.
This issue is no longer be reproduced until now.
Thanks so much.

Comment 25

[BMO Automation]

Firefox OS is not being worked on