862097 - [Inari] crash in mozilla::layers::ShadowLayersChild::ActorDestroy with abort message: "ActorDestroy by IPC channel failure at ShadowLayersChild: file ../../../gecko/gfx/layers/ipc/ShadowLayersChild.cpp, line 68"

Status: RESOLVED FIXED

(Firefox OS Graveyard :: General, defect)

Description

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

This bug was filed from the Socorro interface and is 
report bp-0f606005-03dd-44c2-a19a-739bd2130415 .
============================================================= 
Frame 	Module 	Signature 	Source
0 	libxul.so 	mozalloc_abort 	mozalloc_abort.cpp:30
1 	libxul.so 	NS_DebugBreak_P 	nsDebugImpl.cpp:423
2 	libxul.so 	mozilla::layers::ShadowLayersChild::ActorDestroy 	ShadowLayersChild.cpp:68
3 	libxul.so 	mozilla::layers::PLayersChild::DestroySubtree 	PLayersChild.cpp:673
4 	libxul.so 	mozilla::layers::PCompositorChild::DestroySubtree 	PCompositorChild.cpp:886
5 	libxul.so 	mozilla::layers::PCompositorChild::OnChannelError 	PCompositorChild.cpp:775
6 	libxul.so 	mozilla::ipc::AsyncChannel::NotifyMaybeChannelError 	AsyncChannel.cpp:549
7 	libxul.so 	mozilla::ipc::AsyncChannel::OnNotifyMaybeChannelError 	AsyncChannel.cpp:514
8 	libxul.so 	RunnableMethod<IPC::ChannelProxy::Context, void , Tuple0>::Run 	tuple.h:383
9 	libxul.so 	MessageLoop::RunTask 	message_loop.cc:334
10 	libxul.so 	MessageLoop::DeferOrRunPendingTask 	message_loop.cc:342
11 	libxul.so 	MessageLoop::DoWork 	message_loop.cc:442
12 	libxul.so 	mozilla::ipc::DoWorkRunnable::Run 	MessagePump.cpp:42
13 	libxul.so 	nsThread::ProcessNextEvent 	nsThread.cpp:620
14 	libxul.so 	NS_ProcessNextEvent_P 	nsThreadUtils.cpp:237
15 	libxul.so 	mozilla::ipc::MessagePump::Run 	MessagePump.cpp:82
16 	libxul.so 	mozilla::ipc::MessagePumpForChildProcess::Run 	MessagePump.cpp:231
17 	libxul.so 	MessageLoop::RunInternal 	message_loop.cc:216
18 	libxul.so 	MessageLoop::Run 	message_loop.cc:209
19 	libxul.so 	nsBaseAppShell::Run 	nsBaseAppShell.cpp:163
20 	libxul.so 	XRE_RunAppShell 	nsEmbedFunctions.cpp:646
21 	libxul.so 	mozilla::ipc::MessagePumpForChildProcess::Run 	MessagePump.cpp:198
22 	libxul.so 	MessageLoop::RunInternal 	message_loop.cc:216
23 	libxul.so 	MessageLoop::Run 	message_loop.cc:209
24 	libxul.so 	XRE_InitChildProcess 	nsEmbedFunctions.cpp:485
25 	plugin-container 	main 	ipc/app/MozillaRuntimeMain.cpp:48
26 	libc.so 	__libc_init 	libc_init_dynamic.c:114
27 		@0xb0001dc5

More reports:
https://crash-stats.mozilla.com/report/list?signature=mozalloc_abort+|+NS_DebugBreak_P+|+mozilla%3A%3Alayers%3A%3AShadowLayersChild%3A%3AActorDestroy

STR: 
1. launch browser app
2. go to http://bit.ly/css3d-mol
3. select any of the links at the bottom of the page (caffiene, Salt, Buckyball)
4. repeat 3 several times.

Expected: no crash
Actual: browser content crash

Comment 1

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

Gecko  http://hg.mozilla.org/releases/mozilla-b2g18_v1_0_1/rev/2b44e2c40cc1
Gaia   c79e761bae4d92f329154c64159f4f5c8eb49c9e
BuildID 20130415070202
Version 18.0
Inari

Comment 2

[Sotaro Ikeda [:sotaro]]

This abort is come from Bug 860892.

Comment 3

[Sotaro Ikeda [:sotaro]]

> 
> More reports:
> https://crash-stats.mozilla.com/report/
> list?signature=mozalloc_abort+|+NS_DebugBreak_P+|+mozilla%3A%3Alayers%3A%3ASh
> adowLayersChild%3A%3AActorDestroy
> 
> STR: 
> 1. launch browser app
> 2. go to http://bit.ly/css3d-mol
> 3. select any of the links at the bottom of the page (caffiene, Salt,
> Buckyball)
> 4. repeat 3 several times.
> 
> Expected: no crash
> Actual: browser content crash

This might be dupe of Bug 851664. The STR is what I wanted! I am going to check it tomorrow.

Comment 4

[Sotaro Ikeda [:sotaro]]

The crash happens by hit rlimit of file descriptor in b2g process. I confirmed it from adb shell logcat on unagi.

Comment 5

[Sotaro Ikeda [:sotaro]]

Attachment: adb logcat of the crash

the log says, b2g process is get kill by genlock_create_lock failure. And then child process recognize, IPC connection is disconnected.

Comment 6

[Sotaro Ikeda [:sotaro]]

On custom ROM that increase file descriptor's rlimit to 4048. The crash does not happen. The page is rendered correctly on the STR.

Comment 7

[Sotaro Ikeda [:sotaro]]

(In reply to Sotaro Ikeda [:sotaro] from comment #6)
> On custom ROM that increase file descriptor's rlimit to 4048. The crash does
> not happen. The page is rendered correctly on the STR.

Curent Firefox OS architecture requires a lot of file descriptor's for gralloc buffers. The buffer is used for Layer's buffers. And all the buffers are delivered to b2g process for rendering. If a web page has a lot of layers, it allocate a lot of gralloc buffers. And there is no way to limit the number.

Comment 8

[Sotaro Ikeda [:sotaro]]

(In reply to Sotaro Ikeda [:sotaro] from comment #6)
> On custom ROM that increase file descriptor's rlimit to 4048. The crash does
> not happen. The page is rendered correctly on the STR.

By using the following command, a number of file descriptor in b2g process increase to about 2200.

Comment 9

[Alex Keybl [:akeybl]]

Before considering upping the number of file descriptors, can you help us understand how the current limit was chosen? We don't want to introduce new bugs, if this bug is already difficult to run into.

Comment 10

[Sotaro Ikeda [:sotaro]]

Current file descriptor's rlimit on v1.01 is linux's default value.
  - soft limit: 1024
  - hard limit: 4096

Comment 11

[Andrew Overholt [:overholt]]

This blocks bug 851664 so should be tef+.  If we fix this by increasing the limits (and fix one other yet-to-be-filed bug), it will likely fix bug 851664.

Comment 12

[Sotaro Ikeda [:sotaro]]

On v1.1, Current file descriptor's rlimit is changed to following in Bug 853977.
  - soft limit: 2048
  - hard limit: 4096

From v1.1, android's media server is used to handle codec and camera hw in it.

Comment 13

[Sotaro Ikeda [:sotaro]]

(In reply to Sotaro Ikeda [:sotaro] from comment #12)
> 
> From v1.1, android's media server is used to handle codec and camera hw in
> it.

android's media server needs more file descriptors than 1024.

Comment 15

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

It took longer, but I was still able to reproduce the issue on master.  I ended up tapping on at least 15 different molecular structures on that web site and placing into landscape view.

Build:2013-04-16-03-10-11
"mozilla-central" revision="1d9c510b3742"
"integration/gaia-central" revision="972f3504af31"
"gecko.git" revision="95bf443385bbd8ee611be1586ec975e609acd006"
"gaia.git" revision="d10e892eb6887967803a68e8c441a7c456446437"
Unagi

I think increasing will delay the crashing; I do not believe that it will stop the crashing...

Comment 16

[Sotaro Ikeda [:sotaro]]

(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #15)
> I think increasing will delay the crashing; I do not believe that it will
> stop the crashing...

How much do you increase the rlimit?

Anyway Bug 862397 fix the crash. Qcom already fixed the crash in recent source. Just apply it to our source.

Comment 17

[Naoki Hirata :nhirata (please use needinfo instead of cc)]

\o/

Once I get a build with the fix, I will try looking at it again.  If you have a build already made with the patch, I will retest with it.

Comment 18

[Sotaro Ikeda [:sotaro]]

(In reply to Naoki Hirata :nhirata (please use needinfo instead of cc) from comment #17)
> \o/
> 
> Once I get a build with the fix, I will try looking at it again.  If you
> have a build already made with the patch, I will retest with it.

I already confirmed the path works locally.

Comment 19

[Sotaro Ikeda [:sotaro]]

Attachment: patch - set RLIMIT_NOFILE of b2g/b2g-container processes to 4096

Confirmed that the patch fix the crash on unagi.

Comment 20

[Sotaro Ikeda [:sotaro]]

Attachment: patch v2 - set RLIMIT_NOFILE of b2g/content processes to 4096

update comment.

Comment 21

[Michael Wu [:mwu]]

Comment on attachment 738696 [details] [diff] [review]
patch v2 - set RLIMIT_NOFILE of b2g/content processes to 4096

I don't think the comments are necessary here. Looks good otherwise.

Comment 22

[Michael Wu [:mwu]]

Merged in https://github.com/mozilla-b2g/gonk-misc/pull/82 . Uplift to follow.

Comment 23

[Michael Wu [:mwu]]

Uplifted to v1-train and v1.0.1.

https://github.com/mozilla-b2g/gonk-misc/commit/73bd5f975f28fb696f3e647c50ac4fe03464d582
https://github.com/mozilla-b2g/gonk-misc/commit/16be480e3254d5dcbaf13807017c125b5a482fb3

Comment 24

[Sotaro Ikeda [:sotaro]]

need to update also b2g-manifest of v1-train and v1.0.1

Comment 25

[Sotaro Ikeda [:sotaro]]

jhford, can you upadte the b2g-manifest of v1-train and v1.0.1 like following?
 - update all devices' gonk-misc revision to most recent one that already have specific revision.

Comment 26

[John Ford [:jhford] CET/CEST Berlin Time]

To github.com:mozilla-b2g/b2g-manifest.git
   8589ac6..9635c27  v1-train -> v1-train
   88be926..bc902bf  v1.0.1 -> v1.0.1

Comment 27

[Sotaro Ikeda [:sotaro]]

On buri device partner build, I found a problem of showing http://bit.ly/css3d-mol. It is a different problem.

it seems that there are no fallback to ashmem. Though it seems that it badly affect to http://bit.ly/css3d-mol in Bug 862097. When doing STR in Bug 862097, "Well, this is embarassing. We tried to display this Web page, but it's not responding" is shown.

Comment 28

[Cody Roesch [:croesch]]

Inari Build: 20130429070204
Gecko  http://hg.mozilla.org/releases/mozilla-b2g18_v1_0_1/rev/45aa5ba0ed53
Gaia   cf2d4136f0ebc66039637fdbeb72ed184dfbc0f2
Kernel:  Feb 21st

On the build above for the Inari device, I was able to switch between the three links for (Salt, Caffine and buckyball) just fine with no problems seen. However going to other links not mentioned in the bug (Graphite, YBCO) did cause the browser to crash. This was not the steps in the bug so not sure if this is a different issue and warrants a different bug.

Leo Build: 20130426070204
Gecko  http://hg.mozilla.org/releases/mozilla-b2g18/rev/6c2493de1441
Gaia   c9046a7acef33328977840176ff5574720d2c74c
Kernel: March 15th

On the Leo build above, when tapping on the Salt link, the browser went to a white screen and the device could not be recovered. Had to pull the battery to get the device to recover. Once the device rebooted, I was presented with a Just Crashed message which I sent on to Mozilla.

Let me know if you need anything more from me here. Logs or screenshots or bugs.

Comment 29

[Deepa Subramanian]

On Inari, When tried tap on any molecules in the last row (Buckyball, graphite, YBCO superconductor or salt) the browser either crashes with a black screen (3/5 times)/screen  with selected molecules are highlighted and 3D structures are not seen. 

Inari Build ID: 20130515070203
Gecko  http://hg.mozilla.org/releases/mozilla-b2g18_v1_0_1/rev/0b6bcb1f4175
Gaia   9648799c2e45917ff150fa9eef8aeac79a9ac008

On leo, Tapping on any of the molecules does not bring up its 3D forms (2/5 times)/ crash report prompted to be submitted.
Leo Build ID: 20130515070208  
Gecko  http://hg.mozilla.org/releases/mozilla-b2g18/rev/d06cfe7d67c2
Gaia   0ddb515f15cbc6b74fc2742b7599d6ae74c6413f

Comment 30

[Sotaro Ikeda [:sotaro]]

Deepa, can you create a new bug for it? And can you attach adb log?

Comment 31

[Deepa Subramanian]

Sotaro Ikeda:

Please see the bug created https://bugzilla.mozilla.org/show_bug.cgi?id=877495