1148324 - b2g main thread is blocked on boot in nexus-5-l with mr1

Status: RESOLVED DUPLICATE of bug 498166

(Firefox OS Graveyard :: General, defect)

Description

[Kai-Zhen Li [:seinlin][:kli]]

(gdb) info threads
  Id   Target Id         Frame 
* 1    Thread 889.889    syscall () at bionic/libc/arch-arm/bionic/syscall.S:44
(gdb) where
#0  syscall () at bionic/libc/arch-arm/bionic/syscall.S:44
#1  0xb6d7641e in __futex (timeout=0x0, value=2, op=128, ftx=0xb6f7a498 <init_lock>) at bionic/libc/private/bionic_futex.h:45
#2  __futex_wait_ex (ftx=ftx@entry=0xb6f7a498 <init_lock>, shared=shared@entry=false, value=value@entry=2, timeout=0x0)
    at bionic/libc/private/bionic_futex.h:66
#3  0xb6d767b2 in _normal_lock (shared=0, mutex=0xb6f7a498 <init_lock>) at bionic/libc/bionic/pthread_mutex.cpp:337
#4  pthread_mutex_lock (mutex=0xb6f7a498 <init_lock>) at bionic/libc/bionic/pthread_mutex.cpp:457

#5  0xb6f52f40 in malloc_mutex_lock (mutex=0xb6f7a498 <init_lock>) at ../../../gecko/memory/mozjemalloc/jemalloc.c:1669
#6  malloc_init_hard () at ../../../gecko/memory/mozjemalloc/jemalloc.c:5552
#7  0xb6f53360 in malloc_init () at ../../../gecko/memory/mozjemalloc/jemalloc.c:5528
#8  0xb6f537b4 in je_malloc (size=20) at ../../../gecko/memory/mozjemalloc/jemalloc.c:6146
#9  0xb6d756fe in pthread_atfork (prepare=0xb6f51361 <_malloc_prefork>, parent=0xb6f513c5 <_malloc_postfork>, 
    child=0xb6f513c5 <_malloc_postfork>) at bionic/libc/bionic/pthread_atfork.cpp:89
#10 0xb6f53106 in malloc_init_hard () at ../../../gecko/memory/mozjemalloc/jemalloc.c:5843
#11 0xb6f53360 in malloc_init () at ../../../gecko/memory/mozjemalloc/jemalloc.c:5528
#12 0xb6f537b4 in je_malloc (size=1024) at ../../../gecko/memory/mozjemalloc/jemalloc.c:6146

#13 0xb6d92bfe in __smakebuf (fp=fp@entry=0xb6dd1ca4 <usual>) at bionic/libc/upstream-openbsd/lib/libc/stdio/makebuf.c:62
#14 0xb6d93026 in __srefill (fp=fp@entry=0xb6dd1ca4 <usual>) at bionic/libc/upstream-openbsd/lib/libc/stdio/refill.c:98
#15 0xb6d91dc8 in fgets (buf=buf@entry=0xbef1954c "", n=255, n@entry=256, fp=fp@entry=0xb6dd1ca4 <usual>)
    at bionic/libc/upstream-openbsd/lib/libc/stdio/fgets.c:66
#16 0xb6d77c3e in fgets (stream=0xb6dd1ca4 <usual>, size=256, dest=0xbef1954c "") at bionic/libc/include/stdio.h:427
#17 __sysconf_nprocessors_onln () at bionic/libc/bionic/sysconf.cpp:105
#18 0xb6d77dca in sysconf (name=name@entry=97) at bionic/libc/bionic/sysconf.cpp:339
#19 0xb6da91d0 in malloc_ncpus () at external/jemalloc/src/jemalloc.c:258
#20 malloc_init_hard () at external/jemalloc/src/jemalloc.c:797
#21 0xb6d71408 in malloc_init () at external/jemalloc/src/jemalloc.c:294
#22 jemalloc_constructor () at external/jemalloc/src/jemalloc.c:1898
#23 0xb6f815b6 in soinfo::CallFunction (this=0xb6dee2a4, function_name=0xb6f8a2cf "function", function=0xb6d713e9 <jemalloc_constructor>)
    at bionic/linker/linker.cpp:1642
#24 0xb6f8168c in soinfo::CallArray (this=0xb6dee2a4, array_name=0xb6f8a396 "DT_INIT_ARRAY", functions=<optimized out>, 
    count=<optimized out>, reverse=false) at bionic/linker/linker.cpp:1630
#25 0xb6f81862 in soinfo::CallConstructors (this=0xb6dee2a4) at bionic/linker/linker.cpp:1687
#26 0xb6f81820 in operator() (__closure=<optimized out>, si=<optimized out>) at bionic/linker/linker.cpp:1680
#27 operator() (__closure=<optimized out>, si=<optimized out>) at bionic/linker/linked_list.h:91
#28 visit<LinkedList<T, Allocator>::for_each(F) [with F = soinfo::CallConstructors()::__lambda8; T = soinfo; Allocator = SoinfoListAllocator]::__lambda0> (this=<optimized out>, action=...) at bionic/linker/linked_list.h:99
#29 for_each<soinfo::CallConstructors()::__lambda8> (action=..., this=<optimized out>) at bionic/linker/linked_list.h:90
#30 soinfo::CallConstructors (this=0xb6f7d2a4) at bionic/linker/linker.cpp:1681
#31 0xb6f81820 in operator() (__closure=<optimized out>, si=<optimized out>) at bionic/linker/linker.cpp:1680
---Type <return> to continue, or q <return> to quit---
#32 operator() (__closure=<optimized out>, si=<optimized out>) at bionic/linker/linked_list.h:91
#33 visit<LinkedList<T, Allocator>::for_each(F) [with F = soinfo::CallConstructors()::__lambda8; T = soinfo; Allocator = SoinfoListAllocator]::__lambda0> (this=<optimized out>, action=...) at bionic/linker/linked_list.h:99
#34 for_each<soinfo::CallConstructors()::__lambda8> (action=..., this=<optimized out>) at bionic/linker/linked_list.h:90
#35 soinfo::CallConstructors (this=0xb6f7d154) at bionic/linker/linker.cpp:1681
#36 0xb6f81820 in operator() (__closure=<optimized out>, si=<optimized out>) at bionic/linker/linker.cpp:1680
#37 operator() (__closure=<optimized out>, si=<optimized out>) at bionic/linker/linked_list.h:91
#38 visit<LinkedList<T, Allocator>::for_each(F) [with F = soinfo::CallConstructors()::__lambda8; T = soinfo; Allocator = SoinfoListAllocator]::__lambda0> (this=<optimized out>, action=...) at bionic/linker/linked_list.h:99
#39 for_each<soinfo::CallConstructors()::__lambda8> (action=..., this=<optimized out>) at bionic/linker/linked_list.h:90
#40 soinfo::CallConstructors (this=0xb6f7d004) at bionic/linker/linker.cpp:1681
#41 0xb6f83aea in __linker_init_post_relocation (linker_base=<optimized out>, args=...) at bionic/linker/linker.cpp:2465
#42 __linker_init (raw_args=<optimized out>) at bionic/linker/linker.cpp:2600
#43 0xb6f80a9c in __dl__start () at bionic/linker/arch/arm/begin.S:33
#44 0xb6f80a9c in __dl__start () at bionic/linker/arch/arm/begin.S:33

Comment 1

[Kai-Zhen Li [:seinlin][:kli]]

Attachment: bug-1148324-disable-moz_memory.patch

Mike, I think we need to disable MOZ_MEMORY on B2G build after lollipop release as jemalloc is supported in bionic. Would you mind review this patch? Thanks!

Comment 2

[Mike Hommey [:glandium]]

Comment on attachment 8585875 [details] [diff] [review]
bug-1148324-disable-moz_memory.patch

Review of attachment 8585875 [details] [diff] [review]:
-----------------------------------------------------------------

::: configure.in
@@ +3983,5 @@
>          MOZ_THEME_FASTSTRIPE=1
>          MOZ_TREE_FREETYPE=1
> +        if test -n "$MOZ_MEMORY"; then
> +          MOZ_MEMORY=1
> +        fi

This patch disables jemalloc on android.

Now, with that being said, there are several things to consider:
- From a cursory look at the bionic repository, bionic using jemalloc doesn't only depend on the version of bionic, but also on how bionic is being built, which means the choice must be made according to that.
- DMD requires replace-malloc, which requires MOZ_MEMORY. Disabling MOZ_MEMORY means losing DMD support. That can be changed, but this has to be considered.
- for about:memory, it would be better to use the MOZ_NATIVE_JEMALLOC path, which requires unsetting MOZ_MEMORY, setting MOZ_JEMALLOC3 and having libc expose mallctl and nallocx (does bionic built with jemalloc do that?)

Comment 3

[Cervantes Yu [:cyu] [:cervantes]]

It looks like pthread_atfork() is causing the deadlock. I think BionicGlue.cpp has one wrapper that is used to avoid this deadlock. I guess removing the 'ANDROID_VERSION < 1' check fixes this problem.

Comment 4

[Kai-Zhen Li [:seinlin][:kli]]

I try to use BionicGlue.cpp's one wrapper and hit another error. B2G will not boot up correctly. When I try to start b2g in adb shell, I can see the following error and b2g is not started.

---------

[14018] WARNING: Tried to RegisterCallback without an AtExitManager: file ../../../gecko/ipc/chromium/src/base/at_exit.cc, line 40
*****************************************************************
***
*** This is stdout. Most of the useful output will be in logcat.
***
*****************************************************************
1427878962878	Marionette	INFO	marionette enabled via build flag and pref
1427878963114	Marionette	INFO	marionette-server.js loaded
1427878963225	Marionette	INFO	B2G emulator: no
1427878963226	Marionette	INFO	Device detected is hammerhead
1427878963227	Marionette	INFO	Listening on port 2828

1427878963228	Marionette	INFO	Marionette server ready
MobileConnectionService: init complete
-*- SettingsService: Lock finalize failed!
-*- SettingsRequestManager: Transaction for lock {01c11d1c-ff84-47f7-a639-6b184c7574d6} aborted
Activities didn't register correctly!Crash reporter : exception: [Exception... "Component returned failure code: 0xc1f30001 (NS_ERROR_NOT_INITIALIZED) [nsICrashReporter.annotateCrashReport]"  nsresult: "0xc1f30001 (NS_ERROR_NOT_INITIALIZED)"  location: "JS frame :: chrome://b2g/content/shell.js :: shell_start/< :: line 272"  data: no]###################################### forms.js loaded
############################### browserElementPanning.js loaded
######################## BrowserElementChildPreload.js loaded
*** UTM:SVC TimerManager:registerTimer - id: user-agent-updates-timer
############################### browserElementPanning.js loaded
######################## BrowserElementChildPreload.js loaded
XXX FIXME : Got a mozContentEvent: system-message-listener-ready
11|root@hammerhead:/ # [Child 14018] ###!!! ABORT: constructor for actor failed: file PContentChild.cpp, line 1423

Comment 5

[Michael Wu [:mwu]]

(In reply to Mike Hommey [:glandium] from comment #2)
> - From a cursory look at the bionic repository, bionic using jemalloc
> doesn't only depend on the version of bionic, but also on how bionic is
> being built, which means the choice must be made according to that.

It's currently being built on the Nexus 5 with jemalloc, and we can just require bionic to be built with jemalloc.

> - DMD requires replace-malloc, which requires MOZ_MEMORY. Disabling
> MOZ_MEMORY means losing DMD support. That can be changed, but this has to be
> considered.

We can disable MOZ_MEMORY by default but have it turn back on if DMD is requested. Not sure if that will deadlock (depending on whether mozjemalloc's symbols get renamed or not), but that can be worked around if it does.

> - for about:memory, it would be better to use the MOZ_NATIVE_JEMALLOC path,
> which requires unsetting MOZ_MEMORY, setting MOZ_JEMALLOC3 and having libc
> expose mallctl and nallocx (does bionic built with jemalloc do that?)

If it doesn't, we can patch bionic to expose necessary functions.

In general, I'd like to avoid having multiple allocators if we can.

Comment 6

[Kai-Zhen Li [:seinlin][:kli]]

(In reply to Kai-Zhen Li [:kli][:seinlin] from comment #4)
> *** UTM:SVC TimerManager:registerTimer - id: user-agent-updates-timer
> ############################### browserElementPanning.js loaded
> ######################## BrowserElementChildPreload.js loaded
> XXX FIXME : Got a mozContentEvent: system-message-listener-ready
> 11|root@hammerhead:/ # [Child 14018] ###!!! ABORT: constructor for actor
> failed: file PContentChild.cpp, line 1423

When I try flash system.img instead of flash gecko only, this abort issue was gone. By using BionicGlue.cpp's wrapper b2g can boot up correctly.

Comment 7

[Kai-Zhen Li [:seinlin][:kli]]

Now I am verifying unsetting MOZ_MEMORY and setting MOZ_JEMALLOC3. I will update the result on bug when I get it.

Comment 8

[Kai-Zhen Li [:seinlin][:kli]]

After unset MOZ_MEMORY and set MOZ_JEMALLOC3, b2g can boot up correctly. When I try get-about-memory, I can see this warning.

WARNING: the 'heap-allocated' memory reporter does not work for this platform and/or configuration. This means that 'heap-unclassified' is not shown and the 'explicit' tree shows less memory than it should.

Comment 9

[Kai-Zhen Li [:seinlin][:kli]]

Attachment: bug-1148324-disable-moz_memory.patch

Mike, I updated the patch only disable MOZ_MEMORY in b2g lollipop build. As comment 8, when MOZ_MEMORY is disabled 'heap-allocated' memory reporter does not work. But due to the schedule can we disable MOZ_MEMORY first and then file a new bug to fix memory reporter later?

Comment 10

[Kai-Zhen Li [:seinlin][:kli]]

Comment on attachment 8588906 [details] [diff] [review]
bug-1148324-disable-moz_memory.patch

Mike, As per offline discussion, I'll request for review when I have a new patch. Thanks for you suggestion.

Comment 11

[Kai-Zhen Li [:seinlin][:kli]]

I'll file a follow bug to fix error in comment 8 after bug 1151787 is resolved fixed. Before bug 1151787 is resolved there is no issue.

Comment 12

[Mike Hommey [:glandium]]

It turns out this is a dupe of bug 498166, which was fixed 3 months ago. Bug 1151787 can thus be backed out.