959254 - Crash loading nss or sqlite libs while running tests

Reporter

Description

•

10 years ago

Consolidating my earlier reports from a few different bugs....

We are seeing startup crashes during various robocop tests (and perhaps mochitests also). Logcats vary, but often show trouble during nss load:

01-02 03:59:35.900 E/GeckoJNI( 2237): missing NSS_Initialize
01-02 03:59:35.900 E/GeckoLibLoad( 2237): Throw
01-02 03:59:35.900 E/GeckoLibLoad( 2237): Load nss done
01-02 03:59:35.900 W/dalvikvm( 2237): threadid=12: thread exiting with uncaught exception (group=0x4001d848)
01-02 03:59:35.900 E/GeckoAppShell( 2237): >>> REPORTING UNCAUGHT EXCEPTION FROM THREAD 14 ("Gecko")
01-02 03:59:35.900 E/GeckoAppShell( 2237): java.lang.Exception: Error loading nss libraries
01-02 03:59:35.900 E/GeckoAppShell( 2237): 	at org.mozilla.gecko.mozglue.GeckoLoader.loadNSSLibsNative(Native Method)
01-02 03:59:35.900 E/GeckoAppShell( 2237): 	at org.mozilla.gecko.mozglue.GeckoLoader.loadNSSLibs(GeckoLoader.java:236)
01-02 03:59:35.900 E/GeckoAppShell( 2237): 	at org.mozilla.gecko.GeckoThread.initGeckoEnvironment(GeckoThread.java:116)
01-02 03:59:35.900 E/GeckoAppShell( 2237): 	at 


14:31:17     INFO -  01-07 14:30:28.585 W/NetworkManagementSocketTagger( 1403): setKernelCountSet(10043, 0) failed with errno -2
14:31:17     INFO -  01-07 14:30:28.609 E/GeckoLibLoad( 2147): Load nss start
14:31:17     INFO -  01-07 14:30:28.617 E/GeckoLibLoad( 2147): Load sqlite start
14:31:17     INFO -  01-07 14:30:28.617 F/libc    ( 2147): Fatal signal 8 (SIGFPE) at 0x00000863 (code=-6)
14:31:17     INFO -  01-07 14:30:28.617 E/GeckoLinker( 2147): /data/app/org.mozilla.fennec-1.apk!/assets/libnss3.so: Warning: unhandled flags #8 not handled
14:31:17     INFO -  01-07 14:30:28.625 E/GeckoLinker( 2147): /data/app/org.mozilla.fennec-1.apk!/assets/libnss3.so: Warning: relocation to NULL @0x0018dc24
14:31:17     INFO -  01-07 14:30:28.625 E/GeckoLinker( 2147): /data/app/org.mozilla.fennec-1.apk!/assets/libnss3.so: Warning: relocation to NULL @0x0018dfdc for symbol "__cxa_begin_cleanup"
14:31:17     INFO -  01-07 14:30:28.625 E/GeckoLinker( 2147): /data/app/org.mozilla.fennec-1.apk!/assets/libnss3.so: Warning: relocation to NULL @0x0018dfe0 for symbol "__cxa_type_match"
14:31:17     INFO -  01-07 14:30:28.632 E/GeckoLibLoad( 2147): Load nss done


14:04:05  WARNING -  PROCESS-CRASH | testFindInPage | application crashed [@ CustomElf::GetSymbol(char const*, unsigned long) const]
14:04:05     INFO -  Crash dump filename: /tmp/tmpPe2qto/6633cf0e-4cb6-791d-65fcff0c-325f2ea6.dmp
14:04:05     INFO -  Operating system: Android
14:04:05     INFO -                    0.0.0 Linux 3.2.0+ #2 SMP PREEMPT Thu Nov 29 08:06:57 EST 2012 armv7l pandaboard/pandaboard/pandaboard:4.0.4/IMM76I/5:eng/test-keys
14:04:05     INFO -  CPU: arm
14:04:05     INFO -       2 CPUs
14:04:05     INFO -  Crash reason:  SIGSEGV
14:04:05     INFO -  Crash address: 0x0
14:04:05     INFO -  Thread 10 (crashed)
14:04:05     INFO -   0  libmozglue.so!CustomElf::GetSymbol(char const*, unsigned long) const [CustomElf.cpp:b35dc4d2e1b9 : 370 + 0x2]
14:04:05     INFO -       r4 = 0x5bf20110    r5 = 0x5bf20110    r6 = 0x5bd79d44    r7 = 0x5bd73538
14:04:05     INFO -       r8 = 0x5bd73538    r9 = 0x50b8fe00   r10 = 0x409e1f44    fp = 0x5bd30c04
14:04:05     INFO -       sp = 0x5bd30b58    lr = 0x5bd6cc63    pc = 0x5bd54816
14:04:05     INFO -      Found by: given as instruction pointer in context
14:04:05     INFO -   1  libmozglue.so!CustomElf::GetSymbolPtr(char const*, unsigned long) const [CustomElf.cpp:b35dc4d2e1b9 : 277 + 0x3]
14:04:05     INFO -       r4 = 0x5bf20110    r5 = 0x5bf20110    r6 = 0x5bd79d44    r7 = 0x5bd73538
14:04:05     INFO -       r8 = 0x5bd30bf0    r9 = 0x50b8fe00   r10 = 0x409e1f44    fp = 0x5bd30c04
14:04:05     INFO -       sp = 0x5bd30b70    pc = 0x5bd54855
14:04:05     INFO -      Found by: call frame info
14:04:05     INFO -   2  libmozglue.so!__wrap_dlsym [ElfLoader.cpp:b35dc4d2e1b9 : 73 + 0x5]
14:04:05     INFO -       r4 = 0x5bf20110    r5 = 0x5bf20110    r6 = 0x5bf20110    r7 = 0x5bd73538
14:04:05     INFO -       r8 = 0x5bd30bf0    r9 = 0x50b8fe00   r10 = 0x409e1f44    fp = 0x5bd30c04
14:04:05     INFO -       sp = 0x5bd30b98    pc = 0x5bd55c41
14:04:05     INFO -      Found by: call frame info
14:04:05     INFO -   3  libmozglue.so!setup_nss_functions(void*, void*, void*) [NSSBridge.cpp:b35dc4d2e1b9 : 58 + 0x9]
14:04:05     INFO -       r4 = 0x5bf20110    r5 = 0x5bf20110    r6 = 0x5bf20110    r7 = 0x5bd73538
14:04:05     INFO -       r8 = 0x5bd30bf0    r9 = 0x50b8fe00   r10 = 0x409e1f44    fp = 0x5bd30c04
14:04:05     INFO -       sp = 0x5bd30ba0    pc = 0x5bd59e9d
14:04:05     INFO -      Found by: call frame info
14:04:05     INFO -   4  libmozglue.so!loadNSSLibs [APKOpen.cpp:b35dc4d2e1b9 : 299 + 0x7]
14:04:05     INFO -       r4 = 0x02084820    r5 = 0x5bf161c0    r6 = 0x24f00005    r7 = 0x50b8fe08
14:04:05     INFO -       r8 = 0x5bd30bf0    r9 = 0x50b8fe00   r10 = 0x409e1f44    fp = 0x5bd30c04
14:04:05     INFO -       sp = 0x5bd30bb8    pc = 0x5bd59555
14:04:05     INFO -      Found by: call frame info
14:04:05     INFO -   5  libmozglue.so!loadSQLiteLibs [APKOpen.cpp:b35dc4d2e1b9 : 223 + 0xb]
14:04:05     INFO -       r4 = 0x02084820    r5 = 0x5bd7a2f4    r6 = 0x24f00005    r7 = 0x50b8fe08
14:04:05     INFO -       r8 = 0x5bd30bf0    r9 = 0x50b8fe00   r10 = 0x409e1f44    fp = 0x5bd30c04
14:04:05     INFO -       sp = 0x5bd30bd0    pc = 0x5bd5958b
14:04:05     INFO -      Found by: call frame info

01-13 07:41:02.554 E/GeckoJNI( 1828): missing NSS_Initialize
01-13 07:41:02.554 E/GeckoLibLoad( 1828): Throw
01-13 07:41:02.554 E/GeckoLibLoad( 1828): Load nss done
01-13 07:41:02.554 W/dalvikvm( 1828): threadid=12: thread exiting with uncaught exception (group=0x4001d848)
01-13 07:41:02.564 E/GeckoAppShell( 1828): >>> REPORTING UNCAUGHT EXCEPTION FROM THREAD 14 ("Gecko")
01-13 07:41:02.564 E/GeckoAppShell( 1828): java.lang.Exception: Error loading nss libraries
01-13 07:41:02.564 E/GeckoAppShell( 1828): 	at org.mozilla.gecko.mozglue.GeckoLoader.loadNSSLibsNative(Native Method)
01-13 07:41:02.564 E/GeckoAppShell( 1828): 	at org.mozilla.gecko.mozglue.GeckoLoader.loadNSSLibs(GeckoLoader.java:236)
01-13 07:41:02.564 E/GeckoAppShell( 1828): 	at org.mozilla.gecko.GeckoThread.initGeckoEnvironment(GeckoThread.java:116)
01-13 07:41:02.564 E/GeckoAppShell( 1828): 	at org.mozilla.gecko.GeckoThread.run(GeckoThread.java:168)


See blocked bugs for complete logs.

Geoff Brown [:gbrown]

Reporter

Comment 1

•

10 years ago

:glandium -- Hope you can help here. Transferring my earlier need-info's here (it's all the same issue as far as I can tell).

Flags: needinfo?(mh+mozilla)

Mike Hommey [:glandium]

Assignee

Comment 2

•

10 years ago

(In reply to Geoff Brown [:gbrown] from comment #1)
> :glandium -- Hope you can help here. Transferring my earlier need-info's
> here (it's all the same issue as far as I can tell).

Bug 722166 has accumulated so many different kind of failures, that we should do something to stop filling it any further.

Not all of the remaining ones have the same crash characteristics. The weirdest one is the "mprotect failed" one. I'll attach a patch to make those messages more useful for debugging.

For the other ones, it would be helpful to have more debugging log. How feasible would it be to run tests with MOZ_DEBUG_LINKER set to 1?

Flags: needinfo?(mh+mozilla)

Mike Hommey [:glandium]

Assignee

Updated

•

10 years ago

Whiteboard: [leave open]

Mike Hommey [:glandium]

Assignee

Comment 3

•

10 years ago

Attached patch Make MappableSeekableZStream::ensure mprotect error message more useful for debugging failures — Details — Splinter Review

Attachment #8359612 - Flags: review?(nfroyd)

Mike Hommey [:glandium]

Assignee

Updated

•

10 years ago

Assignee: nobody → mh+mozilla

Status: NEW → ASSIGNED

Mike Hommey [:glandium]

Assignee

Updated

•

10 years ago

OS: Mac OS X → Android

Hardware: x86 → ARM

Nathan Froyd [:froydnj]

Updated

•

10 years ago

Attachment #8359612 - Flags: review?(nfroyd) → review+

Mike Hommey [:glandium]

Assignee

Comment 4

•

10 years ago

Comment on attachment 8359612 [details] [diff] [review]
Make MappableSeekableZStream::ensure mprotect error message more useful for debugging failures

https://hg.mozilla.org/integration/mozilla-inbound/rev/e955d3ecdbae

Attachment #8359612 - Flags: checkin+

Ed Morley [:emorley]

Comment 5

•

10 years ago

https://hg.mozilla.org/mozilla-central/rev/e955d3ecdbae

Geoff Brown [:gbrown]

Reporter

Comment 6

•

10 years ago

(In reply to Mike Hommey [:glandium] from comment #2)
> For the other ones, it would be helpful to have more debugging log. How
> feasible would it be to run tests with MOZ_DEBUG_LINKER set to 1?

I think it is easy, but I think we wouldn't want to have that defined for all of our tests all the time. I'll try to reproduce with a try run with MOZ_DEBUG_LINKER=1.

Geoff Brown [:gbrown]

Reporter

Comment 7

•

10 years ago

I messed up my first try run, https://tbpl.mozilla.org/?tree=Try&rev=1f46dafa68b4. Starting over...

Geoff Brown [:gbrown]

Reporter

Comment 8

•

10 years ago

https://tbpl.mozilla.org/?tree=Try&rev=4bf4952dab80 has MOZ_DEBUG_LINKER=1. Note that some green jobs seem to crash, such as the first Android 4.0 rc2, https://tbpl.mozilla.org/php/getParsedLog.php?id=33074301&tree=Try&full=1#error0.

Mike Hommey [:glandium]

Assignee

Comment 9

•

10 years ago

(In reply to Geoff Brown [:gbrown] from comment #8)
> https://tbpl.mozilla.org/?tree=Try&rev=4bf4952dab80 has MOZ_DEBUG_LINKER=1.
> Note that some green jobs seem to crash, such as the first Android 4.0 rc2,
> https://tbpl.mozilla.org/php/getParsedLog.
> php?id=33074301&tree=Try&full=1#error0.

19:37:40     INFO -  01-15 19:37:35.914 D/AndroidRuntime( 3853): Shutting down VM
19:37:40     INFO -  01-15 19:37:35.914 I/ActivityManager( 1400): Force stopping package org.mozilla.fennec uid=10046
19:37:40     INFO -  01-15 19:37:35.914 I/ActivityManager( 1400): Killing proc 3864:org.mozilla.fennec/10046: force stop

It's being killed by the activity manager. Debugging messages might make it too unresponsive... We'd need something preventing the activity manager from killing us.

Geoff Brown [:gbrown]

Reporter

Comment 10

•

10 years ago

I still see crashes on m-i, etc:

https://tbpl.mozilla.org/php/getParsedLog.php?id=33097789&tree=Mozilla-Inbound

but have had no luck reproducing with MOZ_DEBUG_LINKER: 

https://tbpl.mozilla.org/?tree=Try&rev=4bf4952dab80

Ryan VanderMeulen [:RyanVM]

Comment 11

•

10 years ago

Looks related:
https://tbpl.mozilla.org/php/getParsedLog.php?id=33177965&tree=Mozilla-Inbound

Geoff Brown [:gbrown]

Reporter

Updated

•

10 years ago

Updated

•

10 years ago

Blocks: 963100

Geoff Brown [:gbrown]

Reporter

Updated

•

10 years ago

Blocks: 966827

Geoff Brown [:gbrown]

Reporter

Updated

•

10 years ago

Updated

•

10 years ago

Blocks: 966788

Ryan VanderMeulen [:RyanVM]

Updated

•

10 years ago

Depends on: 970592

Geoff Brown [:gbrown]

Reporter

Updated

•

10 years ago

No longer blocks: 966788

Geoff Brown [:gbrown]

Reporter

Comment 12

•

10 years ago

I'm trying to reproduce a crash with MOZ_DEBUG_LINKER again today. Some of these have the most-frequently logged statements suppressed, to see if that helps:

https://tbpl.mozilla.org/?tree=Try&rev=ec17c9b06b95
https://tbpl.mozilla.org/?tree=Try&rev=e3117e611d44
https://tbpl.mozilla.org/?tree=Try&rev=1e46fd09f860

Geoff Brown [:gbrown]

Reporter

Comment 13

•

10 years ago

It's slow going, but I have 2 crashes so far...

https://tbpl.mozilla.org/php/getParsedLog.php?id=34882945&tree=Try&full=1#error0

18:58:05     INFO -  02-18 18:56:29.023 E/GeckoLinker( 2150): CustomElf::GetSymbolPtr(0x5bf20110 ["/data/app/org.mozilla.fennec-1.apk!/assets/libnss3.so"], "sqlite3_column_blob") = 0x5fafb7c9
18:58:05     INFO -  02-18 18:56:29.023 E/GeckoLinker( 2150): CustomElf::GetSymbolPtr(0x5bf20110 ["/data/app/org.mozilla.fennec-1.apk!/assets/libnss3.so"], "sqlite3_column_bytes") = 0x5fafb7f9
18:58:05     INFO -  02-18 18:56:29.023 E/GeckoLinker( 2150): CustomElf::GetSymbolPtr(0x5bf20110 ["/data/app/org.mozilla.fennec-1.apk!/assets/libnss3.so"], "sqlite3_column_text") = 0x5fafb7b1
18:58:05     INFO -  02-18 18:56:29.023 E/GeckoLinker( 2150): CustomElf::GetSymbolPtr(0x5bf20110 ["/data/app/org.mozilla.fennec-1.apk!/assets/libnss3.so"], "sqlite3_changes") = 0x5faeb505
18:58:05     INFO -  02-18 18:56:29.023 E/GeckoLinker( 2150): CustomElf::GetSymbolPtr(0x5bf20110 ["/data/app/org.mozilla.fennec-1.apk!/assets/libnss3.so"], "sqlite3_last_insert_rowid") = 0x5faeb4fd
18:58:05     INFO -  02-18 18:56:29.023 E/GeckoLibLoad( 2150): Load sqlite done
18:58:05     INFO -  02-18 18:56:29.023 D/Robocop ( 2150): waiting for Gecko:Ready
18:58:05     INFO -  02-18 18:56:29.148 I/GeckoDistribution( 2150): Getting file from distribution.
18:58:05     INFO -  02-18 18:56:29.187 E/GeckoLinker( 2150): CustomElf::GetSymbolPtr(0x5bf20110 ["/data/app/org.mozilla.fennec-1.apk!/assets/libnss3.so"], "NSS_Initialize") = 0x0
18:58:05     INFO -  02-18 18:56:29.187 E/GeckoJNI( 2150): missing NSS_Initialize
18:58:05     INFO -  02-18 18:56:29.187 E/GeckoLibLoad( 2150): Throw
18:58:05     INFO -  02-18 18:56:29.187 E/GeckoLibLoad( 2150): Load nss done
18:58:05     INFO -  02-18 18:56:29.187 W/dalvikvm( 2150): threadid=15: thread exiting with uncaught exception (group=0x40b6b1f8)
18:58:05     INFO -  02-18 18:56:29.195 E/GeckoAppShell( 2150): >>> REPORTING UNCAUGHT EXCEPTION FROM THREAD 94 ("Gecko")
18:58:05     INFO -  02-18 18:56:29.195 E/GeckoAppShell( 2150): java.lang.Exception: Error loading nss libraries
18:58:05     INFO -  02-18 18:56:29.195 E/GeckoAppShell( 2150): 	at org.mozilla.gecko.mozglue.GeckoLoader.loadNSSLibsNative(Native Method)
18:58:05     INFO -  02-18 18:56:29.195 E/GeckoAppShell( 2150): 	at org.mozilla.gecko.mozglue.GeckoLoader.loadNSSLibs(GeckoLoader.java:243)
18:58:05     INFO -  02-18 18:56:29.195 E/GeckoAppShell( 2150): 	at org.mozilla.gecko.GeckoThread.initGeckoEnvironment(GeckoThread.java:116)
18:58:05     INFO -  02-18 18:56:29.195 E/GeckoAppShell( 2150): 	at org.mozilla.gecko.GeckoThread.run(GeckoThread.java:168)



https://tbpl.mozilla.org/php/getParsedLog.php?id=34882913&tree=Try&full=1#error0

18:57:44     INFO -  02-18 18:57:32.375 E/GeckoLinker( 2156): Caught segmentation fault @0x0
18:57:44     INFO -  02-18 18:57:32.375 E/GeckoLinker( 2156): Redispatching to registered handler @0x63a52b49
18:57:44     INFO -  02-18 18:57:32.375 I/BootReceiver( 1403): Copying /data/tombstones/tombstone_08 to DropBox (SYSTEM_TOMBSTONE)
18:57:44     INFO -  02-18 18:57:32.406 I/dalvikvm-heap( 1403): Grow heap (frag case) to 8.958MB for 87396-byte allocation
18:57:44     INFO -  02-18 18:57:32.492 I/dalvikvm-heap( 1403): Grow heap (frag case) to 9.061MB for 174776-byte allocation
18:57:44     INFO -  02-18 18:57:32.562 I/dalvikvm-heap( 1403): Grow heap (frag case) to 9.144MB for 87396-byte allocation
18:57:44     INFO -  02-18 18:57:32.687 I/dalvikvm-heap( 1403): Grow heap (frag case) to 9.291MB for 196628-byte allocation
18:57:44     INFO -  02-18 18:57:32.734 E/GeckoLinker( 2229): Caught segmentation fault @0x637d97a8
18:57:44     INFO -  02-18 18:57:32.734 E/GeckoLinker( 2229): ensure @0x637d97a8
18:57:44     INFO -  02-18 18:57:32.734 E/GeckoLinker( 2229): cacheflush(0x608cd000, 0x608d1000)
18:57:44     INFO -  02-18 18:57:32.781 E/GeckoLinker( 2229): Caught segmentation fault @0x63e89c8f
18:57:44     INFO -  02-18 18:57:32.781 E/GeckoLinker( 2229): ensure @0x63e89c8f
18:57:44     INFO -  02-18 18:57:32.781 E/GeckoLinker( 2229): cacheflush(0x60f7d000, 0x60f81000)
18:57:44     INFO -  02-18 18:57:32.867 E/GeckoLinker( 2156): Caught segmentation fault @0x0
18:57:44     INFO -  02-18 18:57:32.867 E/GeckoLinker( 2156): Redispatching to registered handler @0x63a52b49
18:57:44     INFO -  02-18 18:57:32.906 E/GeckoLinker( 2156): Caught segmentation fault @0x0
18:57:44     INFO -  02-18 18:57:32.906 E/GeckoLinker( 2156): Redispatching to registered handler @0xb0006305
18:57:44     INFO -  02-18 18:57:32.906 F/libc    ( 2156): Fatal signal 11 (SIGSEGV) at 0x00000000 (code=1)


Mike -- is this useful?

Flags: needinfo?(mh+mozilla)

Geoff Brown [:gbrown]

Reporter

Comment 14

•

10 years ago

A few more examples:

https://tbpl.mozilla.org/php/getParsedLog.php?id=34901618&tree=Try&full=1#error0 (tegra)
https://tbpl.mozilla.org/php/getParsedLog.php?id=34891405&tree=Try&full=1#error0
https://tbpl.mozilla.org/php/getParsedLog.php?id=34881330&tree=Try&full=1#error1
https://tbpl.mozilla.org/php/getParsedLog.php?id=34891669&tree=Try&full=1#error1

By the way, I recently noticed that Robocop calls GeckoLoader.loadSQLiteLibs() - http://hg.mozilla.org/mozilla-central/annotate/bf0e76f2a7d4/build/mobile/robocop/FennecNativeActions.java#l47 - which is normally first called from GeckoThread. I don't see a problem, but wonder if it might be related.

Geoff Brown [:gbrown]

Reporter

Comment 15

•

10 years ago

It seems to me that the great majority of these crashes are in robocop tests, but it also affects other tests. Here's a crash on jsreftest startup: https://tbpl.mozilla.org/php/getParsedLog.php?id=34743958&full=1&branch=mozilla-inbound#error0

Geoff Brown [:gbrown]

Reporter

Comment 16

•

10 years ago

I occasionally see the new diagnostics (Comment 4):

https://tbpl.mozilla.org/php/getParsedLog.php?id=34778820&full=1&branch=mozilla-inbound#error0

02-17 00:11:26.227 E/GeckoLinker( 1898): mprotect @0x52cea000, 0xffeb6000, 0x3 failed with errno 12
02-17 00:11:26.227 E/GeckoLinker( 1898): mprotect failed
02-17 00:11:26.227 E/GeckoLibLoad( 1898): Load nss done

Mike Hommey [:glandium]

Assignee

Comment 17

•

10 years ago

At this point, I think I need a full core dump to get any further. To do that, we have to let the kernel handle the crash with the following patch:

diff --git a/mozglue/linker/ElfLoader.cpp b/mozglue/linker/ElfLoader.cpp
--- a/mozglue/linker/ElfLoader.cpp
+++ b/mozglue/linker/ElfLoader.cpp
@@ -1084,21 +1084,17 @@ void SEGVHandler::handler(int signum, si
       CustomElf *elf = static_cast<CustomElf *>(static_cast<LibHandle *>(handle));
       if (elf->mappable->ensure(info->si_addr))
         return;
     }
   }
 
   /* Redispatch to the registered handler */
   SEGVHandler &that = ElfLoader::Singleton;
-  if (that.action.sa_flags & SA_SIGINFO) {
-    DEBUG_LOG("Redispatching to registered handler @%p",
-              FunctionPtr(that.action.sa_sigaction));
-    that.action.sa_sigaction(signum, info, context);
-  } else if (that.action.sa_handler == SIG_DFL) {
+  if (true) {
     DEBUG_LOG("Redispatching to default handler");
     /* Reset the handler to the default one, and trigger it. */
     sys_sigaction(signum, &that.action, nullptr);
     raise(signum);
   } else if (that.action.sa_handler != SIG_IGN) {
     DEBUG_LOG("Redispatching to registered handler @%p",
               FunctionPtr(that.action.sa_handler));
     that.action.sa_handler(signum);

With that, and setting /proc/sys/kernel/core_pattern appropriately, hopefully, we can get the kernel to dump a core at a location we can pull from. Geoff, would you mind taking care of this?

Flags: needinfo?(gbrown)

Mike Hommey [:glandium]

Assignee

Updated

•

10 years ago

Flags: needinfo?(mh+mozilla)

Geoff Brown [:gbrown]

Reporter

Comment 18

•

10 years ago

I am having trouble producing a core file. I have made the ElfLoader change and have MOZILLA_OFFICIAL defined, but SIGSEGV just seems to spawn a new process:

app_47    4457  2543  666652 126168 ffffffff 400de1fc t org.mozilla.fennec_gbrown
app_47    4508  4457  650092 51680 c0070ef8 400de1fc S org.mozilla.fennec_gbrown

and no core is produced.

Geoff Brown [:gbrown]

Reporter

Comment 19

•

10 years ago

Mike -- I would like to help with this, but I am hung up on producing a core file. If you can verify the code and procedure necessary to generate a core on a local device, I am happy to repro this crash on try and retrieve the core.

Flags: needinfo?(gbrown)

Mike Hommey [:glandium]

Assignee

Comment 20

•

10 years ago

Try this patch instead:

diff --git a/mozglue/linker/ElfLoader.cpp b/mozglue/linker/ElfLoader.cpp
--- a/mozglue/linker/ElfLoader.cpp
+++ b/mozglue/linker/ElfLoader.cpp
@@ -1084,24 +1084,24 @@ void SEGVHandler::handler(int signum, si
       CustomElf *elf = static_cast<CustomElf *>(static_cast<LibHandle *>(handle));
       if (elf->mappable->ensure(info->si_addr))
         return;
     }
   }
 
   /* Redispatch to the registered handler */
   SEGVHandler &that = ElfLoader::Singleton;
-  if (that.action.sa_flags & SA_SIGINFO) {
-    DEBUG_LOG("Redispatching to registered handler @%p",
-              FunctionPtr(that.action.sa_sigaction));
-    that.action.sa_sigaction(signum, info, context);
-  } else if (that.action.sa_handler == SIG_DFL) {
+  if (true) {
+    struct sigaction action;
+    action.sa_handler = SIG_DFL;
+    sigemptyset(&action.sa_mask);
+    action.sa_flags = 0;
     DEBUG_LOG("Redispatching to default handler");
     /* Reset the handler to the default one, and trigger it. */
-    sys_sigaction(signum, &that.action, nullptr);
+    sys_sigaction(signum, &action, nullptr);
     raise(signum);
   } else if (that.action.sa_handler != SIG_IGN) {
     DEBUG_LOG("Redispatching to registered handler @%p",
               FunctionPtr(that.action.sa_handler));
     that.action.sa_handler(signum);
   } else {
     DEBUG_LOG("Ignoring");
   }

Geoff Brown [:gbrown]

Reporter

Comment 21

•

10 years ago

Mike, thanks for the new patch. It seems like a step in the right direction, in that a SIGSEGV now ends Fennec abruptly...but I still cannot find a core file.

Have you been able to generate a core file on a device with this patch?

Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)

Updated

•

10 years ago

Blocks: 980105

Geoff Brown [:gbrown]

Reporter

Updated

•

10 years ago

Blocks: 957185

Geoff Brown [:gbrown]

Reporter

Comment 22

•

10 years ago

(In reply to Geoff Brown [:gbrown] from comment #21)
> Mike, thanks for the new patch. It seems like a step in the right direction,
> in that a SIGSEGV now ends Fennec abruptly...but I still cannot find a core
> file.
> 
> Have you been able to generate a core file on a device with this patch?

:glandium -- Reminder that I'm waiting for you to verify a patch that can produce a core. I'm happy to repro this crash and get a core file for you once you give me that patch. This is still a fairly frequent crash and sheriffs are anxious to have this resolved.

Flags: needinfo?(mh+mozilla)

Geoff Brown [:gbrown]

Reporter

Updated

•

10 years ago

Blocks: 982444

Ed Morley [:emorley]

Comment 23

•

10 years ago

Mike, I'm keen to stop us filling up bug 722166 further - for the various failure modes that are related to this bug (that end up over there) - what extra logging can we add to give something for TBPL to star against from the logcat? (Or alternatively, what strings can we add to TBPL's regex?)

Ed Morley [:emorley]

Updated

•

10 years ago

Depends on: 1017578

Ed Morley [:emorley]

Comment 24

•

10 years ago

(In reply to Ed Morley [:edmorley UTC+0] from comment #23)
> Mike, I'm keen to stop us filling up bug 722166 further - for the various
> failure modes that are related to this bug (that end up over there) - what
> extra logging can we add to give something for TBPL to star against from the
> logcat? (Or alternatively, what strings can we add to TBPL's regex?)

Filed bug 722166 for this.

Geoff Brown [:gbrown]

Reporter

Updated

•

10 years ago

Blocks: 953083

Geoff Brown [:gbrown]

Reporter

Updated

•

10 years ago

Blocks: 966788

Geoff Brown [:gbrown]

Reporter

Updated

•

10 years ago

Blocks: 1015499

Geoff Brown [:gbrown]

Reporter

Updated

•

10 years ago

Blocks: 1012894

Geoff Brown [:gbrown]

Reporter

Updated

•

10 years ago

Blocks: 1006244

Geoff Brown [:gbrown]

Reporter

Comment 25

•

10 years ago

(In reply to Geoff Brown [:gbrown] from comment #22)
> (In reply to Geoff Brown [:gbrown] from comment #21)
> > Mike, thanks for the new patch. It seems like a step in the right direction,
> > in that a SIGSEGV now ends Fennec abruptly...but I still cannot find a core
> > file.
> > 
> > Have you been able to generate a core file on a device with this patch?
> 
> :glandium -- Reminder that I'm waiting for you to verify a patch that can
> produce a core. I'm happy to repro this crash and get a core file for you
> once you give me that patch. This is still a fairly frequent crash and
> sheriffs are anxious to have this resolved.

My offer still stands. Intermittent failures continue, as seen here: https://tbpl.mozilla.org/php/getParsedLog.php?id=45938009&full=1&branch=mozilla-inbound#error6

Clearing the ni simply to tidy my dashboard.

Flags: needinfo?(mh+mozilla)

Mike Hommey [:glandium]

Assignee

Comment 26

•

4 years ago

We stopped using MappableSeekableZStream in bug 1307886.

Status: ASSIGNED → RESOLVED

Closed: 4 years ago

Resolution: --- → WORKSFORME

Nobody; OK to take it and work on it

Updated

•

3 years ago

Product: Firefox for Android → Firefox for Android Graveyard