985775 - Debugging child processes with MOZ_DEBUG_CHILD_PROCESS=1 ./run-gdb.sh doesn't work any more

Status: RESOLVED WONTFIX

(Firefox OS Graveyard :: GonkIntegration, defect)

Description

[Gregor Wagner [:gwagner]]

On my nexus 4 with --enable-debug --enable-optimize gecko.

Attaching to a new app makes gdb stop twice now and it gets stuck on the page with the icon.

https://pastebin.mozilla.org/4630715

Comment 1

[Gregor Wagner [:gwagner]]

Maybe related to bug 977859?
When I break now I see
Program received signal SIGINT, Interrupt.
[Switching to Thread 1321.1321]
syscall () at bionic/libc/arch-arm/bionic/syscall.S:41
41	    ldmfd   sp!, {r4, r5, r6, r7}
(gdb) bt
#0  syscall () at bionic/libc/arch-arm/bionic/syscall.S:41
#1  0xb4badd28 in BroadcastSetThreadSandbox () at ../../../../security/sandbox/linux/Sandbox.cpp:360
#2  mozilla::SetCurrentProcessSandbox () at ../../../../security/sandbox/linux/Sandbox.cpp:425
#3  0xb52ebd22 in mozilla::dom::ContentChild::RecvSetProcessSandbox (this=<optimized out>) at ../../../dom/ipc/ContentChild.cpp:696
#4  0xb4e004ea in mozilla::dom::PContentChild::OnMessageReceived (this=0xb3d45018, __msg=...) at PContentChild.cpp:3477
#5  0xb4da9e34 in mozilla::ipc::MessageChannel::DispatchAsyncMessage (this=0xb3d45048, aMsg=...) at ../../../ipc/glue/MessageChannel.cpp:1142
#6  0xb4dacee4 in mozilla::ipc::MessageChannel::OnMaybeDequeueOne (this=0xb3d45048) at ../../../ipc/glue/MessageChannel.cpp:1039
#7  0xb4da9840 in DispatchToMethod<mozilla::ipc::MessageChannel, void (mozilla::ipc::MessageChannel::*)()> (method=
    (void (mozilla::ipc::MessageChannel::*)(mozilla::ipc::MessageChannel * const)) 0xb4dace51 <mozilla::ipc::MessageChannel::OnMaybeDequeueOne()>, obj=<optimized out>, arg=<optimized out>)
    at ../../../ipc/chromium/src/base/tuple.h:383

Comment 2

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

The sandbox code sends signal 35 to non-main threads that exist when the sandbox is started to make them sandbox themselves (seccomp-bpf state is per-thread and inherited on clone).  gdb and/or gdbserver seems to be confused about what to do with that signal.

Using the gdb command "handle 35 nostop noprint" doesn't prevent this.

Currently the only workaround is --disable-content-sandbox, which isn't ideal.

Comment 3

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

I've strace'd gdb:

  8709  send(6, "$vCont;c:p319.-1#7b", 19, 0) = 19
  8709  recv(6, "$T8f0b:e06ed841;0d:901b2640;0f:c0f30440;thread:p319.33e;core:0;#3b", 8192, 0) = 66

Given https://sourceware.org/gdb/onlinedocs/gdb/Stop-Reply-Packets.html#Stop-Reply-Packets and a copy of the gdb source, that's signal 0x8f = 143 (in the gdb protocol signal numbering) which is this:

  /* Some signal we don't know about.  */
  SET (GDB_SIGNAL_UNKNOWN, 143, NULL, "Unknown signal")

That may also explain why gdb can't continue from it.

I'd debug gdbserver itself, but the binary in prebuilt/android-arm/gdbserver is stripped and accompanied only by this message:

  To compile a new gdbserver, see
  //toolchain/gdb/src/gdb/gdbserver/README_ANDROID.txt

Which is not maximally helpful.  But I have a copy of gdbserver I cross-compiled from the upstream source on 2013-08-17 (commit d08069649dcb) when I was failing to make hardware watchpoints work, and it does this:

  8979  recv(6, "$T2f0b:e06ed841;0d:901b2640;0f:c0f30440;thread:p3bf.3eb;core:0;#c2", 8192, 0) = 66

Which is 0x2f = 47 = this:

  SET (GDB_SIGNAL_REALTIME_35, 47, "SIG35", "Real-time event 35")

And displays like this, as expected:

  Program received signal SIG35, Real-time event 35.

And I can continue, or (better) do "handle SIG35 nostop" and let gdb auto-continue, and it works.

Comment 4

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

So… how do we update our /system/bin/gdbserver?

Comment 5

[Gregor Wagner [:gwagner]]

(In reply to Jed Davis [:jld] from comment #4)
> So… how do we update our /system/bin/gdbserver?

Comment 6

[Michael Wu [:mwu]]

Two options:

1. Switch to a newer branch that has a working gdbserver.
2. Fork the appropriate repo and update it ourselves.

Comment 7

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

(In reply to Michael Wu [:mwu] from comment #6)
> 1. Switch to a newer branch that has a working gdbserver.

The "prebuilt" repo, which is used for ICS, has the following android-arm gdbservers, identified by the (truncated) object ID of the git blob for the gdbserver binary:

# ./gdbserver-2476e3c --version
GNU gdbserver 6.6
Copyright (C) 2006 Free Software Foundation, Inc.

# ./gdbserver-54b19a3 --version
GNU gdbserver (GDB) 7.0
Copyright (C) 2009 Free Software Foundation, Inc.

# ./gdbserver-9dd08eb --version
GNU gdbserver (GDB) 7.0
Copyright (C) 2009 Free Software Foundation, Inc.

# ./gdbserver-ae5668d --version
GNU gdbserver (GDB) 7.1-android-gg2
Copyright (C) 2010 Free Software Foundation, Inc.

The newest one, ae5668d, is what we're currently using.  I haven't tested the older ones; I'm assuming they'd be at least as broken.

The "prebuilts/misc" repo, used for JB and KK, also has ae5668d on its JB branches.  But the KitKat branches have this:

# ./gdbserver-6cf69a8 --version
GNU gdbserver (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.

And that one works.  This leaves the question of how to incorporate it into the ICS and JB manifests in a way that works with the build system.

Comment 8

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

So, for JB we might be able to get away with changing base-jb.xml to force a particular KK tag for prebuilts/misc.  But I'm not sure what the side-effects would be, and this could break devices I don't have access to test on.  

ICS will be harder.  We can't just include prebuilts/misc, because that will cause multiple conflicts with the "prebuilt" repo, including gdbserver.  So, unless there's some Android.mk magic that I don't know about that would allow overriding the existing gdbserver, we'll have to fork the "prebuilt" repo, and fork every distinct revision of it that's used by an ICS device.

Comment 9

[Dave Hylands [:dhylands]]

Apparently you can do the following Android.mk magic.

Create some new gdbserver/Android.mk and in it put:

LOCAL_MODULE := moz_gdbserver
LOCAL_MODULE_OVERRIDES_PACKAGES := gdbserver

I haven't tested this.

Comment 10

[Michael Wu [:mwu]]

I'm not too concerned about ICS though, unless we think we'll be able to deliver an update to ICS devices with seccomp support. Even if we do need to fork "prebuilt", it should probably be fine - I don't think vendors generally change that repo too much.

Comment 11

[Gregor Wagner [:gwagner]]

Can we back out the patch that caused the regression until we find out what to do here?

Comment 12

[Guillaume Destuynder [:kang] [this account is no longer active]]

export MOZ_DISABLE_CONTENT_SANDBOX=1 should also work as a work-around btw.

Comment 13

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

(In reply to Guillaume Destuynder [:kang] (use NEEDINFO!) from comment #12)
> export MOZ_DISABLE_CONTENT_SANDBOX=1 should also work as a work-around btw.

It actually won't; this is why I suggested --disable-content-sandbox.

The other workaround is to cd into a clone of platform/prebuilts/misc (doesn't matter what branch is checked out) and `git show 6cf69a8 > .../path/to/B2G/prebuilt/android-arm/gdbserver/gdbserver`.

Comment 14

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

Attachment: Step 1/2: Import new gdbserver & install as gdbserver.new

A first draft.  The makefile magic may not be the right way to do what it's doing.  Also, I should probably condition this on the underlying Android version somehow so that it doesn't become a gdbserver downgrade in the future.

Comment 15

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

Attachment: Step 2/2: Use gdbserver.new in run-gdb.sh if present.

Comment 16

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

(In reply to Dave Hylands [:dhylands] from comment #9)
> LOCAL_MODULE_OVERRIDES_PACKAGES := gdbserver

This appears to be LOCAL_OVERRIDES_PACKAGES, but I couldn't get it to work, and then I found this comment above the definition of `_get-package-overrides` in build/core/definitions.mk:

# Given a list of installed modules (short or long names)
# return a list of the packages (yes, .apk packages, not
# modules in general) that are overridden by this list and,

So I don't think this helps us for gdbserver.

Comment 17

[Gregor Wagner [:gwagner]]

Whats the status here?

Comment 18

[Michael Wu [:mwu]]

Comment on attachment 8396774 [details] [diff] [review]
Step 1/2: Import new gdbserver & install as gdbserver.new

Discussed over IRC, resulting in the alternative in bug 993145.

Comment 19

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

(In reply to Michael Wu [:mwu] from comment #18)
> Discussed over IRC, resulting in the alternative in bug 993145.

That won't help seccomp-enabled ICS/JB devices (e.g., Geeksphone), however.

Comment 20

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

(In reply to Jed Davis [:jld] from comment #19)
> That won't help seccomp-enabled ICS/JB devices (e.g., Geeksphone), however.

Nor, indeed, the ICS emulator, which is still the version that people use unless they have a specific reason to do otherwise.  I don't know when(/if?) we plan to deprecate it in favor of emulator-kk.

Comment 21

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

WONTFIXing because there really isn't a good answer for this, and devices that are still affected (ICS/JB with seccomp-bpf patches) are increasingly less common.

Also, flame-kk and all Lollipop devices don't do the RT signal spam at all anymore, thanks to bug 1004011.