Closed Bug 985775 Opened 6 years ago Closed 4 years ago

Debugging child processes with MOZ_DEBUG_CHILD_PROCESS=1 ./run-gdb.sh doesn't work any more

Categories

(Firefox OS Graveyard :: GonkIntegration, defect, blocker)

x86
macOS
defect
Not set
blocker

Tracking

(b2g-v1.2 unaffected, b2g-v1.3 affected, b2g-v1.3T affected, b2g-v1.4 affected, b2g-v2.0 affected)

RESOLVED WONTFIX
Tracking Status
b2g-v1.2 --- unaffected
b2g-v1.3 --- affected
b2g-v1.3T --- affected
b2g-v1.4 --- affected
b2g-v2.0 --- affected

People

(Reporter: gwagner, Assigned: jld)

References

Details

Attachments

(2 files)

On my nexus 4 with --enable-debug --enable-optimize gecko.

Attaching to a new app makes gdb stop twice now and it gets stuck on the page with the icon.

https://pastebin.mozilla.org/4630715
Maybe related to bug 977859?
When I break now I see
Program received signal SIGINT, Interrupt.
[Switching to Thread 1321.1321]
syscall () at bionic/libc/arch-arm/bionic/syscall.S:41
41	    ldmfd   sp!, {r4, r5, r6, r7}
(gdb) bt
#0  syscall () at bionic/libc/arch-arm/bionic/syscall.S:41
#1  0xb4badd28 in BroadcastSetThreadSandbox () at ../../../../security/sandbox/linux/Sandbox.cpp:360
#2  mozilla::SetCurrentProcessSandbox () at ../../../../security/sandbox/linux/Sandbox.cpp:425
#3  0xb52ebd22 in mozilla::dom::ContentChild::RecvSetProcessSandbox (this=<optimized out>) at ../../../dom/ipc/ContentChild.cpp:696
#4  0xb4e004ea in mozilla::dom::PContentChild::OnMessageReceived (this=0xb3d45018, __msg=...) at PContentChild.cpp:3477
#5  0xb4da9e34 in mozilla::ipc::MessageChannel::DispatchAsyncMessage (this=0xb3d45048, aMsg=...) at ../../../ipc/glue/MessageChannel.cpp:1142
#6  0xb4dacee4 in mozilla::ipc::MessageChannel::OnMaybeDequeueOne (this=0xb3d45048) at ../../../ipc/glue/MessageChannel.cpp:1039
#7  0xb4da9840 in DispatchToMethod<mozilla::ipc::MessageChannel, void (mozilla::ipc::MessageChannel::*)()> (method=
    (void (mozilla::ipc::MessageChannel::*)(mozilla::ipc::MessageChannel * const)) 0xb4dace51 <mozilla::ipc::MessageChannel::OnMaybeDequeueOne()>, obj=<optimized out>, arg=<optimized out>)
    at ../../../ipc/chromium/src/base/tuple.h:383
The sandbox code sends signal 35 to non-main threads that exist when the sandbox is started to make them sandbox themselves (seccomp-bpf state is per-thread and inherited on clone).  gdb and/or gdbserver seems to be confused about what to do with that signal.

Using the gdb command "handle 35 nostop noprint" doesn't prevent this.

Currently the only workaround is --disable-content-sandbox, which isn't ideal.
I've strace'd gdb:

  8709  send(6, "$vCont;c:p319.-1#7b", 19, 0) = 19
  8709  recv(6, "$T8f0b:e06ed841;0d:901b2640;0f:c0f30440;thread:p319.33e;core:0;#3b", 8192, 0) = 66

Given https://sourceware.org/gdb/onlinedocs/gdb/Stop-Reply-Packets.html#Stop-Reply-Packets and a copy of the gdb source, that's signal 0x8f = 143 (in the gdb protocol signal numbering) which is this:

  /* Some signal we don't know about.  */
  SET (GDB_SIGNAL_UNKNOWN, 143, NULL, "Unknown signal")

That may also explain why gdb can't continue from it.

I'd debug gdbserver itself, but the binary in prebuilt/android-arm/gdbserver is stripped and accompanied only by this message:

  To compile a new gdbserver, see
  //toolchain/gdb/src/gdb/gdbserver/README_ANDROID.txt

Which is not maximally helpful.  But I have a copy of gdbserver I cross-compiled from the upstream source on 2013-08-17 (commit d08069649dcb) when I was failing to make hardware watchpoints work, and it does this:

  8979  recv(6, "$T2f0b:e06ed841;0d:901b2640;0f:c0f30440;thread:p3bf.3eb;core:0;#c2", 8192, 0) = 66

Which is 0x2f = 47 = this:

  SET (GDB_SIGNAL_REALTIME_35, 47, "SIG35", "Real-time event 35")

And displays like this, as expected:

  Program received signal SIG35, Real-time event 35.

And I can continue, or (better) do "handle SIG35 nostop" and let gdb auto-continue, and it works.
So… how do we update our /system/bin/gdbserver?
Blocks: 981202
(In reply to Jed Davis [:jld] from comment #4)
> So… how do we update our /system/bin/gdbserver?
Flags: needinfo?(mwu)
Two options:

1. Switch to a newer branch that has a working gdbserver.
2. Fork the appropriate repo and update it ourselves.
Flags: needinfo?(mwu)
Assignee: nobody → jld
(In reply to Michael Wu [:mwu] from comment #6)
> 1. Switch to a newer branch that has a working gdbserver.

The "prebuilt" repo, which is used for ICS, has the following android-arm gdbservers, identified by the (truncated) object ID of the git blob for the gdbserver binary:

# ./gdbserver-2476e3c --version
GNU gdbserver 6.6
Copyright (C) 2006 Free Software Foundation, Inc.

# ./gdbserver-54b19a3 --version
GNU gdbserver (GDB) 7.0
Copyright (C) 2009 Free Software Foundation, Inc.

# ./gdbserver-9dd08eb --version
GNU gdbserver (GDB) 7.0
Copyright (C) 2009 Free Software Foundation, Inc.

# ./gdbserver-ae5668d --version
GNU gdbserver (GDB) 7.1-android-gg2
Copyright (C) 2010 Free Software Foundation, Inc.

The newest one, ae5668d, is what we're currently using.  I haven't tested the older ones; I'm assuming they'd be at least as broken.

The "prebuilts/misc" repo, used for JB and KK, also has ae5668d on its JB branches.  But the KitKat branches have this:

# ./gdbserver-6cf69a8 --version
GNU gdbserver (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.

And that one works.  This leaves the question of how to incorporate it into the ICS and JB manifests in a way that works with the build system.
So, for JB we might be able to get away with changing base-jb.xml to force a particular KK tag for prebuilts/misc.  But I'm not sure what the side-effects would be, and this could break devices I don't have access to test on.  

ICS will be harder.  We can't just include prebuilts/misc, because that will cause multiple conflicts with the "prebuilt" repo, including gdbserver.  So, unless there's some Android.mk magic that I don't know about that would allow overriding the existing gdbserver, we'll have to fork the "prebuilt" repo, and fork every distinct revision of it that's used by an ICS device.
Apparently you can do the following Android.mk magic.

Create some new gdbserver/Android.mk and in it put:

LOCAL_MODULE := moz_gdbserver
LOCAL_MODULE_OVERRIDES_PACKAGES := gdbserver

I haven't tested this.
I'm not too concerned about ICS though, unless we think we'll be able to deliver an update to ICS devices with seccomp support. Even if we do need to fork "prebuilt", it should probably be fine - I don't think vendors generally change that repo too much.
Can we back out the patch that caused the regression until we find out what to do here?
export MOZ_DISABLE_CONTENT_SANDBOX=1 should also work as a work-around btw.
(In reply to Guillaume Destuynder [:kang] (use NEEDINFO!) from comment #12)
> export MOZ_DISABLE_CONTENT_SANDBOX=1 should also work as a work-around btw.

It actually won't; this is why I suggested --disable-content-sandbox.

The other workaround is to cd into a clone of platform/prebuilts/misc (doesn't matter what branch is checked out) and `git show 6cf69a8 > .../path/to/B2G/prebuilt/android-arm/gdbserver/gdbserver`.
A first draft.  The makefile magic may not be the right way to do what it's doing.  Also, I should probably condition this on the underlying Android version somehow so that it doesn't become a gdbserver downgrade in the future.
Attachment #8396774 - Flags: feedback?(mwu)
(In reply to Dave Hylands [:dhylands] from comment #9)
> LOCAL_MODULE_OVERRIDES_PACKAGES := gdbserver

This appears to be LOCAL_OVERRIDES_PACKAGES, but I couldn't get it to work, and then I found this comment above the definition of `_get-package-overrides` in build/core/definitions.mk:

# Given a list of installed modules (short or long names)
# return a list of the packages (yes, .apk packages, not
# modules in general) that are overridden by this list and,

So I don't think this helps us for gdbserver.
Whats the status here?
Attachment #8396775 - Attachment is patch: true
Attachment #8396775 - Attachment mime type: application/mbox → text/plain
Comment on attachment 8396774 [details] [diff] [review]
Step 1/2: Import new gdbserver & install as gdbserver.new

Discussed over IRC, resulting in the alternative in bug 993145.
Attachment #8396774 - Flags: feedback?(mwu)
Attachment #8396775 - Flags: feedback?(mwu)
(In reply to Michael Wu [:mwu] from comment #18)
> Discussed over IRC, resulting in the alternative in bug 993145.

That won't help seccomp-enabled ICS/JB devices (e.g., Geeksphone), however.
(In reply to Jed Davis [:jld] from comment #19)
> That won't help seccomp-enabled ICS/JB devices (e.g., Geeksphone), however.

Nor, indeed, the ICS emulator, which is still the version that people use unless they have a specific reason to do otherwise.  I don't know when(/if?) we plan to deprecate it in favor of emulator-kk.
WONTFIXing because there really isn't a good answer for this, and devices that are still affected (ICS/JB with seccomp-bpf patches) are increasingly less common.

Also, flame-kk and all Lollipop devices don't do the RT signal spam at all anymore, thanks to bug 1004011.
Status: NEW → RESOLVED
Closed: 4 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.