1581971 - Teach `mach run` about `--debug` and friends for `mobile/android`

Status: RESOLVED FIXED

(Firefox Build System :: Mach Core, enhancement, P1)

Description

[Nick Alexander :nalexander [he/him]]

Running GeckoView-based Apps under a debugger has always been difficult but is now only supported via Android Studio. However, there's no hard limitation here: we just need to do what Android Studio does. Bug 1521996 paved the way to do that work specifically for mobile/android.

What needs to be done is:

1. lldb-server pushed to Android device
2. a script like start_lldb_server.sh run on the Android device (content below)
3. lldb invoked on the host with a command line like:

lldb -O 'platform select remote-android' -O 'platform connect unix-abstract-connect:///org.mozilla.geckoview_example-0/platform-1561487213163.sock' -S ../objdirs/objdir-droid/.lldbinit -O 'process attach -p 3639'

where the socket, .lldbinit script, and Android process ID are determined.

#!/system/bin/sh

# This script launches lldb-server on Android device from application subfolder - /data/data/$packageId/lldb/bin.
# Native run configuration is expected to push this script along with lldb-server to the device prior to its execution.
# Following command arguments are expected to be passed - lldb package directory and lldb-server listen port.

umask 0002

LLDB_DIR=$1
LISTENER_SCHEME=$2
DOMAINSOCKET_DIR=$3
PLATFORM_SOCKET=$4
LOG_CHANNELS=$5

BIN_DIR=$LLDB_DIR/bin
LOG_DIR=$LLDB_DIR/log
TMP_DIR=$LLDB_DIR/tmp
PLATFORM_LOG_FILE=$LOG_DIR/platform.log

export LLDB_DEBUGSERVER_LOG_FILE=$LOG_DIR/gdb-server.log
export LLDB_SERVER_LOG_CHANNELS="$LOG_CHANNELS"
export LLDB_DEBUGSERVER_DOMAINSOCKET_DIR=$DOMAINSOCKET_DIR

# This directory already exists. Make sure it has the right permissions.
chmod 0775 "$LLDB_DIR"

rm -r $TMP_DIR
mkdir $TMP_DIR
export TMPDIR=$TMP_DIR

rm -r $LOG_DIR
mkdir $LOG_DIR

# LLDB would create these files with more restrictive permissions than our umask above. Make sure
# he doesn't get a chance.
# "touch" does not exist on pre API-16 devices. This is a poor man's replacement
cat </dev/null >"$LLDB_DEBUGSERVER_LOG_FILE" 2>"$PLATFORM_LOG_FILE"

cd $TMP_DIR # change cwd

$BIN_DIR/lldb-server platform --server --listen $LISTENER_SCHEME://$DOMAINSOCKET_DIR/$PLATFORM_SOCKET --log-file "$PLATFORM_LOG_FILE" --log-channels "$LOG_CHANNELS" </dev/null >$LOG_DIR/platform-stdout.log 2>&1

Comment 1

[Nick Alexander :nalexander [he/him]]

Emilio: I think you got this working with gdb as well. Anything to add?

Comment 2

[Emilio Cobos Álvarez (:emilio)]

I wrote what I had to do here. TLDR in an adb shell:

# gdbserver :$PORT --attach $PID

And in a regular shell:

$ adb forward tcp:$PORT tcp:$PORT
$ gdb
(gdb) target remote :$PORT
(gdb) add-symbol-file /path/to/libxul.so <the-right-address>

Not sure how easy to automate would that be, specially the "get the right address" bit.

Comment 3

[James Willcox (:snorp) ([e-mail]) (he/him)]

Attachment: lldb-attach-android

I wrote a script that seems to work reasonably well. Symbolication only works because I have a ~/.lldbinit, though. The mach command could just include the right stuff in the lldb startup commands.

Comment 4

[James Willcox (:snorp) ([e-mail]) (he/him)]

Attachment: lldb-attach-android

This version tickles over jdb to try and support wait-for-debugger functionality.

Comment 5

[(No longer employed by Mozilla) Aaron Klotz]

I've got a WIP for this, nominating for 84 sprint.

Comment 6

[Nick Alexander :nalexander [he/him]]

Attachment: Bug 1581971 - Pre: Enable verbose logging in `mach android-emulator`.

Comment 7

[Nick Alexander :nalexander [he/him]]

Attachment: Bug 1581971 - Pre: Lift `prepend_debugger_args`. r?#build

Depends on D93675

Comment 8

[Nick Alexander :nalexander [he/him]]

Attachment: Bug 1581971 - WIP - Teach MOZ_DEBUG_{CHILD_}WAIT_FOR_JAVA_DEBUGGER about GeckoView states.

Depends on D93676

Comment 9

[Nick Alexander :nalexander [he/him]]

Attachment: Bug 1581971 - Pre: Allow simultaneous ADB and interactive subprocesses.

This method of starting ADB processes really doesn't handle
interaction, so it shouldn't inherit parent's stdin. Without this, a
shared stdin will race between two child subprocesses, making it
impossible to run an ADB process in the background and an interactive
process in the foreground. One use case is to run a debug server on a
target device and an interactive debugger on the host.

Depends on D93677

Comment 10

[Nick Alexander :nalexander [he/him]]

Attachment: Bug 1581971 - Pre: Return port number from ADB forwarding.

adb accepts the special syntax tcp:0, which allocates an arbitrary
open port on the target (forward) or the host (reverse). However, not
all Android/ADB versions support this functionality, so return the
port as an optional string.

Depends on D93678

Comment 11

[Nick Alexander :nalexander [he/him]]

Attachment: Bug 1581971 - Pre: Allow long-running ADB processes.

This allows to run, say, a debug server on a target device for the
duration of an interactive debugging session on a host, and then to
kill the process on the target cleanly.

Depends on D93679

Comment 12

[Nick Alexander :nalexander [he/him]]

Attachment: Bug 1581971 - WIP towards `mach run --debug` on Android.

Depends on D93680

Comment 13

[(No longer employed by Mozilla) Aaron Klotz]

Attachment: Bug 1581971: Part 1 - Bump Android NDK requirement to 21d; r=#build,snorp!

NDK 21 includes lldb-server, which we need in order to support
./mach run --debug with lldb.

The Android SDK manager no longer includes a standalone lldb package; perhaps
it was deprecated? Anyway, this appears to currently be the best way to get
lldb-server into a location that is easy to find during build configuration.

Comment 14

[(No longer employed by Mozilla) Aaron Klotz]

Attachment: Bug 1581971: Part 2 - Add configure support for lldb-server on Android; r=#build

We want to find the full path to the correct lldb-server in the NDK.
We reference this variable in a later patch when preparing the device for
debugging.

Depends on D94379

Comment 15

[(No longer employed by Mozilla) Aaron Klotz]

Attachment: Bug 1581971: Part 3 - ADBDevice changes to support mach run --debug; r=aerickson,gbrown,jmaher

• We make pidof public and we change it to guarantee that its resulting list
  of pids consists of integers.
• We update forward and its dependencies to return the value of the port
  that was assigned by adb so that clients may learn which port to use when
  forwarding from 'tcp:0'

Depends on D94380

Comment 16

[(No longer employed by Mozilla) Aaron Klotz]

Attachment: Bug 1581971: Part 4 - Change Android's priority debugger to lldb; r=ahal

./mach run --debug is being enabled for lldb only, so it should be
the preferred debugger instead of gdb.

Depends on D94381

Comment 17

[(No longer employed by Mozilla) Aaron Klotz]

Attachment: Bug 1581971: Part 5 - Update android_device.py to support installing and running lldb-server; r=aerickson, gbrown, jmaher

In verify_android_device, when debugger is true, we setup lldb-server. We
also add a new method, run_lldb_server, for actually starting lldb-server.

Both cases call into a new function, _setup_or_run_lldb_server.

For setup functionality, this function copies lldb-server to the device and
then sets up its perms within the target app's data directory.

For run functionality, this function sets some environment variables and
(re)creates directories and log files as needed. Finally it starts the server
and returns to the caller the absolute path to the socket file on the device.
The client provides that path to lldb in order to connect to the server.

Both cases use ADBDevice.batch_execute to run several commands. The
LLDB_SERVER_INSTALL_COMMANDS_SCRIPT and LLDB_SERVER_START_COMMAND_SCRIPT
commands are intended to produce the same result as the start_lldb_server.sh
script that is provided by Android SDK.

Depends on D94382

Comment 18

[(No longer employed by Mozilla) Aaron Klotz]

Attachment: Bug 1581971: Part 6 - Support mach run --debug with lldb for Android; r=#build

• We add new options to the Android variant of mach run:
  • --debug: enables running with a debugger;
  • --debugger: Allows the user to override the default debugger (lldb).
    The provided argument must still be lldb compatible; this
    is for enabling the ability to specify some kind of wrapper
    script or other debugger front-end, if desired;
  • --debugger-args: Additional arguments to pass to the debugger's command line;
  • --no-attach: Runs the app and prepares the device for debugging, but does
    not actually attach any debuggers. The required ports are
    printed to the log, and then mach exits, thus allowing for
    the user to manually connect.
  • --use-existing-process: This allows the user to attach to an existing
    process, instead of killing existing process(es)
    and starting from scratch. This is useful for
    users who want to attach to an existing process
    that is already in a desired state.

When debugging is enabled:

BEFORE the app starts:

• verify_android_device will install lldb-server if necessary;
• We run am set-debug-app -w to ensure that the app is set as the device's
  debug app. Since we pass -w, when Android starts the target app, it will
  wait for jdb to attach before proceeding.

AFTER the app starts:

• We start lldb-server and obtain the name of its socket file;
• We obtain the pid of the parent process. Alternatively, if
  --use-existing-process was specified and there are already extant child
  processes, we prompt the user to choose which process to which they would
  initially like to attach.
• We forward a local TCP port for jdb debugging.
• We run jdb in the background to connect to the process and then quit.
  This is solely for the purpose of dismissing Android's "waiting for debugger"
  dialog.
• In the foreground, we run lldb, specifying a set of initial commands that
  are required to for symbol resolution and to automatically connect to the
  target pid.

Why lldb? I chose it for consistency with Android Studio. Somebody else is
welcome to implement gdb support if they wish. :-)

Depends on D94383

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/077f29a531ec
Part 1 - Bump Android NDK requirement to 21d; r=snorp,firefox-build-system-reviewers,mhentges
https://hg.mozilla.org/integration/autoland/rev/dcc6da681504
Part 2 - Add configure support for lldb-server on Android; r=firefox-build-system-reviewers,mhentges
https://hg.mozilla.org/integration/autoland/rev/5bbe1c5ebe71
Part 3 - ADBDevice changes to support mach run --debug; r=gbrown,jmaher
https://hg.mozilla.org/integration/autoland/rev/164efe7a3c53
Part 4 - Change Android's priority debugger to lldb; r=ahal
https://hg.mozilla.org/integration/autoland/rev/a16c11c1f02c
Part 5 - Update android_device.py to support installing and running lldb-server; r=gbrown,jmaher
https://hg.mozilla.org/integration/autoland/rev/d77dae42c6ea
Part 6 - Support mach run --debug with lldb for Android; r=mhentges

Comment 20

[(No longer employed by Mozilla) Aaron Klotz]

Attachment: Bug 1581971: Part 7 - Fix path quirk with x86 lldb-server detection; r=bustage

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2871d20800e1
Part 7 - Fix path quirk with x86 lldb-server detection; r=bustage CLOSED TREE

Comment 22

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/077f29a531ec
https://hg.mozilla.org/mozilla-central/rev/dcc6da681504
https://hg.mozilla.org/mozilla-central/rev/5bbe1c5ebe71
https://hg.mozilla.org/mozilla-central/rev/164efe7a3c53
https://hg.mozilla.org/mozilla-central/rev/a16c11c1f02c
https://hg.mozilla.org/mozilla-central/rev/d77dae42c6ea
https://hg.mozilla.org/mozilla-central/rev/2871d20800e1