1333813 - No way to attach gdb to a process when inside a one-click-loaner

Status: REOPENED

(Taskcluster :: Workers, enhancement, P5)

Description

[Paul Adenot (:padenot)]

Using a one-click-loaner, I need to debug a issue with Firefox e10s. This means that I need to attach gdb to the child process (I run the initial process with gdb with `mach mochitest --debugger=gdb some/test/path`).

For that, we need the PID of the child, but this is doable with:

> MOZ_DEBUG_CHILD_PROCESS=1 mach mochitest [...]

which prints the PID of the child, when it's forked out.

Now, I have the PID. I have found that the binaries are in `~/workspace/build/application/firefox`, and I attempt to attach gdb to the right process (that is kept paused using the other gdb):

> gdb firefox $PID

this errors out:

> Attaching to program: /home/worker/workspace/build/application/firefox/firefox,   
> process 6254
> Could not attach to process.  If your uid matches the uid of the target
> process, check the setting of /proc/sys/kernel/yama/ptrace_scope, or try
> again as the root user.  For more details, see /etc/sysctl.d/10-ptrace.conf
> ptrace: Operation not permitted.

I am root

> $ whoiam
> root

but it still does not work. Attempting to change the scope for ptrace does not work:

> echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope
> sudo: unable to resolve host taskcluster-worker
> tee: /proc/sys/kernel/yama/ptrace_scope: Read-only file system
> 0

Trying to re-mount /dev/proc as rw also does not work:

> # mount -o rw,remount /proc/sys
> mount: cannot remount proc read-write, is write-protected

which makes sense, since this is a docker container.

Comment 1

[Jonas Finnemann Jensen (:jonasfj)]

Probably we can set:
> echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope

In docker-worker and that will solve the problem.

There might be security implications, so perhaps it has to be a config option we can disable for machines that builds releases.
Note: I don't think processes inside the container will be able to see reference pids on other namespaces, so this might not allow for docker escapes who knows.

Note: this can't be a per-task option, as it's global for the system.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

I'm not sure what /yama/ is, but note that we have an allowPtrace feature for docker-worker already.  It is a security issue, but one we can tolerate for testers since they do not produce output that goes into releases.

As I recall, that feature is set on testers, because crashreporter uses it.

Does the one-click loaner process strip the feature?

Comment 3

[Steve Fink [:sfink] [:s:]]

https://www.kernel.org/doc/Documentation/security/Yama.txt

It helpfully refers specifically to Firefox and crashreporter, too!

Comment 4

[(not currently active) Ted Mielczarek]

(In reply to Dustin J. Mitchell [:dustin] from comment #2)
> I'm not sure what /yama/ is, but note that we have an allowPtrace feature
> for docker-worker already.  It is a security issue, but one we can tolerate
> for testers since they do not produce output that goes into releases.
> 
> As I recall, that feature is set on testers, because crashreporter uses it.
> 
> Does the one-click loaner process strip the feature?

This is a separate *additional* restriction on ptrace in the kernel. The `allowPtrace` feature was required to allow using ptrace *at all*, which is restricted by the default docker configuration. The `ptrace_scope` security disallows ptracing arbitrary processes--you can ptrace child processes, and processes that call `prctl(PR_SET_PTRACER, pid, ...)` (which is what the Firefox crashreporter code does, which is why those tests work), but not other processes, which is what `gdb firefox $PID` needs to work.

I don't know if there's a way to enable this just inside a single container. Setting the kernel setting would mean that any task running on that instance would gain this permission. Now, short of finding a kernel exploit, processes still *should* only be able to ptrace other processes inside the container, so it's not a gaping security hole or anything, but defense in depth, etc.

Paul: as a workaround, you could stick prctl calls in the Firefox startup path like:
`prctl(PR_SET_PTRACER, PR_SET_PTRACER_ANY, ...)`

You'd have to make sure it gets hit by the content process startup path as well if you need to attach to that process.

Comment 5

[Paul Adenot (:padenot)]

Ted, I filed bug 1334132 for that. I don't think it's unreasonable to set this in debug mode.

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Thanks for setting me straight :)

If that's not sufficient, I'd be comfortable doing what Jonas suggests in comment 1 when the allowPtrace feature is active.  We're already accepting the (minor) risk of compromise via ptrace with that feature.  So as long as setting this value to 0 doesn't allow escape from the container's pid namespace, it doesn't seem like it adds any additional risk.

Comment 7

[Pete Moore [:pmoore][:pete]]

Jonas, do we have a strategy for attaching debuggers to interactive loaners in general? I know it is something we've discussed quite a lot in the past.

Comment 8

[Jonas Finnemann Jensen (:jonasfj)]

Yeah, you can run with allowPtrace, then you can trace child processes.

But IMO, we should fix this by allowing you to ptrace any PID, should be an easy fix to be honest.
This is a minor system tweak in docker-worker AMIs.

Comment 9

[Pete Moore [:pmoore][:pete]]

Wander, is this something you have bandwidth to look into?

Comment 10

[Wander Lairson Costa]

(In reply to Pete Moore [:pmoore][:pete] from comment #9)
> Wander, is this something you have bandwidth to look into?

Not right now, kind of busy with docker-engine.

Comment 11

[Chris Cooper [:coop] (he/him)]

We'll be focusing on moving to generic-worker first, and will then schedule some work to bring one-click loaners up to scratch.

Comment 12

[Chris Cooper [:coop] (he/him)]

Pete: is this part of your current slate of work on interactive tasks?

Comment 13

[Dustin J. Mitchell [:dustin] (he/him)]

(closed in bulk closure of old P5 bugs, representing that they are unlikely to ever be worked on)

Comment 14

[Pete Moore [:pmoore][:pete]]

Reopening inactive bugs, because they may still need attention. Historically, inactive bugs were closed, but this hides the fact there are genuine issues which have not been resolved.