Closed Bug 2071467 Opened 2 months ago Closed 25 days ago

Allow use of forked executor in minidump-writer

Categories

(Toolkit :: Crash Reporting, task)

task

Tracking

()

RESOLVED FIXED
161 Branch
Tracking Status
firefox161 --- fixed

People

(Reporter: cmartin, Assigned: schopin)

References

(Blocks 1 open bug)

Details

Attachments

(4 external files, 2 obsolete files)

minidump-writer has the changes necessary to support a remote executor running in a forked process -- Let's put it all together in Firefox so m-w can finally work with Android's Isolated Process feature enabled.

As per our discussions this is the code used to integrate the proxy minidump writer code which can generate minidumps for isolated processes. The current limitations are as following:

  • This only works with content sandbox set to 0, otherwise both fork() and socketpair() are blocked and we can't spawn the cloned process that serves as a proxy of the crashed process. This needs a bit of thought on how to do without weakening the sandbox.
  • We can't figure out whether we're in an isolated process or not (yet)
  • We might need to read a Firefox preference from within Breakpad code or find an alternative
Assignee: cmartin → schopin

Do we have the content sandbox on Android though? If I read the code correctly, it shouldn't be enabled since Android isn't a GNU OS.

In which case, the fork() question is only relevant for the tests. Regarding socketpair(), can't create them on the crashhelper side instead of the client side?

Android doesn't use the content sandbox (hence the need for isolated process support). The issue with fork() affects Linux if we run tests with isolated process support enabled - which we want to do so that we have coverage even before we ship on Android. Regarding socketpair() the only hard requirement we have is that the client endpoint is inherited by the clone of the crashed child... so we could create the pair during startup instead of at crashtime. I don't know how soon we set the sandbox on child processes, but worst case we could create the pair in the parent, pass both ends as parameters during process creation so that the child has them ready right when it starts.

For this first version, annotation support is out of scope, to limit the complexity of the implementation. It will come in a follow-up patch set.

Blocks: 2082922

As far as detection of isolated processes, I'd like to try a simple fallback approach: first try the normal way, and if it fails fall back to the remote executor.
Is there any way to disable the sandbox for specific tests? Weakening the sandbox just for the benefit of testing doesn't feel like a good idea, and I don't see a way out of using fork() or clone().

Jed, we discussed this briefly in Toronto, what do you think would be the better approach here? The remote dumper for isolated processes fork()s a child at crash time and we'd like to be able to test it on Linux in automation where the sandbox would prevent us from forking. Should we put in an escape hatch specifically for this use-case or is there a more robust approach? This isn't a problem on Android where we have no such restriction on fork().

Flags: needinfo?(jld)

Alright, so I misunderstood something here. I assumed at first that both fork() and socketpair() were blocked by the Isolated Process sandbox. Comment 5 corrected me on fork(), but after extensive testing, it turns out that Isolated Processes don't care about socketpair() either, so it's also something related to desktop sandboxing. Apparently, only our GMP sandbox denies that one.

Since GMP processes on Android already use Isolated Processes on nightly, this is still a problem as it would be the first target for our tests.

This also comforts me in using a fallback approach at dump time rather than forcing the same method for all processes on a given install. That way, when we land the changes, the new codepath gets exercised for GMP crashes without having to change any setting, and other processes get the old, safer codepath.

(In reply to Gabriele Svelto [:gsvelto] from comment #8)

Jed, we discussed this briefly in Toronto, what do you think would be the better approach here? The remote dumper for isolated processes fork()s a child at crash time and we'd like to be able to test it on Linux in automation where the sandbox would prevent us from forking. Should we put in an escape hatch specifically for this use-case or is there a more robust approach? This isn't a problem on Android where we have no such restriction on fork().

For mochtests and xpcshell tests, you can use the environment manifest key to set env vars; searchfox can find some examples. For mochitest it has to be in the [DEFAULT] section (so you'd need a separate manifest for this vs. normal desktop tests; see the various security/sandbox/test/browser_*.toml files for an example); for xpcshell it apparently can be set per-test instead. IIRC what I suggested in Toronto was using that to set whichever of the MOZ_DISABLE_*_SANDBOX vars are needed.

It looks like you'd already need to use the prefs key to turn on force-remoting mode, and that has the same DEFAULT-only restriction for mochitest, so this shouldn't be a problem. Also, from looking at the test harness code just now to make sure I knew what I was talking about, I see there's an args key, and I don't know if we'd need that here, but apparently it's a thing.

(In reply to Simon Chopin [:schopin] from comment #9)

but after extensive testing, it turns out that Isolated Processes don't care about socketpair() either, so it's also something related to desktop sandboxing. Apparently, only our GMP sandbox denies that one.

Since GMP processes on Android already use Isolated Processes on nightly, this is still a problem as it would be the first target for our tests.

Yeah, that's originally from bug 1066750 (and for principle-of-least-privilege reasons, as long as it's not needed). But you should be able to turn off desktop sandboxing for these test cases, so that should take care of that.

Flags: needinfo?(jld)

I'm splitting off the Isolated Process detection thing into its own bug, I'll just focus on integration for desktop on this first patch set. I already know how I'll do the second part, but I'd like the first part to be a bit more battle-tested before enabling it on Android.

I'm planning on doing away with the pref entirely, to favor environment variables for the desktop use case. Prefs are a huge pain to pipe through to the crashhelper since we spawn it very early.

Blocks: 2084934
See Also: → 2084970

We'll need to have a bidirectional dialogue between client and server in
order to set up the remote executor, for which the good old pipe isn't
enough.

Since the GMP sandbox doesn't allow socketpair(), we create it when
setting up the client, before the sandbox kicks in. Sadly, it's single-use,
so any subsequent dump simply falls back to creating one on the fly,
which will still fail on sandboxed GMP, but that's still a huge
improvement.

Co-authored-by: Chris Martin <cmartin@mozilla.com>

The forked executor is disabled by default, but the crashhelper will use
it if the env var MOZ_CRASHREPORTER_FORCE_REMOTE_EXECUTOR is set to 1.
This will allow us to test it on desktop Linux.

Co-authored-by: Chris Martin <cmartin@mozilla.com>

This isn't mandatory per se, but is fairly useful to make sure we
actually used the right dumping mechanism in our tests. This annotation
is client-scoped, so no data risk here.

We force the remote executor, and disable the content sandbox since it
doesn't let us fork.

Attachment #9641328 - Attachment is obsolete: true
Attachment #9641329 - Attachment is obsolete: true
Pushed by schopin@mozilla.com: https://github.com/mozilla-firefox/firefox/commit/a7048c12404d https://hg.mozilla.org/integration/autoland/rev/55607e0c71f2 breakpad: use a socket pair for the crash generation channel r=crashreporting-reviewers,afranchuk https://github.com/mozilla-firefox/firefox/commit/c68cd46b6120 https://hg.mozilla.org/integration/autoland/rev/019213c8aa87 Allow dumping child processes through a forked executor r=crashreporting-reviewers,afranchuk https://github.com/mozilla-firefox/firefox/commit/ea1ed8a26398 https://hg.mozilla.org/integration/autoland/rev/1a0caf74fcd5 Add a testing annotation recording use of the remote executor r=crashreporting-reviewers,afranchuk https://github.com/mozilla-firefox/firefox/commit/36102cc3b807 https://hg.mozilla.org/integration/autoland/rev/07c19b3c00aa Test the remote executor on desktop Linux r=crashreporting-reviewers,afranchuk
QA Whiteboard: [qa-triage-done-c162/b161]
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: