Closed Bug 1692394 Opened 2 months ago Closed 2 months ago

Include NSException stack in the app notes

Categories

(Toolkit :: Crash Reporting, task)

task

Tracking

()

RESOLVED FIXED
87 Branch
Tracking Status
firefox87 --- fixed

People

(Reporter: mstange, Assigned: mstange)

References

(Blocks 1 open bug)

Details

Attachments

(1 file)

We have a number of places in the tree that catch Objective C exceptions and log them to the app notes. However, we currently only log exception name and "reason", but not the stack at which the exception was thrown.

We should add the exception stack to the app notes.

This is a different stack than the crash stack; after an exception occurs, we usually just move on. We may later crash in a totally different place, or not at all.

NSException has a callStackReturnAddresses property which contains a callstack from the point where the exception was thrown; it seems that NSException eagerly performs a framepointer stackwalk when it is created. Also, getting the callStackReturnAddresses is fast because it does not perform symbolication.
Since it's using framepointer stackwalk, this means we can get the ~full stack in Nightly builds, and partial stacks in Release builds that will at least include the Objective C part of the stack.

This stack will let us see where the exception is triggered and should make it much easier to stop them from firing, or to catch them in a more tightly-scoped fashion.

This bug is step 2 of bug 1692375.

This will let us divine where the exception fired. At the moment, we only see
the stack at which a later crash happened, which could happen much later or even
never. If a crash does happen, it'll usually be in a completely different stack.

We will need to symbolicate these addresses manually, on a per crash report basis.

Depends on D104960

Attachment #9202789 - Attachment description: Bug 1692394 - Add the stack at which an NSException was raised to the app notes, as unsymbolicated hex addresses. r=gsvelto!,haik → Bug 1692394 - Add the stack at which an NSException was thrown to the app notes, as unsymbolicated hex addresses. r=gsvelto!,haik
Pushed by mstange@themasta.com:
https://hg.mozilla.org/integration/autoland/rev/e76de8fe86a2
Add the stack at which an NSException was thrown to the app notes, as unsymbolicated hex addresses. r=gsvelto
Blocks: 1692538
Status: ASSIGNED → RESOLVED
Closed: 2 months ago
Resolution: --- → FIXED
Target Milestone: --- → 87 Branch

We have our first crash report with this data!
https://crash-stats.mozilla.org/report/index/ecc5657a-9edf-49b2-b486-a8faa0210219
(I filed bug 1693872 about it.)

Now I'll try to write a script that does the symbolication.

Here's a user script that you can add to Tampermonkey to have the symbolication done automatically:

// ==UserScript==
// @name         Symbolicate Objective C exception stacks in crash reports
// @namespace    https://crash-stats.mozilla.org/
// @version      0.1
// @description  When navigating to a crash report on crash-stats.mozilla.org, this script automatically looks for Objective C exception stacks in the app notes and replaces them with symbols from the Mozilla symbol server.
// @author       Markus Stange <mstange.moz@gmail.com>
// @match        https://crash-stats.mozilla.org/report/index/*
// @grant        none
// ==/UserScript==

(async function() {
  'use strict';

  function moduleContainsAddress(module, address) {
    const baseAddress = parseInt(module.base_addr, 16);
    const endAddress = parseInt(module.end_addr, 16);
    return baseAddress <= address && address < endAddress;
  }

  async function symbolicateAddresses(addresses, modules) {
    const map = new Map();
    const addressToStackPayloadIndexMap = new Map();
    const stackPayload = [];
    for (const address of addresses) {
      const moduleIndex = modules.findIndex(module => moduleContainsAddress(module, address));
      if (moduleIndex === -1) {
        map.set(address, `0x${address.toString(16)} (unknown module)`);
        continue;
      }

      const module = modules[moduleIndex];
      const baseAddress = parseInt(module.base_addr, 16);
      const relativeAddress = address - baseAddress;
      const stackPayloadIndex = stackPayload.length;
      stackPayload.push([moduleIndex, relativeAddress]);
      addressToStackPayloadIndexMap.set(address, stackPayloadIndex);
    }

    const body = {
      "jobs": [
        {
          "memoryMap": modules.map(m => ([m.debug_file, m.debug_id])),
          "stacks": [stackPayload]
        }
      ]
    };

    const result = await fetch('https://symbols.mozilla.org/symbolicate/v5', {
      body: JSON.stringify(body),
      method: 'POST',
      mode: 'cors',
    }).then(response => response.json());

    const addressResults = result.results[0].stacks[0];
    for (const [address, stackPayloadIndex] of addressToStackPayloadIndexMap) {
      const addressResult = addressResults[stackPayloadIndex];
      const moduleName = addressResult.module;
      const symbolName = addressResult.function ?? addressResult.module_offset;
      const functionOffset = addressResult.function_offset;
      map.set(address, `${symbolName} (in ${moduleName}) + ${functionOffset}`);
    }

    return map;
  }

  async function symbolicateAppNotes(appNotes, modules) {
    const re = /(?<=Obj-C Exception data.*Thrown at stack\:\n)((0x[0-9a-f]+\n)+)/gs;
    const stackMatches = appNotes.matchAll(re);
    const addresses = new Set();
    for (const stackMatch of stackMatches) {
      const stack = stackMatch[1];
      for (const addressString of stack.trimEnd().split("\n")) {
        const address = parseInt(addressString, 16);
        addresses.add(address);
      }
    }
    const symbolMap = await symbolicateAddresses(addresses, modules);
    return appNotes.replaceAll(re, stack => {
      const addresses = stack.trimEnd().split("\n").map(s => parseInt(s, 16));
      const symbols = addresses.map(a => symbolMap.get(a) ?? `<unknown> for ${a}`);
      return symbols.join("\n") + "\n";
    });
  }

  const modules = JSON.parse(document.querySelector("#minidump-stackwalk-json").dataset.minidumpstackwalk).modules;
  const appNotesElem = document.querySelector('tr[title~="app_notes"] > td > pre');
  appNotesElem.textContent = await symbolicateAppNotes(appNotesElem.textContent, modules);
})();
You need to log in before you can comment on or make changes to this bug.