1385970 - cli for signature generation

Status: RESOLVED FIXED

(Socorro :: Processor, task, P2)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

Socorro generates signatures during processing based on information from the raw and processed crashes.

The signature generation "algorithm" is formed from a pipeline of processor rules as specified in configuration in the -stage and -prod environments plus a set of lists located in the socorro/siglists/ directory.

There are a number of problems with this:

1. there's no way to test the signature generation changes without black magic or landing changes and testing them in -stage

2. there's no way to test signature generation independent of all the other rules in the processor

3. it's difficult to see how a change to signature generation affects a number of crashes; we're essentially throwing stuff out there, doing some minor spot checks, and hoping for the best


This bug covers reworking signature generation so that it can be used from the command line and so that it's not tied up with the rest of the processor.

Once we've got that landed, we can iterate on that and further improve the scaffolding and solve other signature generation problems.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

I started thinking about this at the All Hands and threw together a preliminary spec for it and built a prototype.

My prototype had a cli that worked something like this:

    python -m socorro.signature <CRASH_ID> [<CRASH_ID> ...]


Then you can use xargs and normal unix tools.

The output is a work in progress.

The prototype pulls the crash data from -prod using the existing API endpoints for raw and processed crashes. It doesn't use any PII, so it doesn't require tokens or logging in. It also captures the signature that's in -prod and can compare the old signature to the new one.

Then I figured out the list of things I need to do to move this forward safely.

PR #3871 (https://github.com/mozilla-services/socorro/pull/3871) was the first step. It cleaned up most of the signature generation code.

Next step is to move that code into a separate Python module, remove all the Processor scaffolding from it, remove the rest of the configman bits, and then write a single SignatureGenerationRule that triggers it.

Getting this done will make it a lot easier to review incoming signature generation changes and it'll make it easier to experiment with the signature generation algorithm. That'd be super helpful, so I'm working on it now.

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

Currently, it's giving this output when used:

app@34dc76dac769:/app$ python -m socorro.signature 98a17ab0-50fc-4a28-91b7-07db80170720
Working on: 98a17ab0-50fc-4a28-91b7-07db80170720
Original: AsyncShutdownTimeout | profile-change-teardown | Extension Shutdown: jid0-GXjLLfbCoAx0LcltEdFrEkQdQPI@jetpack (Awesome Screenshot - Capture, Annotate & More)
New:      AsyncShutdownTimeout | profile-change-teardown | Extension Shutdown: jid0-GXjLLfbCoAx0LcltEdFrEkQdQPI@jetpack (Awesome Screenshot - Capture, Annotate & More)
Notes: (1)
   Signature replaced with a Shutdown Timeout signature, was: "Abort | OOM | unknown | mozalloc_abort | NS_DebugBreak | nsDebugImpl::Abort | XPTC__InvokebyIndex"
app@34dc76dac769:/app$ 


Plus it has a verbose mode which gives you a clear view of which rules did what:

app@34dc76dac769:/app$ python -m socorro.signature -v 98a17ab0-50fc-4a28-91b7-07db80170720
Working on: 98a17ab0-50fc-4a28-91b7-07db80170720
DEBUG: SignatureGenerationRule
DEBUG:     Signature: u'mozalloc_abort | NS_DebugBreak | nsDebugImpl::Abort | XPTC__InvokebyIndex'
DEBUG: StackwalkerErrorSignatureRule
DEBUG:     Signature: u'mozalloc_abort | NS_DebugBreak | nsDebugImpl::Abort | XPTC__InvokebyIndex'
DEBUG: OOMSignature
DEBUG:     Signature: u'OOM | unknown | mozalloc_abort | NS_DebugBreak | nsDebugImpl::Abort | XPTC__InvokebyIndex'
DEBUG: AbortSignature
DEBUG:     Signature: 'Abort | OOM | unknown | mozalloc_abort | NS_DebugBreak | nsDebugImpl::Abort | XPTC__InvokebyIndex'
DEBUG: SignatureShutdownTimeout
DEBUG:     Signature: u'AsyncShutdownTimeout | profile-change-teardown | Extension Shutdown: jid0-GXjLLfbCoAx0LcltEdFrEkQdQPI@jetpack (Awesome Screenshot - Capture, Annotate & More)'
DEBUG: SignatureRunWatchDog
DEBUG:     Signature: u'AsyncShutdownTimeout | profile-change-teardown | Extension Shutdown: jid0-GXjLLfbCoAx0LcltEdFrEkQdQPI@jetpack (Awesome Screenshot - Capture, Annotate & More)'
DEBUG: SignatureIPCChannelError
DEBUG:     Signature: u'AsyncShutdownTimeout | profile-change-teardown | Extension Shutdown: jid0-GXjLLfbCoAx0LcltEdFrEkQdQPI@jetpack (Awesome Screenshot - Capture, Annotate & More)'
DEBUG: SignatureIPCMessageName
DEBUG:     Signature: u'AsyncShutdownTimeout | profile-change-teardown | Extension Shutdown: jid0-GXjLLfbCoAx0LcltEdFrEkQdQPI@jetpack (Awesome Screenshot - Capture, Annotate & More)'
DEBUG: SigTrim
DEBUG:     Signature: u'AsyncShutdownTimeout | profile-change-teardown | Extension Shutdown: jid0-GXjLLfbCoAx0LcltEdFrEkQdQPI@jetpack (Awesome Screenshot - Capture, Annotate & More)'
DEBUG: SigTrunc
DEBUG:     Signature: u'AsyncShutdownTimeout | profile-change-teardown | Extension Shutdown: jid0-GXjLLfbCoAx0LcltEdFrEkQdQPI@jetpack (Awesome Screenshot - Capture, Annotate & More)'
DEBUG: SignatureJitCategory
DEBUG:     Signature: u'AsyncShutdownTimeout | profile-change-teardown | Extension Shutdown: jid0-GXjLLfbCoAx0LcltEdFrEkQdQPI@jetpack (Awesome Screenshot - Capture, Annotate & More)'
Original: AsyncShutdownTimeout | profile-change-teardown | Extension Shutdown: jid0-GXjLLfbCoAx0LcltEdFrEkQdQPI@jetpack (Awesome Screenshot - Capture, Annotate & More)
New:      AsyncShutdownTimeout | profile-change-teardown | Extension Shutdown: jid0-GXjLLfbCoAx0LcltEdFrEkQdQPI@jetpack (Awesome Screenshot - Capture, Annotate & More)
Notes: (1)
   Signature replaced with a Shutdown Timeout signature, was: "Abort | OOM | unknown | mozalloc_abort | NS_DebugBreak | nsDebugImpl::Abort | XPTC__InvokebyIndex"
app@34dc76dac769:/app$


I'm sure other output formats will be helpful for various use cases (CSV, JSON, etc), but I'm not going to work on those in this pass.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

I rewrote it from scratch fixing a bunch of bugs in the process. I also wrote a CSV outputter to make regression testing easier.

I'm running it through 1,000 crashes we got yesterday. For most crashes, the new code is getting the same signature as what we have in -prod. Some are getting different signatures. I'll look into those tomorrow.

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

In order for the SignatureJitCategory rule to work, it requires "classifiers" information which is only available from /api/ProcessedCrash/ if you've got an API token with "View Personally Identifiable Information" permission.

I don't think there's a way around that (which is fine) and it's just that one rule, so I'm going to note that in the command line documentation. This only affects the cli--signature generation as part of the processor seems to work fine.

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

Attachment: pr 3960 - signature generation overhaul

Comment 6

[[github robot]]

Commit pushed to master at https://github.com/mozilla-services/socorro

https://github.com/mozilla-services/socorro/commit/1443fd5a0226dc860fe2a3ad76a62db902ec0c6f
fixes 1385970 signature cli (#3960)

* fixes bug 1385970 - implement signature generation cli

This implements a signature generation cli that lets you use and test signature
generation in isolation from the rest of the Socorro processor.

Usage:

    python -m socorro.signature CRASHID [CRASHID ...]

It also has a verbose mode which lets you see how the signature changes
rule-by-rule which can help when debugging the algorithm.

* bug 1383113 - implement treelib

* Get tests working; remove configman requirement

* Code cleanup

* Squash CSignatureTool classes; code cleanup

* Fix OOMSignature rule

The rule checked to see if it was in the raw crash at all. That's problematic
because it allows for ever shifting raw crash shape. If we ever switched to
using a schema and filling in missing values, that'd break.

* Fix setup for SignatureGenerationRule

This fixes the processed_crash_minimal where the thread data was wrong, so then
the SignatureGenerationRule didn't work for any crashes where the crashing
thread wasn't 0. Oops.

* Remove notes argument from predicate

* Add doc-string

* Implement outputters and CSV output

* Fix watchdog rule

* Add note about classifications

* Implement SOCORRO_API_TOKEN use and add better help text

* Add default keyword to tree_get calls

* Cleanup and docs

* Add signature generation docs to manual

* Fix debug logging and a few other issues

* Cleanup based on PR comments

* fix imports in mozilla_transform_rules.py
* improve treelib per Peter's suggestions
* nix file redirection in cli example
* Ravn DSN -> Sentry DSN
* change behavior of empty paths and edges in tree_get
* remove some unnecessary code
* handle case where processed_crash might not have a signature key
* handle 429 and 500 when retrieving raw and processed crashes
* add tests for SignatureGenerator