1156828 - [Raptor] Add support for monitoring/alerting functionality

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::PerformanceTest, defect)

Description

[:Eli Perelman]

With Datazilla deprecated, along with it goes the fxos-perf-alerts list, which sends emails out based on whether regressions were detected in Datazilla data. We should have a similar option for Raptor automation data: to subscribe to email alerts if a regression is detected.

As a nice-to-have: alerts could be segregated to particular areas, e.g. I can subscribe to email application launch regressions without subscribing to *all* application launch regressions.

Comment 1

[William Lachance (:wlach)]

I have a preliminary hacky script which tries to detect regressions in the raptor data. It reuses some code that was previously written for graphserver, and that which we're planning to use to detect regressions in Talos in Perfherder (http://treeherder.mozilla.org/perf.html):

https://github.com/wlach/phanalysis/blob/master/phanalyzer/analyze_raptor.py

To try it, do the following (assumes you are running Mac or Linux, and have python+virtualenv installed):

    virtualenv raptor
    cd raptor
    source bin/activate
    pip install -U git+git://github.com/wlach/phanalysis.git
    analyze_raptor goldiewilson-onepointtwentyone-1.c.influxdb.com heroku heroku

It should output a bunch of json. The keys are the names of the applications. The values are a list of regressions that it thinks it detected (along with some metadata about them: revision, magnitude, confidence, etc.). Note that Raptor does not seem to be storing a complete set of revision information, so that part might be somewhat inaccurate.

Comment 2

[William Lachance (:wlach)]

If you want to check out a local copy and make changes, follow this procedure instead:

    virtualenv raptor
    cd raptor
    source bin/activate
    git clone https://github.com/wlach/phanalysis.git
    pip install -e phanalysis
    analyze_raptor goldiewilson-onepointtwentyone-1.c.influxdb.com heroku heroku

If you have an existing checkout of phanalysis, just pip install -e that directory.

Comment 3

[Bobby Chien]

[Tracking Requested - why for this release]:

Comment 4

[:Eli Perelman]

OK, to update here, I have a fork of Will's repo that handles what I need for Raptor regression alerting. Pretty much all the changes exist in here:

https://github.com/eliperelman/phanalysis/blob/master/phanalyzer/analyze_raptor.py

That repo will report to another endpoint, which takes care of the "alerting":

https://github.com/eliperelman/raptor-bugs

raptor-bugs will accept the output of phanalysis, and it will file bugs in respective components if regressions are detected. These repos are currently running in Heroku daily, and singular app regressions are filed in their apps, and multiple regressions are filed in "Performance" for a deeper investigation into a platform/System-level regression.

Ping me if you want further details. :)

Comment 5

[Bobby Chien]

Eli, how to subscribe alerting message?

Comment 6

[:Eli Perelman]

What's cool about having this go through Bugzilla is that all subscribing should happen internally. For you though Bobby, I had already built in a special case [1] :)

For everyone else though, we should stick to Bugzilla management. Here are ways I can see knowing about regressions:

1. Subscribe to the Firefox OS :: Performance component
2. Subscribe to the individual application components, e.g. Gaia :: E-Mail
3. Set up User Watching for [e-mail] (https://bugzilla.mozilla.org/userprefs.cgi?tab=email)
4. Create a saved bug search using the Firefox OS component, and include the perf and regression keywords, since these are what Raptor files bugs under [2].

[1] https://github.com/eliperelman/raptor-bugs/blob/master/config.json#L4
[2] https://github.com/eliperelman/raptor-bugs/blob/master/regression/index.js#L6-L7