1209982 - raptor-regression doesn't seem to work properly

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::PerformanceTest, defect)

Description

[:Eli Perelman]

Attachment: data.json

In trying to use the raptor-regression package, it appears that something doesn't work properly; it doesn't report a regression in data where there clearly is one. I just assume that something got lost in translation between the python package and the Node.js package.

Here is the query I am using:

SELECT PERCENTILE(value, 95) as value FROM "measure" WHERE "metric" = 'visuallyLoaded' AND "context" = 'clock.gaiamobile.org' AND "branch" = 'master' AND "device" = 'flame-kk' AND "memory" = '319' AND "test" = 'cold-launch' AND time > '2015-09-09' AND time < '2015-09-17' GROUP BY time(15m), branch, device, memory fill(none)

I have attached the JSON output of this query.

Comment 1

[Robert Wood [:rwood]]

I hacked the existing python regression detection tool to accept the data sample attached, and it did find the regression:

'results': [{'confidence': 20.812692458402307,
              'gaia_revision': 'gaia',
              'gecko_revision': 'gecko',
              'newavg': 4030.25,
              'oldavg': 1658.0833333333333,
              'prev_gaia_revision': 'gaia',
              'prev_gecko_revision': 'gecko',
              'push_timestamp': u'2015-09-15T07:45:00Z'}]}

https://gist.github.com/rwood-moz/90391c3b110b0c06a479

Comment 2

[Robert Wood [:rwood]]

Attachment: https://github.com/rwood-moz/raptor-regression/pull/3

Thanks for your help during our online mega debugging session!