raptor-regression doesn't seem to work properly

RESOLVED FIXED

Status

Firefox OS
Gaia::PerformanceTest
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: Eli, Assigned: rwood)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [systemsfe])

Attachments

(2 attachments)

(Reporter)

Description

3 years ago
Created attachment 8667922 [details]
data.json

In trying to use the raptor-regression package, it appears that something doesn't work properly; it doesn't report a regression in data where there clearly is one. I just assume that something got lost in translation between the python package and the Node.js package.

Here is the query I am using:

SELECT PERCENTILE(value, 95) as value FROM "measure" WHERE "metric" = 'visuallyLoaded' AND "context" = 'clock.gaiamobile.org' AND "branch" = 'master' AND "device" = 'flame-kk' AND "memory" = '319' AND "test" = 'cold-launch' AND time > '2015-09-09' AND time < '2015-09-17' GROUP BY time(15m), branch, device, memory fill(none)

I have attached the JSON output of this query.
(Assignee)

Comment 1

3 years ago
I hacked the existing python regression detection tool to accept the data sample attached, and it did find the regression:

'results': [{'confidence': 20.812692458402307,
              'gaia_revision': 'gaia',
              'gecko_revision': 'gecko',
              'newavg': 4030.25,
              'oldavg': 1658.0833333333333,
              'prev_gaia_revision': 'gaia',
              'prev_gecko_revision': 'gecko',
              'push_timestamp': u'2015-09-15T07:45:00Z'}]}

https://gist.github.com/rwood-moz/90391c3b110b0c06a479
(Assignee)

Comment 2

3 years ago
Created attachment 8669128 [details] [review]
https://github.com/rwood-moz/raptor-regression/pull/3

Thanks for your help during our online mega debugging session!
Attachment #8669128 - Flags: review?(eperelman)
(Reporter)

Updated

3 years ago
Attachment #8669128 - Flags: review?(eperelman) → review+
(Assignee)

Updated

3 years ago
Assignee: nobody → rwood
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.