Closed Bug 1194399 Opened 10 years ago Closed 10 years ago

[raptor] large variances in raptor results on gaia-ci

Categories

(Firefox OS Graveyard :: Gaia::PerformanceTest, defect)

Product:
Firefox OS Graveyard
Component:
Gaia::PerformanceTest
Platform:
ARM
Gonk (Firefox OS)
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: rwood, Assigned: rwood)

Details

(Whiteboard: [systemsfe])

Something has changed somewhere, now I am seeing wild variances on Raptor results on gaia-ci, where huge regressions are detected, and then on the next run (the retry, with same gecko/gaia) 0 regression is detected. For example, from this sample on gaia: https://treeherder.mozilla.org/#/jobs?repo=gaia&revision=1796ff37b69ea9b9e412912783042bf56fcbc12e Raptor launch test results summary for dialer: coldlaunch.visuallyLoaded p95 has regressed by 91.345%, which is above the 15% threshold On retry: Raptor launch test results summary for dialer: No regression detected, p95 median value has not increased Raptor launch test results summary for fm: coldlaunch.visuallyLoaded p95 has regressed by 109.524%, which is above the 15% threshold On retry: Raptor launch test results summary for fm: No regression detected, p95 median value has not increased Raptor launch test results summary for music: coldlaunch.visuallyLoaded p95 has regressed by 55.774%, which is above the 15% threshold On retry: Raptor launch test results summary for music: No regression detected, p95 median value has not increased Other examples: https://tools.taskcluster.net/task-inspector/#Bnlp6kBzSiqMGkzXvABzIQ/0 https://tools.taskcluster.net/task-inspector/#IZlBNl6hSLWeufkrScYDjw/0 https://tools.taskcluster.net/task-inspector/#eYO2lFNuQJS6DOQi8rDoRg/0 https://tools.taskcluster.net/task-inspector/#SWChMDLySPu-81ozhqrsig/0 https://tools.taskcluster.net/task-inspector/#PFujzHHQTY6RHfWijOvgwg/0 I've inquired with the taskcluster guys and they are not aware of any changes being made to the raptor worker type. I'll try to trace back to when it began.
On gaia-master this goes back at least as far as this commit (which is before upgrading to the latest Raptor CLI 1.6.0) but after switching over to the new Raptor CLI on gaia-ci. It's very tough to narrow down beyond this as there are so many failures/errors from the marionette socket timeout etc. that were introduced originally on the emulator with the new CLI. https://treeherder.mozilla.org/#/jobs?repo=gaia-master&revision=474784fc494519a0fab866407c0215aacf984ed9
Whiteboard: [systemsfe]
Status: NEW → RESOLVED
Closed: 10 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.