Closed Bug 1168928 Opened 8 years ago Closed 8 years ago

[raptor] Improve raptor test task duration consistency


(Taskcluster :: General, defect)

Not set


(Not tracked)



(Reporter: rwood, Assigned: rwood)




(2 files)

There seems to be a large variance in the runtime duration for raptor test tasks, even though they are all running on the same 'raptor' worker type.

For example, on gaia, this raptor launch test task took 30 minutes to run:

And this task, that runs the exact same test, took 15 minutes to run:

Another example, on gaia-master, this test task took 49 minutes:

And this task that runs the exact same test:

James noted it possibly might have to do with having 'isolated containers' set to true in the 'raptor' worker type. Try running raptor test tasks in parallel on the 'raptor-single-capacity' worker type James created.
Blocks: 1149266
Wow, when the raptor test task duration is fast the launch results are noticeably faster.

Launch test task, duration 41 minutes, p95 visuallyLoaded median result: 16660

Same launch test in same suite, next task, duration 24 minutes, p95 visuallyLoaded median result: 4419

Will try the raptor-single-capacity worker type...
Wow that's pretty cool, huge improvement:

Going to switch the raptor tasks from the 'raptor' worker to the 'raptor-single-capacity' worker.
There is a little bit more variation (but not too badly) in the test tasks for the launch test on 'test-startup-limit' app. However it doesn't seem to effect the actual numbers.

Example, this task took 21 minutes:, p95 visuallyLoaded: 3723

Same test, this task took 11 minutes:, p95 visuallyLoaded: 3739

However the duration didn't effect the actual app launch times - the task that ran faster actually had slightly longer app launch times. Most of the test tasks on that app are pretty consistent around the 11 or 15 minute mark. I'd say still a good move to change to the 'raptor-single-capacity' worker type.
Attachment #8612910 - Flags: review?(garndt)
Attachment #8612910 - Flags: review?(garndt) → review?(jlal)
Attachment #8612910 - Flags: review?(jlal) → review?(garndt)
Comment on attachment 8612910 [details] [review]
[gaia] rwood-moz:bug1168928-worker > mozilla-b2g:master

comment in the pull, nothing major or that needs to be addressed at this time
Attachment #8612910 - Flags: review?(garndt) → review+
Thanks Greg. Landed:
Closed: 8 years ago
Resolution: --- → FIXED
Component: TaskCluster → General
Product: Testing → Taskcluster
Target Milestone: --- → mozilla41
Version: unspecified → Trunk
Resetting Version and Target Milestone that accidentally got changed...
Target Milestone: mozilla41 → ---
Version: Trunk → unspecified
You need to log in before you can comment on or make changes to this bug.