1168928 - [raptor] Improve raptor test task duration consistency

Status: RESOLVED FIXED

(Taskcluster :: General, defect)

Description

[Robert Wood [:rwood]]

There seems to be a large variance in the runtime duration for raptor test tasks, even though they are all running on the same 'raptor' worker type.

For example, on gaia, this raptor launch test task took 30 minutes to run:
http://docs.taskcluster.net/tools/task-inspector/#KG-2axH1Qdehyj6SXKpT0w/0

And this task, that runs the exact same test, took 15 minutes to run:
http://docs.taskcluster.net/tools/task-inspector/#TSjR6xhWR9qoHaDpBK6r7Q/0

Another example, on gaia-master, this test task took 49 minutes:
http://docs.taskcluster.net/tools/task-inspector/#tI7eHHDQRB-g4n-3QJkhbA/0

And this task that runs the exact same test:
http://docs.taskcluster.net/tools/task-inspector/#-iBfYubvRN6wUW7Fjelc4w/0

James noted it possibly might have to do with having 'isolated containers' set to true in the 'raptor' worker type. Try running raptor test tasks in parallel on the 'raptor-single-capacity' worker type James created.

Comment 1

[Robert Wood [:rwood]]

Wow, when the raptor test task duration is fast the launch results are noticeably faster.

Launch test task, duration 41 minutes, p95 visuallyLoaded median result: 16660
http://docs.taskcluster.net/tools/task-inspector/#T3sjA3vIRKirwp1zojBysQ/0

Same launch test in same suite, next task, duration 24 minutes, p95 visuallyLoaded median result: 4419
http://docs.taskcluster.net/tools/task-inspector/#YaccmH4oSwiW0sjeZLyC2A/1

Will try the raptor-single-capacity worker type...

Comment 2

[GitHub Autolander Bot]

Attachment: [gaia] rwood-moz:bug1168928 > mozilla-b2g:master

Comment 3

[Robert Wood [:rwood]]

Wow that's pretty cool, huge improvement:

https://gist.github.com/rwood-moz/297455942d75fb4bfca3

Going to switch the raptor tasks from the 'raptor' worker to the 'raptor-single-capacity' worker.

Comment 4

[Robert Wood [:rwood]]

There is a little bit more variation (but not too badly) in the test tasks for the launch test on 'test-startup-limit' app. However it doesn't seem to effect the actual numbers.

Example, this task took 21 minutes:
http://docs.taskcluster.net/tools/task-inspector/#i8klciIBRMidWh8JkgfHoQ/0, p95 visuallyLoaded: 3723

Same test, this task took 11 minutes:
http://docs.taskcluster.net/tools/task-inspector/#CPaxen6jR9iBMG1Wjfp51Q/0, p95 visuallyLoaded: 3739

However the duration didn't effect the actual app launch times - the task that ran faster actually had slightly longer app launch times. Most of the test tasks on that app are pretty consistent around the 11 or 15 minute mark. I'd say still a good move to change to the 'raptor-single-capacity' worker type.

Comment 5

[GitHub Autolander Bot]

Attachment: [gaia] rwood-moz:bug1168928-worker > mozilla-b2g:master

Comment 6

[Greg Arndt [:garndt]]

Comment on attachment 8612910 [details] [review]
[gaia] rwood-moz:bug1168928-worker > mozilla-b2g:master

comment in the pull, nothing major or that needs to be addressed at this time

Comment 7

[Robert Wood [:rwood]]

Thanks Greg. Landed:
https://github.com/mozilla-b2g/gaia/commit/823be2cd940d10ebd1e4e1455c4bd50a3969f896

Comment 8

[Pete Moore [:pmoore][:pete]]

Resetting Version and Target Milestone that accidentally got changed...