Report on timing/duration changes for key jobs

NEW
Unassigned

Status

Release Engineering
General
3 years ago
3 months ago

People

(Reporter: coop, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Whiteboard: [reporting])

(Reporter)

Description

3 years ago
As part of the diagnosis for bug 1049227, Callek gathered some stats about recent increases in execution time for some tests on Windows. Here's an excerpt from his mail:

>>>
Some interesting numbers to correlate to windows jobs taking longer overall,
and thus the backlog:

(data taken from
https://secure.pub.build.mozilla.org/buildapi/reports/builders
--- warning loading can kill buildapi if you're not very very careful)

On July 16->July 18

Job Type:                |   Avg Runtime   | Min  Runtime    | Max Runtime
win7-ix opt "unittest"   |    18:35        |   7:13          |     37:48
win7-ix opt "talos"      |    22:58        |  10:28          |   1:02:10
win7-ix debug "unittest" |    32:35        |   8:35          |   1:25:59
win8 opt "unittest"      |    18:51        |   6:28          |   1:01:34
win8 opt "talos"         |    25:14        |  13:31          |   1:02:25
win8 debug "unittest"    |    32:06        |   7:50          |   1:31:26
wXP  opt "unittest"      |    16:15        |   3:59          |     38:49
wXP  opt "talos"         |    22:15        |  10:57          |   1:03:08
wXP  debug "unittest"    |    30:52        |   7:31          |   1:15:12
--------------------------------------------------------------------------
https://secure.pub.build.mozilla.org/buildapi/reports/builders?detail_level=job_type&platform=win7,win7-ix,win764,win8,xp,xp-ix&build_type=debug,opt&job_type=build,repack,talos,unittest&starttime=1405483200&endtime=1405569600


For ~ the past day (Aug/04->Aug/05)

Job Type:                |   Avg Runtime   | Min  Runtime    | Max Runtime
win7-ix opt "unittest"   |    25:28        |   8:16          |   1:07:57
win7-ix opt "talos"      |    27:41        |  10:41          |   1:18:26
win7-ix debug "unittest" |    42:56        |  10:26          |   1:59:19
win8 opt "unittest"      |    24:51        |   8:54          |     46:27
win8 opt "talos"         |    28:25        |  14:12          |     49:52
win8 debug "unittest"    |    41:39        |   8:53          |   1:37:24
wXP  opt "unittest"      |    22:18        |   5:35          |   1:10:13
wXP  opt "talos"         |    28:18        |  11:48          |     55:55
wXP  debug "unittest"    |    40:15        |   8:15          |   1:42:35
--------------------------------------------------------------------------
https://secure.pub.build.mozilla.org/buildapi/reports/builders?detail_level=job_type&platform=win7,win7-ix,win764,win8,xp,xp-ix&build_type=debug,opt&job_type=build,repack,talos,unittest


Now the differences
Job Type:                |   Avg Runtime Difference
win7-ix opt "unittest"   |    + 6:53   ->  + 37%
win7-ix opt "talos"      |    + 4:43   ->  + 20%
win7-ix debug "unittest" |    +10:21   ->  + 31%
win8 opt "unittest"      |    + 6:00   ->  + 53%
win8 opt "talos"         |    + 3:11   ->  + 12%
win8 debug "unittest"    |    + 9:33   ->  + 29%
wXP  opt "unittest"      |    + 6:03   ->  + 61%
wXP  opt "talos"         |    + 6:03   ->  + 45%
wXP  debug "unittest"    |    + 9:23   ->  + 30%
--------------------------------------------------------------------------
>>>

This seems like something we should be able to script up and run on a regular cadence, allowing us to spot these issues before they become acute.

We should obviously do this across all platforms, not just windows.

Depending on the cadence (weekly?), we might be able to do it for all jobs. Otherwise, we might choose just a few representative jobs per platform.
(Assignee)

Updated

3 months ago
Component: Tools → General
Product: Release Engineering → Release Engineering
You need to log in before you can comment on or make changes to this bug.