Closed Bug 1050594 Opened 10 years ago Closed 6 years ago

Report on timing/duration changes for key jobs

Categories

(Release Engineering :: General, defect)

x86
macOS
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: coop, Unassigned)

References

Details

(Whiteboard: [reporting])

As part of the diagnosis for bug 1049227, Callek gathered some stats about recent increases in execution time for some tests on Windows. Here's an excerpt from his mail:

>>>
Some interesting numbers to correlate to windows jobs taking longer overall,
and thus the backlog:

(data taken from
https://secure.pub.build.mozilla.org/buildapi/reports/builders
--- warning loading can kill buildapi if you're not very very careful)

On July 16->July 18

Job Type:                |   Avg Runtime   | Min  Runtime    | Max Runtime
win7-ix opt "unittest"   |    18:35        |   7:13          |     37:48
win7-ix opt "talos"      |    22:58        |  10:28          |   1:02:10
win7-ix debug "unittest" |    32:35        |   8:35          |   1:25:59
win8 opt "unittest"      |    18:51        |   6:28          |   1:01:34
win8 opt "talos"         |    25:14        |  13:31          |   1:02:25
win8 debug "unittest"    |    32:06        |   7:50          |   1:31:26
wXP  opt "unittest"      |    16:15        |   3:59          |     38:49
wXP  opt "talos"         |    22:15        |  10:57          |   1:03:08
wXP  debug "unittest"    |    30:52        |   7:31          |   1:15:12
--------------------------------------------------------------------------
https://secure.pub.build.mozilla.org/buildapi/reports/builders?detail_level=job_type&platform=win7,win7-ix,win764,win8,xp,xp-ix&build_type=debug,opt&job_type=build,repack,talos,unittest&starttime=1405483200&endtime=1405569600


For ~ the past day (Aug/04->Aug/05)

Job Type:                |   Avg Runtime   | Min  Runtime    | Max Runtime
win7-ix opt "unittest"   |    25:28        |   8:16          |   1:07:57
win7-ix opt "talos"      |    27:41        |  10:41          |   1:18:26
win7-ix debug "unittest" |    42:56        |  10:26          |   1:59:19
win8 opt "unittest"      |    24:51        |   8:54          |     46:27
win8 opt "talos"         |    28:25        |  14:12          |     49:52
win8 debug "unittest"    |    41:39        |   8:53          |   1:37:24
wXP  opt "unittest"      |    22:18        |   5:35          |   1:10:13
wXP  opt "talos"         |    28:18        |  11:48          |     55:55
wXP  debug "unittest"    |    40:15        |   8:15          |   1:42:35
--------------------------------------------------------------------------
https://secure.pub.build.mozilla.org/buildapi/reports/builders?detail_level=job_type&platform=win7,win7-ix,win764,win8,xp,xp-ix&build_type=debug,opt&job_type=build,repack,talos,unittest


Now the differences
Job Type:                |   Avg Runtime Difference
win7-ix opt "unittest"   |    + 6:53   ->  + 37%
win7-ix opt "talos"      |    + 4:43   ->  + 20%
win7-ix debug "unittest" |    +10:21   ->  + 31%
win8 opt "unittest"      |    + 6:00   ->  + 53%
win8 opt "talos"         |    + 3:11   ->  + 12%
win8 debug "unittest"    |    + 9:33   ->  + 29%
wXP  opt "unittest"      |    + 6:03   ->  + 61%
wXP  opt "talos"         |    + 6:03   ->  + 45%
wXP  debug "unittest"    |    + 9:23   ->  + 30%
--------------------------------------------------------------------------
>>>

This seems like something we should be able to script up and run on a regular cadence, allowing us to spot these issues before they become acute.

We should obviously do this across all platforms, not just windows.

Depending on the cadence (weekly?), we might be able to do it for all jobs. Otherwise, we might choose just a few representative jobs per platform.
Component: Tools → General
buildbot is dying!
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.