1478811 - Linux opt-talos-damp-e10s is nearing its max-run-time

Status: RESOLVED FIXED

(Testing :: Talos, defect)

Description

[Geoff Brown [:gbrown]]

In bug 1469720 I am monitoring task run-times in hopes of avoiding task time outs. Queries of ActiveData suggest that Linux Talos-damp tasks are running close to their configured max-run-time of 2700 seconds (45 minutes). A quick spot-check of mozilla-central damp tasks in treeherder show run times around 40 minutes.

Long running test tasks on mozilla-central,mozilla-inbound,autoland between 2018-07-19 and 2018-07-26
test-linux64-qr/opt-talos-damp-e10s                    : 39 of 270 runs (14.4%) exceeded 90% of max-run-time (2700 s)
test-linux64/opt-talos-damp-e10s                       : 2 of 207 runs (1.0%) exceeded 90% of max-run-time (2700 s)


Talos-damp has a max-run-time of 5400 seconds on other platforms:

https://dxr.mozilla.org/mozilla-central/rev/085cdfb90903d4985f0de1dc7786522d9fb45596/taskcluster/ci/test/talos.yml#82-83


This is an opportunity to consider increasing Linux Talos-damp max-run-time to avoid intermittent task time outs.

Comment 1

[Geoff Brown [:gbrown]]

Most linux damp tasks are still not failing, but remain close to the max-run-time.

Long running test tasks on mozilla-central,mozilla-inbound,autoland between 2018-07-25 and 2018-08-01
test-linux64-qr/opt-talos-damp-e10s                    : 79 of 293 runs (27.0%) exceeded 90% of max-run-time (2700 s)
test-linux64/opt-talos-damp-e10s                       : 4 of 194 runs (2.1%) exceeded 90% of max-run-time (2700 s)

There are a few failures, like

https://treeherder.mozilla.org/logviewer.html#?job_id=191374935&repo=autoland&lineNumber=10220

Comment 2

[Geoff Brown [:gbrown]]

Attachment: use the same max-run-time (5400 seconds/90 minutes) for damp on all platforms

Comment 3

[Robert Wood [:rwood]]

Comment on attachment 8996835 [details] [diff] [review]
use the same max-run-time (5400 seconds/90 minutes) for damp on all platforms

Awesome thanks this should reduce some intermittents.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/c918e687d594
Increase max-run-time for linux talos damp; r=rwood

Comment 5

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/c918e687d594