1643825 - Some windows shippable Talos-with-profiling tasks are near perma-fail: max run time exceeded

Status: RESOLVED FIXED

(Testing :: Talos, defect, P1)

Description

[Geoff Brown [:gbrown] (pto Jul 1-19)]

Lots of failures seen in bug 1589796. Worst offenders seem to be

talos-tabswitch-profiling-e10s
talos-perf-reftest-profiling-e10s
talos-perf-reftest-singletons-profiling-e10s
talos-chrome-profiling-e10s

Comment 1

[Geoff Brown [:gbrown] (pto Jul 1-19)]

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=windows%2Cshippable%2Ctalos%2Cprofiling&fromchange=cfa4bd8e6f789fcca1de2272f0d5b11c0ded913f&tochange=2ea5446878716fe02bb5624a2740d7c82f8de13e

It looks like these tests started failing with
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&selectedTaskRun=dMNwja_tShS5HmZfoeoHcg-0&searchStr=windows%2Cshippable%2Ctalos%2Cprofiling&revision=272e3c98d0029fe73c0a57c45bc3b11ffa0e2bee

Comment 2

[Geoff Brown [:gbrown] (pto Jul 1-19)]

All of these failures appear to be due to intermittent hangs. Failures in several tasks are frequent, but sometimes succeed; when they succeed, run time is much less than the configured max-run-time. Logs from failures show long periods of inactivity until max-run-time is exhausted.

Hopefully someone familiar with Talos can investigate and fix these soon.

Comment 3

[Geoff Brown [:gbrown] (pto Jul 1-19)]

Recent failures from bug 1589796:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=305448942&repo=mozilla-central&lineNumber=962
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=305449107&repo=mozilla-central&lineNumber=3110
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=305398881&repo=mozilla-central&lineNumber=2034

Comment 4

[Geoff Brown [:gbrown] (pto Jul 1-19)]

This continues to be the #1 cause of failures reported to bug 1589796.

Comment 5

[Geoff Brown [:gbrown] (pto Jul 1-19)]

:sparky - Can you find someone to investigate this soon?

Comment 6

[Greg Mierzwinski [:sparky]]

Yes, :aionescu, can you look into those Talos failures?

Comment 7

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Yes

Comment 8

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Did some backfills to identify the first commit causing this. Still researching.

Comment 9

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Mike, can you help on this investigation, please? Seems like the tests are failing from different causes and we need in-depth knowledge about talos.

Comment 10

[Geoff Brown [:gbrown] (pto Jul 1-19)]

Frequent failures continue - see bug 1589796.

Comment 11

[Geoff Brown [:gbrown] (pto Jul 1-19)]

It looks like mconley is busy elsewhere; is there someone else who can investigate?

Or, could we consider turning off T-Prof on Windows-shippable??

Comment 12

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Fine by me.

Comment 13

[Alexandru Ionescu (needinfo me) [:alexandrui]]

Attachment: Bug 1643825 Disable T-Prof windows shippable tests r?#perftest,sparky

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/1fc40f36f6c9
Disable T-Prof windows shippable tests r=sparky,perftest-reviewers

Comment 15

[Geoff Brown [:gbrown] (pto Jul 1-19)]

Thanks much!

Comment 16

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/1fc40f36f6c9