1297919 - Frequent WinXP and Win7 VM debug reftest command timed out: 7200 seconds elapsed, attempting to kill

Status: RESOLVED WORKSFORME

(Release Engineering :: General, defect)

Description

[Treeherder Bug Filer]

Filed by: philringnalda [at] gmail.com

https://treeherder.mozilla.org/logviewer.html#?job_id=34608403&repo=mozilla-inbound

https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-inbound-win32-debug/1472079626/mozilla-inbound_xp_ix-debug_test-reftest-bm127-tests1-windows-build356.txt.gz

https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-inbound-win32-debug/1472079626/mozilla-inbound_xp_ix-debug_test-reftest-bm127-tests1-windows-build356.txt.gz&only_show_unexpected=1

Comment 1

[Phil Ringnalda (:philor)]

That summary confused me when I read it ten minutes later. The things we're going to have to chunk, even though we can't afford to chunk them, are WinXP debug reftest, Win7 VM debug reftest, and Win7 VM debug reftest-no-accel.

Comment 2

[Astley Chen (inactive)]

could it be possible to chunk them into smaller pieces of test group to avoid timeout before running cover ?

Comment 3

[Astley Chen (inactive)]

also is it a dup of bug 1207900 ?

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

odd, we have opt tests running in ~30 minutes and debug in ~120 minutes- that is crazy.  There is no real reason for this other than each test takes much longer in debug, many of the tests complete in <2 seconds (marked as 1 second on the wall clock).  

We can easily turn this into 8 chunks, in fact we already do that on linux. The danger of having a 120minute timeout is that if something small goes wrong, we waste a lot of machine time waiting for the job to timeout.  If we could split this 8 ways our runtime would be <15 minutes and we could have a 1200second timeout (or 1800 second).   iagree it would not be as optimal in typical runs, but it should help in intermittent and failure cases.

Comment 5

[Phil Ringnalda (:philor)]

On WinXP, you have about a 4 minute lag between test runs, rebooting and whatever else happens, and from the one green run I found to look at, 7 minutes of startup and teardown within the test run for reftest. So going from one chunk to eight would add 77 minutes of waste every time we run reftests on WinXP debug.

Comment 6

[Intermittent Failures Robot]

40 automation job failures were associated with this bug yesterday.

Repository breakdown:
* mozilla-inbound: 30
* mozilla-central: 5
* autoland: 3
* try: 2

Platform breakdown:
* windows7-32-vm: 18
* windowsxp: 17
* windows8-64: 2
* windows7-32: 2
* android-4-3-armv7-api15: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1297919&startday=2016-08-25&endday=2016-08-25&tree=all

Comment 7

[Intermittent Failures Robot]

55 automation job failures were associated with this bug in the last 7 days.

Repository breakdown:
* mozilla-inbound: 42
* mozilla-central: 5
* try: 4
* autoland: 3
* fx-team: 1

Platform breakdown:
* windowsxp: 25
* windows7-32-vm: 23
* windows7-32: 4
* windows8-64: 2
* android-4-3-armv7-api15: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1297919&startday=2016-08-22&endday=2016-08-28&tree=all

Comment 8

[Phil Ringnalda (:philor)]

Bug 1292218 took these from ~90 minutes to ~110, eating up our margin.

Comment 9

[Phil Ringnalda (:philor)]

Backed that out, we shouldn't see this again for a while.

Comment 10

[Intermittent Failures Robot]

1 failures in 521 pushes (0.002 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-esr52: 1

Platform breakdown:
* windows8-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1297919&startday=2018-06-18&endday=2018-06-24&tree=trunk