485355 - moz2-win32-slave07 is consistently failing leak tests

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Dave Townsend [:mossop]]

No other slave seems to be having problems but 4 times in a row the m-c leak test has failed at the same point. Clobbering didn't seem to have an effect. It still might be a code problem but it seems odd that only one slave is seeing it.

It also seems to be failing the 1.9.1 leak test runs too.

Comment 1

[[e-mail] (:bhearsum)]

Here's some stats from the past few days:
m-c:
PASS - (Mar 21 17:57) rev=[8d2c566f3256]
FAIL - (Mar 24 06:24) rev=[83944488fbe6] (in alivetest_2)
PASS - (Mar 24 14:26) rev=[526a987a3b1e]
FAIL - (Mar 25 15:01) rev=[6cfd9a6df378] (in alivetest_2)
FAIL - (Mar 25 16:35) rev=[923816ab8dab] (in alivetest_2)
FAIL - (Mar 26 00:26) rev=[85bd18f6b652] (in alivetest_2)
FAIL - (Mar 26 05:59) rev=[00f9feba2406] (in alivetest_2)

m1.9.1:
PASS - (Mar 21 07:25) rev=[d01073dc8a2e]
PASS - (Mar 22 05:57) rev=[f5340979aee1]
PASS - (Mar 22 13:57) rev=[f5340979aee1]
PASS - (Mar 23 09:05) rev=[96a69edf5317]
PASS - (Mar 24 19:25) rev=[116cd74b54f9]
FAIL - (Mar 25 12:36) rev=[32f5feb0b615] (in alivetest_2)
FAIL - (Mar 25 15:45) rev=[f087a48da189] (in alivetest_4)
FAIL - (Mar 26 07:09) rev=[c6e0ec0a708c] (in alivetest_2)

Were there any checkins that were backed out of m-c, and then re-landed around the same time they went into 1.9.1? It's sure weird that this is only hitting one slave though...

Comment 2

[[e-mail] (:bhearsum)]

Ignoring 1.9.1 for a sec, http://hg.mozilla.org/mozilla-central/rev/94673272aeab is looking pretty suspicious on m-c. it was landed between the last pass and first fail on slave07, backed out between the fail and next pass, and then re-landed again before the first failure

Here's the range between the last pass and first fail:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=8d2c566f3256&tochange=83944488fbe6
Between the fail and next pass:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=83944488fbe6&tochange=526a987a3b1e
Between that pass and the next fail:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=526a987a3b1e&tochange=6cfd9a6df378



Here's the range for 1.9.1:
http://hg.mozilla.org/releases/mozilla-1.9.1/pushloghtml?fromchange=116cd74b54f9&tochange=32f5feb0b615


There's no checkins in both of the ranges, sadly.

Comment 3

[Dave Townsend [:mossop]]

(In reply to comment #2)
> Ignoring 1.9.1 for a sec,
> http://hg.mozilla.org/mozilla-central/rev/94673272aeab is looking pretty
> suspicious on m-c. it was landed between the last pass and first fail on
> slave07, backed out between the fail and next pass, and then re-landed again
> before the first failure

I've backed this out of m-c so lets see if that causes it to clear up there. We might just be dealing with two different issues.

Comment 4

[[e-mail] (:bhearsum)]

I haven't been able to reproduce this failure by running the m-c leak tests manually on moz2-win32-slave07...going to try on 1.9.1 now.

Comment 5

[[e-mail] (:bhearsum)]

I can't repro this on either branch, sadly.

Comment 6

[[e-mail] (:bhearsum)]

I've rebooted moz2-win32-slave07 and it's going to go back in the pool. We'll have to watch and see if it fails again...

Comment 7

[[e-mail] (:bhearsum)]

This seems to have resolved itself after putting the machine back in the pool...it's passed a m-c leak test with and without peterv's patch (http://hg.mozilla.org/mozilla-central/rev/94673272aeab). It hasn't done a 1.9.1 leak test again yet, though..

Comment 8

[Peter Van der Beken [:peterv]]

Note that the patch for bug 484764 just moved some code around, without really changing it. I'd be very surprised if that caused leaks.

Comment 9

[[e-mail] (:bhearsum)]

There's been multiple successful runs on 1.9.1 leak tests on this slave overnight...I don't know what caused this still, but it's FIXED.