767427 - Slaves not clobbering when marked as needing one in the clobberer

Status: RESOLVED WORKSFORME

(Release Engineering :: General, defect, P3)

Description

[Ed Morley [:emorley]]

On mozilla-central, bld-lion-r5-079 just busted a build for something that we knew needed a clobber:
https://tbpl.mozilla.org/php/getParsedLog.php?id=12904192&tree=Firefox

That slave is marked as needing a clobber in the clobberer:
https://build.mozilla.org/clobberer/?branch=mozilla-central

"2012-06-17 06:33:35 PDT by ryanvm@<snip>"

...but yet wasn't force clobbered.

In addition, the clobberer page shows an awful lot of slaves being marked as needing a clobber (and has done for a week or two now off the top of my head) - whereas before the "last clobbered" field would quickly clear as slaves were used.

CCing a few people who might be using the clobberer, to avoid future confusion over clobber-needed bustage.

========= Started checking clobber times (results: 0, elapsed: 0 secs) (at 2012-06-22 08:36:39.904982) =========
python m-cen-osx-dbg/tools/clobberer/clobberer.py -s tools -t 168 http://build.mozilla.org/clobberer/index.php mozilla-central 'OS X 10.7 32-bit mozilla-central leak test build' m-cen-osx-dbg bld-lion-r5-079 http://buildbot-master34.srv.releng.scl3.mozilla.com:8001/
 in dir /builds/slave/m-cen-osx-dbg/.. (timeout 3600 secs)
 watching logfiles {}
 argv: ['python', 'm-cen-osx-dbg/tools/clobberer/clobberer.py', '-s', 'tools', '-t', '168', 'http://build.mozilla.org/clobberer/index.php', 'mozilla-central', 'OS X 10.7 32-bit mozilla-central leak test build', 'm-cen-osx-dbg', 'bld-lion-r5-079', u'http://buildbot-master34.srv.releng.scl3.mozilla.com:8001/']
 environment:
  Apple_PubSub_Socket_Render=/tmp/launch-0tp3F8/Render
  CVS_RSH=ssh
  DISPLAY=/tmp/launch-XbrGbp/org.x:0
  HOME=/Users/cltbld
  LOGNAME=cltbld
  PATH=/tools/buildbot/bin:/tools/python/bin:/opt/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin
  PWD=/builds/slave
  PYTHONPATH=/tools/buildbot/lib/python2.7/site-packages
  SHELL=/bin/bash
  SSH_AUTH_SOCK=/tmp/launch-3QT8k8/Listeners
  TMPDIR=/var/folders/30/yq_p3wk15yb9wdsv6sm_m0v00000gn/T/
  USER=cltbld
  VERSIONER_PYTHON_PREFER_32_BIT=no
  VERSIONER_PYTHON_VERSION=2.7
  __CF_USER_TEXT_ENCODING=0x1F5:0:0
 using PTY: False
Checking clobber URL: http://build.mozilla.org/clobberer/index.php?master=http%3A%2F%2Fbuildbot-master34.srv.releng.scl3.mozilla.com%3A8001%2F&slave=bld-lion-r5-079&builddir=m-cen-osx-dbg&branch=mozilla-central&buildername=OS+X+10.7+32-bit+mozilla-central+leak+test+build
m-aurora-osx-dbg:Our last clobber date:  2012-06-20 21:32:53
m-aurora-osx-dbg:Server clobber date:    2012-06-04 14:47:11
m-in-osx64:Our last clobber date:  2012-06-21 07:52:13
m-in-osx64:Server clobber date:    2012-06-21 07:52:13
m-cen-osx64:Our last clobber date:  2012-06-17 06:33:35
m-cen-osx64:Server clobber date:    2012-06-17 06:33:35
tb-comm-aurora-osx64-dbg:Our last clobber date:  2012-06-12 06:16:58
tb-comm-aurora-osx64-dbg:Server clobber date:    2012-06-04 13:10:11
m-beta-osx64-dbg:Our last clobber date:  2012-06-08 13:34:46
m-beta-osx64-dbg:Server clobber date:    2012-06-08 13:34:46
profiling-osx64-ntly:Our last clobber date:  2012-06-19 01:54:26
profiling-osx64-ntly:Server clobber date:    2012-06-19 01:54:26
m-in-osx64-lion-dbg-spidermonkey-nomethodjit:Our last clobber date:  2012-06-21 07:52:13
m-in-osx64-lion-dbg-spidermonkey-nomethodjit:Server clobber date:    2012-06-21 07:52:13
m-in-osx64-dbg:Our last clobber date:  2012-06-21 07:52:13
m-in-osx64-dbg:Server clobber date:    2012-06-21 07:52:13
ionmonkey-osx64-lion-dbg-spidermonkey-nomethodjit:Our last clobber date:  2012-06-08 02:33:56
ionmonkey-osx64-lion-dbg-spidermonkey-nomethodjit:Server clobber date:    2012-06-08 02:33:56
tb-comm-beta-osx64:Our last clobber date:  2012-06-18 11:48:06
tb-comm-beta-osx64:Server clobber date:    2012-06-18 11:48:06
srv-cen-osx64-dbg:Our last clobber date:  2012-06-13 09:02:01
srv-cen-osx64-dbg:Server clobber date:    2012-06-13 09:02:01
m-cen-osx64-xr:Our last clobber date:  2012-06-22 04:14:23
m-cen-osx64-xr:Server clobber date:    2012-06-22 04:14:23
m-aurora-osx64-ntly:Our last clobber date:  2012-06-12 04:20:32
m-aurora-osx64-ntly:Server clobber date:    2012-06-04 14:47:11
m-cen-osx-dbg:Our last clobber date:  2012-06-17 06:33:35
m-cen-osx-dbg:Server clobber date:    2012-06-17 06:33:35
m-cen-osx64-l10n-ntly:Our last clobber date:  2012-06-22 04:14:23
m-cen-osx64-l10n-ntly:Server clobber date:    2012-06-22 04:14:23
m-in-osx-dbg:Our last clobber date:  2012-06-21 07:52:13
m-in-osx-dbg:Server clobber date:    2012-06-21 07:52:13
pine-osx64:Our last clobber date:  2012-06-12 11:29:52
pine-osx64:Server clobber date:    2012-06-12 11:29:52
pine-osx64-dbg:Our last clobber date:  2012-06-20 14:28:02
pine-osx64-dbg:Server clobber date:    2012-06-12 11:29:52
ionmonkey-osx64:Our last clobber date:  None
ionmonkey-osx64:Server clobber date:    2012-06-08 02:33:56
fx-team-osx64:Our last clobber date:  2012-06-19 01:54:17
fx-team-osx64:Server clobber date:    2012-06-19 01:54:17
srv-cen-osx-dbg:Our last clobber date:  2012-06-13 09:02:01
srv-cen-osx-dbg:Server clobber date:    2012-06-13 09:02:01
ionmonkey-osx64-dbg:Our last clobber date:  2012-06-08 02:33:56
ionmonkey-osx64-dbg:Server clobber date:    2012-06-08 02:33:56
tb-comm-cen-osx64:Our last clobber date:  2012-06-19 15:23:03
tb-comm-cen-osx64:Server clobber date:    2012-06-19 15:23:03
m-in-osx64-lion-dbg-spidermonkey-warnaserrdebug:Our last clobber date:  2012-06-21 07:52:13
m-in-osx64-lion-dbg-spidermonkey-warnaserrdebug:Server clobber date:    2012-06-21 07:52:13
m-in-osx64-ntly:Our last clobber date:  2012-06-21 07:52:13
m-in-osx64-ntly:Server clobber date:    2012-06-21 07:52:13
profiling-osx64:Our last clobber date:  2012-06-19 01:54:26
profiling-osx64:Server clobber date:    2012-06-19 01:54:26
program finished with exit code 0
elapsedTime=0.747198
========= Finished checking clobber times (results: 0, elapsed: 0 secs) (at 2012-06-22 08:36:40.720191) =========

Comment 1

[Rail Aliiev [:rail]]

I suspect that it didn't clobber because it listed twice in the list:
http://people.mozilla.org/~raliiev/sattap/fbe558fa.png

Investigating.

Comment 2

[Ed Morley [:emorley]]

Ah good spot :-)

Did you find anything more out about this?

Comment 3

[Ed Morley [:emorley]]

Any news? :-)

Comment 4

[Chris Cooper [:coop] (he/him)]

Assigning to Rail based on comment #1.

Comment 5

[Rail Aliiev [:rail]]

I believe that this bug was fixed by removing slave-specific clobberer settings in bug 756532.

I've been monitoring this behavior by querying the clobberer from time to time. So far it works as expected.

Feel free to reopen the bug if this happens again.