764020 - B2G builds don't generate the {forced, periodic, free-space} clobber state TinderboxPrint lines since they use mozharness

Status: RESOLVED FIXED

(Release Engineering :: Applications: MozharnessCore, defect, P3)

Description

[Ed Morley [:emorley]]

B2G:
{
========= Started checking clobber times (results: 0, elapsed: 20 secs) (at 2012-06-12 06:36:35.887201) =========
python m-cen-b2g/tools/clobberer/clobberer.py -s tools -t 168 http://build.mozilla.org/clobberer/index.php mozilla-central 'b2g fedora16-i386 mozilla-central build' m-cen-b2g bld-centos6-hp-016 http://buildbot-master13.build.scl1.mozilla.com:8001/
 in dir /builds/slave/m-cen-b2g/.. (timeout 3600 secs)
 watching logfiles {}
 argv: ['python', 'm-cen-b2g/tools/clobberer/clobberer.py', '-s', 'tools', '-t', '168', 'http://build.mozilla.org/clobberer/index.php', 'mozilla-central', 'b2g fedora16-i386 mozilla-central build', 'm-cen-b2g', 'bld-centos6-hp-016', u'http://buildbot-master13.build.scl1.mozilla.com:8001/']
 environment:
  CCACHE_HASHDIR=
  G_BROKEN_FILENAMES=1
  HISTCONTROL=ignoredups
  HISTSIZE=1000
  HOME=/home/cltbld
  HOSTNAME=bld-centos6-hp-016.build.scl1.mozilla.com
  LANG=en_US.UTF-8
  LESSOPEN=|/usr/bin/lesspipe.sh %s
  LOGNAME=cltbld
  MAIL=/var/spool/mail/cltbld
  PATH=/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin::/tools/git/bin/:/tools/python27/bin/:/tools/python27-mercurial/bin::/tools/git/bin/:/tools/python27/bin/:/tools/python27-mercurial/bin:/home/cltbld/bin
  PWD=/builds/slave
  SHELL=/bin/bash
  SHLVL=1
  TERM=linux
  USER=cltbld
  _=/tools/buildbot/bin/python
 using PTY: False
Checking clobber URL: http://build.mozilla.org/clobberer/index.php?master=http%3A%2F%2Fbuildbot-master13.build.scl1.mozilla.com%3A8001%2F&slave=bld-centos6-hp-016&builddir=m-cen-b2g&branch=mozilla-central&buildername=b2g+fedora16-i386+mozilla-central+build
profiling-b2g:Our last clobber date:  2012-06-10 02:48:14
profiling-b2g:Server clobber date:    2012-06-12 06:00:29
profiling-b2g:Server is forcing a clobber, initiated by [e-mail]
profiling-b2g:Clobbering...
Skipping last-clobber
Skipping tools
Removing buildprops.json
Removing build/
m-cen-b2g-ntly:Our last clobber date:  None
m-cen-b2g-ntly:Server clobber date:    2012-06-10 17:23:04
srv-cen-b2g:Our last clobber date:  2012-06-07 13:17:59
srv-cen-b2g:Server clobber date:    2012-06-07 13:17:59
m-in-b2g-ntly:Our last clobber date:  2012-06-11 18:21:04
m-in-b2g-ntly:Server clobber date:    2012-06-11 18:21:04
m-in-b2g:Our last clobber date:  2012-06-11 18:21:04
m-in-b2g:Server clobber date:    2012-06-11 18:21:04
m-cen-b2g:Our last clobber date:  2012-06-10 17:23:04
m-cen-b2g:Server clobber date:    2012-06-10 17:23:04
bld-system-b2g:Our last clobber date:  None
bld-system-b2g:Server clobber date:    2012-05-26 19:48:03
program finished with exit code 0
elapsedTime=20.774374
========= Finished checking clobber times (results: 0, elapsed: 20 secs) (at 2012-06-12 06:36:56.677116) =========
}

In comparison, on other platforms:
{
========= Started checking clobber times (results: 0, elapsed: 0 secs) (at 2012-06-12 06:36:43.732272) =========
python m-cen-lnx64/tools/clobberer/clobberer.py -s tools -t 168 http://build.mozilla.org/clobberer/index.php mozilla-central 'Linux x86-64 mozilla-central build' m-cen-lnx64 bld-centos5-64-vmw-004 http://buildbot-master30.srv.releng.scl3.mozilla.com:8001/
 in dir /builds/slave/m-cen-lnx64/.. (timeout 3600 secs)
 watching logfiles {}
 argv: ['python', 'm-cen-lnx64/tools/clobberer/clobberer.py', '-s', 'tools', '-t', '168', 'http://build.mozilla.org/clobberer/index.php', 'mozilla-central', 'Linux x86-64 mozilla-central build', 'm-cen-lnx64', 'bld-centos5-64-vmw-004', u'http://buildbot-master30.srv.releng.scl3.mozilla.com:8001/']
 environment:
  CCACHE_HASHDIR=
  CVS_RSH=ssh
  G_BROKEN_FILENAMES=1
  HISTSIZE=1000
  HOME=/home/cltbld
  HOSTNAME=bld-centos5-64-vmw-004.build.releng.scl3.mozilla.com
  INPUTRC=/etc/inputrc
  LANG=en_US.UTF-8
  LESSOPEN=|/usr/bin/lesspipe.sh %s
  LOGNAME=cltbld
  LS_COLORS=no=00:fi=00:di=01;34:ln=01;36:pi=40;33:so=01;35:bd=40;33;01:cd=40;33;01:or=01;05;37;41:mi=01;05;37;41:ex=01;32:*.cmd=01;32:*.exe=01;32:*.com=01;32:*.btm=01;32:*.bat=01;32:*.sh=01;32:*.csh=01;32:*.tar=01;31:*.tgz=01;31:*.arj=01;31:*.taz=01;31:*.lzh=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.gz=01;31:*.bz2=01;31:*.bz=01;31:*.tz=01;31:*.rpm=01;31:*.cpio=01;31:*.jpg=01;35:*.gif=01;35:*.bmp=01;35:*.xbm=01;35:*.xpm=01;35:*.png=01;35:*.tif=01;35:
  MAIL=/var/spool/mail/cltbld
  PATH=/opt/local/bin:/tools/python/bin:/tools/buildbot/bin:/usr/kerberos/bin:/usr/lib64/ccache:/usr/local/bin:/bin:/usr/bin:/home/cltbld/bin
  PWD=/builds/slave
  SHELL=/bin/bash
  SHLVL=1
  SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
  TERM=linux
  USER=cltbld
  _=/tools/python/bin/python
 using PTY: False
Checking clobber URL: http://build.mozilla.org/clobberer/index.php?master=http%3A%2F%2Fbuildbot-master30.srv.releng.scl3.mozilla.com%3A8001%2F&slave=bld-centos5-64-vmw-004&builddir=m-cen-lnx64&branch=mozilla-central&buildername=Linux+x86-64+mozilla-central+build
tb-comm-aurora-lnx64-dbg:Our last clobber date:  2012-06-11 11:26:41
...
<snip>
...
m-cen-lnx64-l10n-ntly:Server clobber date:    2012-06-10 17:19:47
program finished with exit code 0
elapsedTime=0.181946
TinderboxPrint: free-space clobber
========= Finished checking clobber times (results: 0, elapsed: 0 secs) (at 2012-06-12 06:36:43.966224) =========
}

Comment 1

[Chris AtLee [:catlee]]

it also doesn't set the usual clobber related buildbot properties.

Comment 2

[Chris AtLee [:catlee]]

The PurgeMixin (http://hg.mozilla.org/build/mozharness/file/caf7d02f45df/mozharness/mozilla/purge.py) is probably the best place to do this. It could parse the output of purge/clobberer and do the appropriate TinderboxPrint statements.

Alternatively, external_tools/clobberer.py itself could be updated to do the TinderboxPrint.

Comment 3

[Ed Morley [:emorley]]

For reference, the lines are output by:
https://hg.mozilla.org/build/buildbotcustom/file/895ab6ae9cfa/steps/misc.py#l416

Comment 4

[Ed Morley [:emorley]]

More recent logs...

Desktop forced clobber:
https://tbpl.mozilla.org/php/getParsedLog.php?id=27204145&tree=Mozilla-Central

B2G forced clobber (search for 'Running clobber step'):
https://tbpl.mozilla.org/php/getParsedLog.php?id=27205754&full=1&branch=mozilla-central

Comment 5

[Ed Morley [:emorley]]

Desktop eg:
m-cen-osx64-d-0000000000000000:Server is forcing a clobber, initiated by [e-mail]
m-cen-osx64-l10n-ntly-00000000:Server is forcing a clobber, initiated by [e-mail]

B2G eg:
b2g_m-cen_inari_eng_ntly-00000:Server is forcing a clobber, initiated by [e-mail]
b2g_fx-team_unagi_dep-00000000:Server is forcing a clobber, initiated by [e-mail]

Comment 6

[Ed Morley [:emorley]]

Huh we don't set builddir on B2G jobs yey...

(Grabbed from looking at the buildbot master for some more random jobs)

Desktop:
http://buildbot-master62.srv.releng.use1.mozilla.com:8001/builders/OS%20X%2010.7%20mozilla-central%20build/builds/74
builddir 	m-cen-osx64-000000000000000000

B2G:
http://buildbot-master66.srv.releng.usw2.mozilla.com:8001/builders/b2g_mozilla-central_emulator_dep/builds/57
(builddir not set)

Comment 7

[Ed Morley [:emorley]]

For desktop builds the builddir property is set by:
https://hg.mozilla.org/build/buildbotcustom/file/895ab6ae9cfa/process/factory.py#l456

We're obviously just not doing this step for B2G builds.

Comment 8

[Ed Morley [:emorley]]

That's about as far as my knowledge goes - you'll need to ask releng as to the best way to add that step to B2G builds - which presumably intentionally miss out a lot of the normal desktop steps, including (accidentally) this one.

Comment 9

[Chris AtLee [:catlee]]

this is primarily a problem due to 2 different implementations of the same thing. the older buildbot factories have this logic, but the newer mozharness scripts do not. you can't really copy/paste from the factory into the mozharness script, but neither are these complicated issues to code from scratch in mozharness.

Comment 10

[Ed Morley [:emorley]]

I'm thinking this approach might just be the best then:

(In reply to Chris AtLee [:catlee] from comment #2)
> Alternatively, external_tools/clobberer.py itself could be updated to do the
> TinderboxPrint.

ie add the TinderboxPrint to both copies of clobberer.py:
https://hg.mozilla.org/build/mozharness/file/acf79f55ede2/external_tools/clobberer.py#l237
https://hg.mozilla.org/build/tools/file/c072db21b01b/clobberer/clobberer.py#l237

And just strip out:
https://hg.mozilla.org/build/buildbotcustom/file/895ab6ae9cfa/steps/misc.py#l385

Comment 11

[Ed Morley [:emorley]]

Attachment: buildbotcustom v1

Comment 12

[Ed Morley [:emorley]]

Attachment: Tools clobberer.py v1

Comment 13

[Ed Morley [:emorley]]

Attachment: mozharness clobberer.py v1

Comment 14

[Ed Morley [:emorley]]

Tools & mozharness parts:
https://hg.mozilla.org/build/tools/rev/8225558b8f88
https://hg.mozilla.org/build/mozharness/rev/3dfee304e19a

Comment 15

[[e-mail] (:bhearsum)]

The mozharness patch is in production.

Comment 16

[Ed Morley [:emorley]]

Patch awaiting review

Comment 17

[Ed Morley [:emorley]]

The landed parts seem to be working well, eg:
https://tbpl.mozilla.org/php/getLogExcerpt.php?id=27432658&type=tinderbox_print
for https://tbpl.mozilla.org/php/getParsedLog.php?id=27432658&tree=Mozilla-Inbound

Comment 18

[Chris AtLee [:catlee]]

Comment on attachment 798032 [details] [diff] [review]
buildbotcustom v1

the clobber properties are useful to keep, and I think we should make the mozharness side set them as well. createSummary() here will need to be adjusted to look for the new clobberer.py output and set the appropriate properties.

Comment 19

[Ed Morley [:emorley]]

Attachment: buildbotcustom v2

* Remove redundant clobber summary now that clobberer.py is outputting the TinderboxPrint lines.
* Set the properties based on the new clobberer.py output + clean up in general.

Comment 20

[Ed Morley [:emorley]]

Attachment: buildbotcustom v2.1

+ Tweak variable names slightly.

Comment 21

[Chris AtLee [:catlee]]

Comment on attachment 811119 [details] [diff] [review]
buildbotcustom v2.1

Review of attachment 811119 [details] [diff] [review]:
-----------------------------------------------------------------

::: steps/misc.py
@@ +382,5 @@
>          master = build.builder.botmaster.parent.buildbotURL
>          self.setProperty('master', master)
>  
>      def createSummary(self, log):
> +        clobberTypeRE = re.compile(r"TinderboxPrint: ([\w]+) clobber")

I was about to r- this, but instead decided to test it out instead. TIL you can use \w (and other character classes) inside []!

Comment 22

[Ed Morley [:emorley]]

(In reply to Chris AtLee [:catlee] from comment #21)
> Comment on attachment 811119 [details] [diff] [review]
> buildbotcustom v2.1
>
> I was about to r- this, but instead decided to test it out instead. TIL you
> can use \w (and other character classes) inside []!

TIL you can use character classes in Python outside the []! 
Landed without them, since looks tidier :-)

https://hg.mozilla.org/build/buildbotcustom/rev/a0b3c6b24d3e

Comment 23

[John Hopkins (:jhopkins)]

in production