657024 - access clobberer internal interface via a cname

Status: RESOLVED FIXED

(Release Engineering :: General, enhancement, P4)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

clobberer currently accesses http://build.mozilla.org directly from the slaves to see if a clobber is necessary (c.f. bug 447972 and bug 599580).

Since we want to move services away from build.mozilla.org as a hostname, and also make clobberer something we can rehost at will, we should reconfigure the slaves to look to a CNAME instead.  I propose

  clobberer-api.build.mozilla.org

such that http(s)://clobberer-api.build.mozilla.org/FOO would be equivalent to the current http(s)://build.mozilla.org/clobberer/FOO.

Comment 1

[Chris Cooper [:coop] (he/him)]

Dustin: is this still desirable from an IT perspective?

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

Yes, although it should be
  http://clobberer-api.pvt.build.mozilla.org
and, for the UI
  https://secure.pub.build.mozilla.org/clobberer
  (with a redirect from http://clobberer.pub.build.mozilla.org if desired)

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Per bug 758615, the API is now also available at

 http://clobberer.pvt.build.mozilkla.org *
 http://clobberer-stage.pvt.build.mozilla.org *
 http://clobberer-preproduction.pvt.build.mozilla.org *
* typos corrected

please make the necessary changes to access clobberer from buildslaves at those URLs.  These URLs are served from the same backend MySQL database as the existing URLs, so there's no need to be careful about coordinating this change.

There's not a hurry here, but can we un-p4-enhancement this, so that it gets done at least a week before never? :)

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

 http://clobberer.pvt.build.mozilla.org **
geez.

Comment 6

[Chris AtLee [:catlee]]

I think this needs changing:
http://hg.mozilla.org/build/buildbot-configs/file/12891f876428/mozilla/production_config.py#l74

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

This is blocking turning off the old clobberer, which is causing confusion like bug 772374.

Comment 8

[Dustin J. Mitchell [:dustin] (he/him)]

triple-checked for lack of typos:
 http://clobberer.pvt.build.mozilla.org
 http://clobberer-stage.pvt.build.mozilla.org
 http://clobberer-preproduction.pvt.build.mozilla.org

Comment 9

[Chris AtLee [:catlee]]

Attachment: use new clobberer cnames

ack -l clobberer | xargs sed -i 's,http://build.mozilla.org/clobberer,http://clobberer.pvt.build.mozilla.org,g'

ack -l stage-clobberer | xargs sed -i 's,http://build.mozilla.org/stage-clobberer,http://clobberer-stage.pvt.build.mozilla.org,g'

ack -l preproduction-clobberer | xargs sed -i 's,http://build.mozilla.org/preproduction-clobberer,http://clobberer-preproduction.pvt.build.mozilla.org,g'

Comment 10

[Dustin J. Mitchell [:dustin] (he/him)]

I'm still seeing lots of access to this on build.mozilla.org/clobberer.  Did something get missed?

access_2012-07-22-23:10.12.52.20 - - [22/Jul/2012:16:02:47 -0700] "GET /clobberer/index.php?master=http%3A%2F%2Fbuildbot-master14.build.scl1.mozilla.com%3A8101%2F&slave=bld-centos6-hp-025&builddir=try-gb-armv7a-gecko-dbg&branch=try&buildername=B2G+gb_armv7a_gecko-debug+try+build HTTP/1.1" 200 - "-" "Python-urllib/2.6"

Comment 11

[Nick Thomas [:nthomas] (UTC+12)]

There's this
  http://mxr.mozilla.org/build/search?string=build.mozilla.org/clobberer
but that doesn't apply to 10.12.52.20 aka bld-centos6-hp-025 doing B2G builds. There might be some inheritance from config.py that is broken, or masters in strange states.

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

OK, let's get those cleaned up and I'll add some 301's so that users end up at secure.pub.b.m.o.

Comment 13

[Dustin J. Mitchell [:dustin] (he/him)]

There are still a bunch of these going on.  I don't think it's from one specific master - the following masters are all implicated today:

[[e-mail]3 httpd]# grep clobberer build.mozilla.org/access_2012-07-30-* | sed 's/.*buildbot-master//' | sed 's/\..*//' | sort -u
12
13
14
25
30
31
32
33
34

The URLs they're using differ, too -- some have index.php, some do not.

build.mozilla.org/access_2012-07-30-22:10.250.49.187 - - [30/Jul/2012:15:01:59 -0700] "GET /clobberer?master=http%3A%2F%2Fbuildbot-master32.srv.releng.scl3.mozilla.com%3A8001%2F&slave=mw32-ix-slave14&builddir=m-in-w32-spidermonkey-warnaserr&branch=mozilla-inbound&buildername=mozilla-inbound_win32_spidermonkey-warnaserr HTTP/1.1" 301 478 "-" "Python-urllib/2.5"
build.mozilla.org/access_2012-07-30-22:10.250.49.187 - - [30/Jul/2012:15:01:59 -0700] "GET /clobberer/?master=http%3A%2F%2Fbuildbot-master32.srv.releng.scl3.mozilla.com%3A8001%2F&slave=mw32-ix-slave14&builddir=m-in-w32-spidermonkey-warnaserr&branch=mozilla-inbound&buildername=mozilla-inbound_win32_spidermonkey-warnaserr HTTP/1.1" 200 1434 "-" "Python-urllib/2.5"
build.mozilla.org/access_2012-07-30-22:10.12.40.98 - - [30/Jul/2012:15:02:24 -0700] "GET /clobberer/index.php?master=http%3A%2F%2Fbuildbot-master13.build.scl1.mozilla.com%3A8001%2F&slave=w64-ix-slave68&builddir=tb-c-cen-w64&branch=comm-central&buildername=TB+WINNT+6.1+x86-64+comm-central+build HTTP/1.1" 200 1208 "-" "Python-urllib/2.6"
build.mozilla.org/access_2012-07-30-22:10.12.40.39 - - [30/Jul/2012:15:03:48 -0700] "GET /clobberer/index.php?master=http%3A%2F%2Fbuildbot-master13.build.scl1.mozilla.com%3A8001%2F&slave=w64-ix-slave19&builddir=tb-c-cen-w32&branch=comm-central&buildername=TB+WINNT+5.2+comm-central+build HTTP/1.1" 200 1278 "-" "Python-urllib/2.6"
build.mozilla.org/access_2012-07-30-22:10.12.55.150 - - [30/Jul/2012:15:10:52 -0700] "GET /clobberer/index.php?master=http%3A%2F%2Fbuildbot-master25.build.scl1.mozilla.com%3A8001%2F&slave=bld-centos6-hp-023&builddir=m-in-gb-armv7a-gecko-dbg&branch=mozilla-inbound&buildername=B2G+gb_armv7a_gecko-debug+mozilla-inbound+build HTTP/1.1" 200 529 "-" "Python-urllib/2.6"
build.mozilla.org/access_2012-07-30-22:10.12.52.31 - - [30/Jul/2012:15:10:53 -0700] "GET /clobberer/index.php?master=http%3A%2F%2Fbuildbot-master25.build.scl1.mozilla.com%3A8001%2F&slave=bld-centos6-hp-014&builddir=m-in-gb-armv7a-gecko&branch=mozilla-inbound&buildername=B2G+gb_armv7a_gecko+mozilla-inbound+build HTTP/1.1" 200 430 "-" "Python-urllib/2.6"
build.mozilla.org/access_2012-07-30-22:10.12.52.10 - - [30/Jul/2012:15:13:46 -0700] "GET /clobberer/index.php?master=http%3A%2F%2Fbuildbot-master14.build.scl1.mozilla.com%3A8101%2F&slave=bld-centos6-hp-035&builddir=try-gb-armv7a-gecko-dbg&branch=try&buildername=B2G+gb_armv7a_gecko-debug+try+build HTTP/1.1" 200 - "-" "Python-urllib/2.6"
build.mozilla.org/access_2012-07-30-22:10.12.52.18 - - [30/Jul/2012:15:13:47 -0700] "GET /clobberer/index.php?master=http%3A%2F%2Fbuildbot-master14.build.scl1.mozilla.com%3A8101%2F&slave=bld-centos6-hp-027&builddir=try-gb-armv7a-gecko&branch=try&buildername=B2G+gb_armv7a_gecko+try+build HTTP/1.1" 200 - "-" "Python-urllib/2.6"
build.mozilla.org/access_2012-07-30-22:10.12.52.29 - - [30/Jul/2012:15:14:04 -0700] "GET /clobberer/index.php?master=http%3A%2F%2Fbuildbot-master13.build.scl1.mozilla.com%3A8001%2F&slave=bld-centos6-hp-016&builddir=srv-cen-gb-armv7a-gecko-dbg&branch=services-central&buildername=B2G+gb_armv7a_gecko-debug+services-central+build HTTP/1.1" 200 477 "-" "Python-urllib/2.6"
build.mozilla.org/access_2012-07-30-22:10.12.55.151 - - [30/Jul/2012:15:14:05 -0700] "GET /clobberer/index.php?master=http%3A%2F%2Fbuildbot-master13.build.scl1.mozilla.com%3A8001%2F&slave=bld-centos6-hp-020&builddir=srv-cen-gb-armv7a-gecko&branch=services-central&buildername=B2G+gb_armv7a_gecko+services-central+build HTTP/1.1" 200 598 "-" "Python-urllib/2.6"

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Update scripts

Here's a sed-style hatchet job on these scripts.

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 648874 [details] [diff] [review]
Update scripts

http://hg.mozilla.org/build/tools/rev/6f6d46fd3d41 (will get picked up as soon as jobs start)

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

I dumped out the builders for a clean build master and can't find any occurrences of build.mozilla.org/clobberer. Meanwhile on buildbot-master12 and 13 the b2g builds are still using the old domain. As bm12 has been up since May 20 we're just going to do a graceful shutdown and restart it to resolve this. I'll take a look at the Apache logs next week and take care of that.

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

bm12 has been restarted.

Comment 18

[Dustin J. Mitchell [:dustin] (he/him)]

Still seeing problems with:

[[e-mail]3 build.mozilla.org]# grep "/clobberer"  access_2012-08-06-* | sed 's/.*master=\([^&]*\)&.*/\1/' | sort -u
http%3A%2F%2Fbuildbot-master13.build.scl1.mozilla.com%3A8001%2F
http%3A%2F%2Fbuildbot-master14.build.scl1.mozilla.com%3A8101%2F
http%3A%2F%2Fbuildbot-master25.build.scl1.mozilla.com%3A8001%2F
http%3A%2F%2Fbuildbot-master32.srv.releng.scl3.mozilla.com%3A8001%2F
http%3A%2F%2Fbuildbot-master34.srv.releng.scl3.mozilla.com%3A8001%2F
http%3A%2F%2Fbuildbot-master35.srv.releng.scl3.mozilla.com%3A8101%2F
http%3A%2F%2Fdev-master01.build.scl1.mozilla.com%3A8049%2F

Thanks for working on this!

Comment 19

[Nick Thomas [:nthomas] (UTC+12)]

We should be in much better shape now that bug 780806 is deployed.

Comment 20

[Dustin J. Mitchell [:dustin] (he/him)]

Indeed, this was the last one:

access_2012-08-08-20:10.26.74.22 - - [08/Aug/2012:13:14:14 -0700] "GET /clobberer/index.php?branch=release-comm-beta HTTP/1.1" 200 15790 "http://build.mozilla.org/clobberer/index.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.7; rv:14.0) Gecko/20100101 Firefox/14.0.1"

Yay!