914965 - clobberer fails in 17.0.9 ESR jobs which still use Python 2.5

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[Nick Thomas [:nthomas] (UTC+12)]

Calling clobberer: /tools/python/bin/python /builds/slave/rel-m-esr17-firefox_tag/scripts/scripts/release/../..//clobberer/clobberer.py -s scripts -s buildprops.json http://clobberer.pvt.build.mozilla.org/always_clobber.php release-mozilla-esr17 release-mozilla-esr17-firefox_tag rel-m-esr17-firefox_tag linux-ix-slave02 http://buildbot-master57.srv.releng.use1.mozilla.com:8001/
Checking clobber URL: http://clobberer.pvt.build.mozilla.org/always_clobber.php?master=http%3A%2F%2Fbuildbot-master57.srv.releng.use1.mozilla.com%3A8001%2F&slave=linux-ix-slave02&builddir=rel-m-esr17-firefox_tag&branch=release-mozilla-esr17&buildername=release-mozilla-esr17-firefox_tag
Error contacting server
program finished with exit code 1
elapsedTime=0.358182

Going to redo the sendchange to restart, which should avoid any issues with schedulers and rebuilding the tag builder.

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

This was from linux-ix-slave02.build.scl1.mozilla.com. When I redid the sendchange, it got linux-ix-slave06 and that also failed. So this looks like clobberer.pvt.build.mozilla.org doesn't have a complete set of netblocks to whitelist. 

The builder only linux-ix-slave01/02/06 configured, plus half a dozen bld-centos5-32-vmw-001-006.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

This is kinda weird. If I make a request from my machine I get a 403 response; since I'm not a build slave this is expected. But the response is cached, so when a build machine makes a request:

cltbld@linux-ix-slave02 ~]$ curl -si "http://clobberer.pvt.build.mozilla.org/always_clobber.php?master=http%3A%2F%2Fbuildbot-master58.srv.releng.usw2.mozilla.com%3A8001%2F&slave=linux-ix-slave06&builddir=rel-m-esr17-firefox_tag&branch=release-mozilla-esr17&buildername=release-mozilla-esr17-firefox_tag"
HTTP/1.1 403 Forbidden
Server: Apache
X-Backend-Server: web2.releng.webapp.scl3.mozilla.com
Content-Type: text/html; charset=iso-8859-1
Date: Wed, 11 Sep 2013 08:59:49 GMT
Content-Length: 220
Connection: Keep-Alive
X-Cache-Info: cached

Once the cache expires:
HTTP/1.1 200 OK
Server: Apache
X-Backend-Server: web2.releng.webapp.scl3.mozilla.com
Cache-Control: max-age=0
Content-Type: text/html; charset=UTF-8
Date: Wed, 11 Sep 2013 09:05:14 GMT
Expires: Wed, 11 Sep 2013 09:05:14 GMT
X-Powered-By: PHP/5.3.3
X-Cache-Info: not cacheable; response specified max-age <= 0
Content-Length: 50

Note this isn't cached. Any thoughts here Dustin ? Can we make Zeus never cache, or fix the Apache config for LDAP auth so prevents caching ? 

I wouldn't have caused the original 403, so someone maybe being malicious (or just curious) or there's some set of slaves not in a whitelist. My IP begins 121.73.

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

Scratch that theory. Here's the verbose output:

Checking clobber URL: http://clobberer.pvt.build.mozilla.org/always_clobber.php?mast...<snip>
Traceback (most recent call last):
  File "/builds/slave/rel-m-esr17-firefox_tag/scripts/scripts/release/../..//clobberer/clobberer.py", line 191, in <module>
    clobberURL, branch, builder, my_builddir, slave, master)
  File "/builds/slave/rel-m-esr17-firefox_tag/scripts/scripts/release/../..//clobberer/clobberer.py", line 146, in getClobberDates
    data = urllib2.urlopen(url, timeout=30).read().strip()
TypeError: urlopen() got an unexpected keyword argument 'timeout'
Error contacting server

This is finding Python 2.5.1 earliest in the path, and urllib2.open() doesn't take a timeout arg until 2.6.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [tools] Test for python version

This should probably land in mozharness too, to keep it in sync.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

http://hg.mozilla.org/build/tools/rev/b070617b29bd

Worked, then we hit bug 915075.

Comment 6

[Ed Morley [:emorley]]

Sigh, can't believe we're still using Python 2.5 :-(
Least we'll be rid of ESR17 soon..

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

The caching question is one for webops.  Beyond my Zeus/Apache skills.

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

(In reply to Nick Thomas [:nthomas] from comment #4)
> This should probably land in mozharness too, to keep it in sync.

https://hg.mozilla.org/build/mozharness/rev/fc9e20f29fb9 on default