clobberer fails in 17.0.9 ESR jobs which still use Python 2.5

RESOLVED FIXED

Status

Release Engineering
General
P2
normal
RESOLVED FIXED
5 years ago
3 months ago

People

(Reporter: nthomas, Assigned: nthomas)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Assignee)

Description

5 years ago
Calling clobberer: /tools/python/bin/python /builds/slave/rel-m-esr17-firefox_tag/scripts/scripts/release/../..//clobberer/clobberer.py -s scripts -s buildprops.json http://clobberer.pvt.build.mozilla.org/always_clobber.php release-mozilla-esr17 release-mozilla-esr17-firefox_tag rel-m-esr17-firefox_tag linux-ix-slave02 http://buildbot-master57.srv.releng.use1.mozilla.com:8001/
Checking clobber URL: http://clobberer.pvt.build.mozilla.org/always_clobber.php?master=http%3A%2F%2Fbuildbot-master57.srv.releng.use1.mozilla.com%3A8001%2F&slave=linux-ix-slave02&builddir=rel-m-esr17-firefox_tag&branch=release-mozilla-esr17&buildername=release-mozilla-esr17-firefox_tag
Error contacting server
program finished with exit code 1
elapsedTime=0.358182

Going to redo the sendchange to restart, which should avoid any issues with schedulers and rebuilding the tag builder.
(Assignee)

Comment 1

5 years ago
This was from linux-ix-slave02.build.scl1.mozilla.com. When I redid the sendchange, it got linux-ix-slave06 and that also failed. So this looks like clobberer.pvt.build.mozilla.org doesn't have a complete set of netblocks to whitelist. 

The builder only linux-ix-slave01/02/06 configured, plus half a dozen bld-centos5-32-vmw-001-006.
(Assignee)

Comment 2

5 years ago
This is kinda weird. If I make a request from my machine I get a 403 response; since I'm not a build slave this is expected. But the response is cached, so when a build machine makes a request:

cltbld@linux-ix-slave02 ~]$ curl -si "http://clobberer.pvt.build.mozilla.org/always_clobber.php?master=http%3A%2F%2Fbuildbot-master58.srv.releng.usw2.mozilla.com%3A8001%2F&slave=linux-ix-slave06&builddir=rel-m-esr17-firefox_tag&branch=release-mozilla-esr17&buildername=release-mozilla-esr17-firefox_tag"
HTTP/1.1 403 Forbidden
Server: Apache
X-Backend-Server: web2.releng.webapp.scl3.mozilla.com
Content-Type: text/html; charset=iso-8859-1
Date: Wed, 11 Sep 2013 08:59:49 GMT
Content-Length: 220
Connection: Keep-Alive
X-Cache-Info: cached

Once the cache expires:
HTTP/1.1 200 OK
Server: Apache
X-Backend-Server: web2.releng.webapp.scl3.mozilla.com
Cache-Control: max-age=0
Content-Type: text/html; charset=UTF-8
Date: Wed, 11 Sep 2013 09:05:14 GMT
Expires: Wed, 11 Sep 2013 09:05:14 GMT
X-Powered-By: PHP/5.3.3
X-Cache-Info: not cacheable; response specified max-age <= 0
Content-Length: 50

Note this isn't cached. Any thoughts here Dustin ? Can we make Zeus never cache, or fix the Apache config for LDAP auth so prevents caching ? 

I wouldn't have caused the original 403, so someone maybe being malicious (or just curious) or there's some set of slaves not in a whitelist. My IP begins 121.73.
Flags: needinfo?(dustin)
(Assignee)

Comment 3

5 years ago
Scratch that theory. Here's the verbose output:

Checking clobber URL: http://clobberer.pvt.build.mozilla.org/always_clobber.php?mast...<snip>
Traceback (most recent call last):
  File "/builds/slave/rel-m-esr17-firefox_tag/scripts/scripts/release/../..//clobberer/clobberer.py", line 191, in <module>
    clobberURL, branch, builder, my_builddir, slave, master)
  File "/builds/slave/rel-m-esr17-firefox_tag/scripts/scripts/release/../..//clobberer/clobberer.py", line 146, in getClobberDates
    data = urllib2.urlopen(url, timeout=30).read().strip()
TypeError: urlopen() got an unexpected keyword argument 'timeout'
Error contacting server

This is finding Python 2.5.1 earliest in the path, and urllib2.open() doesn't take a timeout arg until 2.6.
Flags: needinfo?(dustin)
(Assignee)

Updated

5 years ago
Summary: Failure to reach clobberer in tagging for Firefox 17.0.9 ESR → clobberer fails in some 17.0.9 ESR jobs which use old python
(Assignee)

Comment 4

5 years ago
Created attachment 802904 [details] [diff] [review]
[tools] Test for python version

This should probably land in mozharness too, to keep it in sync.
Attachment #802904 - Flags: review?(catlee)

Updated

5 years ago
Attachment #802904 - Flags: review?(catlee) → review+
(Assignee)

Comment 5

5 years ago
http://hg.mozilla.org/build/tools/rev/b070617b29bd

Worked, then we hit bug 915075.
Blocks: 911194
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Component: Releases → General Automation
QA Contact: bhearsum → catlee
Resolution: --- → FIXED
Summary: clobberer fails in some 17.0.9 ESR jobs which use old python → clobberer fails in 17.0.9 ESR jobs which still use Python 2.5
Sigh, can't believe we're still using Python 2.5 :-(
Least we'll be rid of ESR17 soon..
The caching question is one for webops.  Beyond my Zeus/Apache skills.
(Assignee)

Updated

5 years ago
Blocks: 915356
(Assignee)

Comment 8

5 years ago
(In reply to Nick Thomas [:nthomas] from comment #4)
> This should probably land in mozharness too, to keep it in sync.

https://hg.mozilla.org/build/mozharness/rev/fc9e20f29fb9 on default
Component: General Automation → General
Product: Release Engineering → Release Engineering
You need to log in before you can comment on or make changes to this bug.