Closed
Bug 1164878
Opened 9 years ago
Closed 9 years ago
cancel builds with self serve is broken
Categories
(Release Engineering :: General, defect)
Release Engineering
General
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: bhearsum, Assigned: mrrrgn)
References
Details
Attachments
(3 files)
3.45 KB,
patch
|
bhearsum
:
review+
mrrrgn
:
checked-in+
|
Details | Diff | Splinter Review |
1019 bytes,
patch
|
bhearsum
:
review+
mrrrgn
:
checked-in+
|
Details | Diff | Splinter Review |
60 bytes,
text/x-github-pull-request
|
dustin
:
review+
|
Details | Review |
I discovered this while testing buildbot bridge support for canceling jobs through Taskcluster, which depends on self-serve cancellation. It looks like something related to clobbering is broken, which causes the agent to bail: 198168 2015-05-14 08:27:58,602 Received {u'action': u'cancel_build', u'body': {u'bid': 70961237, u'when': 1431617278, u'request_id': 1313128}, u'who': u'buildbot-bridge'} 198169 2015-05-14 08:27:58,603 Loading masters from https://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/production-masters.json 198170 2015-05-14 08:27:58,839 Loading branches from https://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/production-branches.json 198171 2015-05-14 08:27:58,992 cancelling build by buildbot-bridge of 70961237 198172 2015-05-14 08:27:59,025 build is running, going to cancel it! 198173 2015-05-14 08:27:59,025 getting slavename from http://buildbot-master82.bb.releng.scl3.mozilla.com:8001/builders/OS%20X%2010.7%20alder%20build/builds/3 198174 2015-05-14 08:27:59,034 Clobbering bld-lion-r5-090 OS X 10.7 alder build at https://api.pub.build.mozilla.org/clobberer/clobber/by-builder [{'slave': 'bld-lion-r5-090', 'buildername': 'OS X 10.7 alder build'}] 198175 2015-05-14 08:27:59,054 Couldn't cancel build 198176 Traceback (most recent call last): 198177 File "/builds/selfserve-agent/lib/python2.7/site-packages/buildapi/scripts/selfserve_agent.py", line 445, in do_cancel_build 198178 msg = self._cancel_build(build.claimed_by_name, build.buildername, build.number, who, "Cancelled via self-serve") 198179 File "/builds/selfserve-agent/lib/python2.7/site-packages/buildapi/scripts/selfserve_agent.py", line 173, in _cancel_build 198180 self._clobber_slave(slavename, builder_name) 198181 File "/builds/selfserve-agent/lib/python2.7/site-packages/buildapi/scripts/selfserve_agent.py", line 138, in _clobber_slave 198182 urllib2.urlopen(req, json.dumps(data)) 198183 File "/tools/python27/lib/python2.7/urllib2.py", line 126, in urlopen 198184 return _opener.open(url, data, timeout) 198185 File "/tools/python27/lib/python2.7/urllib2.py", line 406, in open 198186 response = meth(req, response) 198187 File "/tools/python27/lib/python2.7/urllib2.py", line 519, in http_response 198188 'http', request, response, code, msg, hdrs) 198189 File "/tools/python27/lib/python2.7/urllib2.py", line 444, in error 198190 return self._call_chain(*args) 198191 File "/tools/python27/lib/python2.7/urllib2.py", line 378, in _call_chain 198192 result = func(*args) 198193 File "/tools/python27/lib/python2.7/urllib2.py", line 527, in http_error_default 198194 raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) 198195 HTTPError: HTTP Error 403: FORBIDDEN Morgan, I'm not sure if this is your fault or not, but I think you're the last one that touched clobberer!
Flags: needinfo?(winter2718)
Assignee | ||
Comment 1•9 years ago
|
||
Doh, it's because you need to get use a token to clobber. Generate one here: https://api.pub.build.mozilla.org/tokenauth/ with a permission of ' clobberer.post.clobber ' and add that to the request. It'll require modifying the self-serve code unfortunately.
Flags: needinfo?(winter2718)
Assignee | ||
Comment 2•9 years ago
|
||
(In reply to Ben Hearsum [:bhearsum] from comment #0) > I discovered this while testing buildbot bridge support for canceling jobs > through Taskcluster, which depends on self-serve cancellation. It looks like > something related to clobbering is broken, which causes the agent to bail: > 198168 2015-05-14 08:27:58,602 Received {u'action': u'cancel_build', > u'body': {u'bid': 70961237, u'when': 1431617278, u'request_id': 1313128}, > u'who': u'buildbot-bridge'} > 198169 2015-05-14 08:27:58,603 Loading masters from > https://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/ > production-masters.json > 198170 2015-05-14 08:27:58,839 Loading branches from > https://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/ > production-branches.json > 198171 2015-05-14 08:27:58,992 cancelling build by buildbot-bridge of > 70961237 > 198172 2015-05-14 08:27:59,025 build is running, going to cancel it! > 198173 2015-05-14 08:27:59,025 getting slavename from > http://buildbot-master82.bb.releng.scl3.mozilla.com:8001/builders/ > OS%20X%2010.7%20alder%20build/builds/3 > 198174 2015-05-14 08:27:59,034 Clobbering bld-lion-r5-090 OS X 10.7 alder > build at https://api.pub.build.mozilla.org/clobberer/clobber/by-builder > [{'slave': 'bld-lion-r5-090', 'buildername': 'OS X 10.7 alder build'}] > 198175 2015-05-14 08:27:59,054 Couldn't cancel build > 198176 Traceback (most recent call last): > 198177 File > "/builds/selfserve-agent/lib/python2.7/site-packages/buildapi/scripts/ > selfserve_agent.py", line 445, in do_cancel_build > 198178 msg = self._cancel_build(build.claimed_by_name, > build.buildername, build.number, who, "Cancelled via self-serve") > 198179 File > "/builds/selfserve-agent/lib/python2.7/site-packages/buildapi/scripts/ > selfserve_agent.py", line 173, in _cancel_build > 198180 self._clobber_slave(slavename, builder_name) > 198181 File > "/builds/selfserve-agent/lib/python2.7/site-packages/buildapi/scripts/ > selfserve_agent.py", line 138, in _clobber_slave > 198182 urllib2.urlopen(req, json.dumps(data)) > 198183 File "/tools/python27/lib/python2.7/urllib2.py", line 126, in > urlopen > 198184 return _opener.open(url, data, timeout) > 198185 File "/tools/python27/lib/python2.7/urllib2.py", line 406, in open > 198186 response = meth(req, response) > 198187 File "/tools/python27/lib/python2.7/urllib2.py", line 519, in > http_response > 198188 'http', request, response, code, msg, hdrs) > 198189 File "/tools/python27/lib/python2.7/urllib2.py", line 444, in error > 198190 return self._call_chain(*args) > 198191 File "/tools/python27/lib/python2.7/urllib2.py", line 378, in > _call_chain > 198192 result = func(*args) > 198193 File "/tools/python27/lib/python2.7/urllib2.py", line 527, in > http_error_default > 198194 raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) > 198195 HTTPError: HTTP Error 403: FORBIDDEN > > > Morgan, I'm not sure if this is your fault or not, but I think you're the > last one that touched clobberer! I should also make it clear, this is my fault. I added the token auth and didn't think about this client. Sorry!
Assignee | ||
Updated•9 years ago
|
Assignee: nobody → winter2718
Assignee | ||
Comment 4•9 years ago
|
||
Verified to work with a good token. I'll need to add the token via puppet in a subsequent patch.
Attachment #8606994 -
Flags: review?(bhearsum)
Assignee | ||
Comment 5•9 years ago
|
||
Attachment #8607079 -
Flags: review?(bhearsum)
Reporter | ||
Updated•9 years ago
|
Attachment #8607079 -
Flags: review?(bhearsum) → review+
Reporter | ||
Comment 6•9 years ago
|
||
Comment on attachment 8606994 [details] [diff] [review] clobberauth.diff Review of attachment 8606994 [details] [diff] [review]: ----------------------------------------------------------------- Thanks for fixing this up!
Attachment #8606994 -
Flags: review?(bhearsum) → review+
Assignee | ||
Updated•9 years ago
|
Attachment #8606994 -
Flags: checked-in+
Assignee | ||
Updated•9 years ago
|
Attachment #8607079 -
Flags: checked-in+
Assignee | ||
Updated•9 years ago
|
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Reporter | ||
Comment 7•9 years ago
|
||
Just tested this in production - it works!
Status: RESOLVED → VERIFIED
Comment 8•9 years ago
|
||
Morgan, it looks like you issued a user token here -- that's tied to your user account, and would be automatically disabled if you left or lost one of the required permissions. That wouldn't be good! Can you issue a new permanent token for this purpose, and switch it out in hiera?
Status: VERIFIED → REOPENED
Resolution: FIXED → ---
Assignee | ||
Comment 9•9 years ago
|
||
Using a permanent token fails with this. The reason being, clobberer tries to attach an e-mail to each clobber request (for the "who" field). I can create a default in the clobberer app itself, or we can add an e-mail field to the TokenUser. Traceback (most recent call last): File "/data/www/relengapi/virtualenv/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request rv = self.dispatch_request() File "/data/www/relengapi/virtualenv/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request return self.view_functions[rule.endpoint](**req.view_args) File "/data/www/relengapi/virtualenv/lib/python2.7/site-packages/newrelic-2.46.0.37/newrelic/hooks/framework_flask.py", line 40, in _nr_wrapper_handler_ return wrapped(*args, **kwargs) File "/data/www/relengapi/virtualenv/lib/python2.7/site-packages/relengapi/lib/api.py", line 103, in replacement result = wrapped(*args, **kwargs) File "/data/www/relengapi/virtualenv/lib/python2.7/site-packages/relengapi/lib/permissions.py", line 88, in req return wrapped(*args, **kwargs) File "/data/www/relengapi/virtualenv/lib/python2.7/site-packages/relengapi/blueprints/clobberer/__init__.py", line 126, in clobber_by_builder slave=clobber.slave File "/data/www/relengapi/virtualenv/lib/python2.7/site-packages/relengapi/blueprints/clobberer/__init__.py", line 73, in _add_clobber who = current_user.authenticated_email File "/data/www/relengapi/virtualenv/lib/python2.7/site-packages/werkzeug/local.py", line 338, in __getattr__ return getattr(self._get_current_object(), name) AttributeError: 'TokenUser' object has no attribute 'authenticated_email'
Comment 10•9 years ago
|
||
Oh, well, it's not doing clobberer any good to record you as the person doing every clobber! Ideally, we'd find a way to log the person who clicked the button in tbpl, but my understanding is that the chain of causation goes treeherder -> buildapi -> selfserve agent -> relengapi, and that's a loooong way to thread an email address. So maybe just change `if current_user.anonymous is False` to `if hasattr(current_user, 'authenticated_email')`?
Assignee | ||
Comment 11•9 years ago
|
||
Attachment #8612367 -
Flags: review?(dustin)
Updated•9 years ago
|
Attachment #8612367 -
Flags: review?(dustin) → review+
Assignee | ||
Comment 12•9 years ago
|
||
I replaced the clobberer token with a permanent one. If we've not updated the relengapi deployment lately we'll need to do that asap. Otherwise, we're good to go and can close the bug.(In reply to Dustin J. Mitchell [:dustin] from comment #10) > Oh, well, it's not doing clobberer any good to record you as the person > doing every clobber! > > Ideally, we'd find a way to log the person who clicked the button in tbpl, > but my understanding is that the chain of causation goes treeherder -> > buildapi -> selfserve agent -> relengapi, and that's a loooong way to thread > an email address. So maybe just change `if current_user.anonymous is False` > to `if hasattr(current_user, 'authenticated_email')`? I replaced the clobberer token with a permanent one. If we've not updated the relengapi deployment lately we'll need to do that asap. Otherwise, we're good to go and can close the bug.
Assignee | ||
Comment 13•9 years ago
|
||
Verified working in production.
Status: REOPENED → RESOLVED
Closed: 9 years ago → 9 years ago
Resolution: --- → FIXED
Comment 14•9 years ago
|
||
(In reply to Dustin J. Mitchell [:dustin] from comment #10) > Ideally, we'd find a way to log the person who clicked the button in tbpl, > but my understanding is that the chain of causation goes treeherder -> > buildapi -> selfserve agent -> relengapi There's no treeherder integration with the clobberer (unlike retriggers/cancels for buildapi) - sheriffs use the clobberer links for each repo to take them directly to eg https://api.pub.build.mozilla.org/clobberer/?branch=mozilla-inbound
Updated•6 years ago
|
Component: General Automation → General
You need to log in
before you can comment on or make changes to this bug.
Description
•