Closed Bug 1164878 Opened 9 years ago Closed 9 years ago

cancel builds with self serve is broken

Categories

(Release Engineering :: General, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: bhearsum, Assigned: mrrrgn)

References

Details

Attachments

(3 files)

I discovered this while testing buildbot bridge support for canceling jobs through Taskcluster, which depends on self-serve cancellation. It looks like something related to clobbering is broken, which causes the agent to bail:
198168 2015-05-14 08:27:58,602 Received {u'action': u'cancel_build', u'body': {u'bid': 70961237, u'when': 1431617278, u'request_id': 1313128}, u'who': u'buildbot-bridge'}
198169 2015-05-14 08:27:58,603 Loading masters from https://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/production-masters.json
198170 2015-05-14 08:27:58,839 Loading branches from https://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/production-branches.json
198171 2015-05-14 08:27:58,992 cancelling build by buildbot-bridge of 70961237
198172 2015-05-14 08:27:59,025 build is running, going to cancel it!
198173 2015-05-14 08:27:59,025 getting slavename from  http://buildbot-master82.bb.releng.scl3.mozilla.com:8001/builders/OS%20X%2010.7%20alder%20build/builds/3
198174 2015-05-14 08:27:59,034 Clobbering bld-lion-r5-090 OS X 10.7 alder build at https://api.pub.build.mozilla.org/clobberer/clobber/by-builder [{'slave': 'bld-lion-r5-090', 'buildername': 'OS X 10.7 alder build'}]
198175 2015-05-14 08:27:59,054 Couldn't cancel build
198176 Traceback (most recent call last):
198177   File "/builds/selfserve-agent/lib/python2.7/site-packages/buildapi/scripts/selfserve_agent.py", line 445, in do_cancel_build
198178     msg = self._cancel_build(build.claimed_by_name, build.buildername, build.number, who, "Cancelled via self-serve")
198179   File "/builds/selfserve-agent/lib/python2.7/site-packages/buildapi/scripts/selfserve_agent.py", line 173, in _cancel_build
198180     self._clobber_slave(slavename, builder_name)
198181   File "/builds/selfserve-agent/lib/python2.7/site-packages/buildapi/scripts/selfserve_agent.py", line 138, in _clobber_slave
198182     urllib2.urlopen(req, json.dumps(data))
198183   File "/tools/python27/lib/python2.7/urllib2.py", line 126, in urlopen
198184     return _opener.open(url, data, timeout)
198185   File "/tools/python27/lib/python2.7/urllib2.py", line 406, in open
198186     response = meth(req, response)
198187   File "/tools/python27/lib/python2.7/urllib2.py", line 519, in http_response
198188     'http', request, response, code, msg, hdrs)
198189   File "/tools/python27/lib/python2.7/urllib2.py", line 444, in error
198190     return self._call_chain(*args)
198191   File "/tools/python27/lib/python2.7/urllib2.py", line 378, in _call_chain
198192     result = func(*args)
198193   File "/tools/python27/lib/python2.7/urllib2.py", line 527, in http_error_default
198194     raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
198195 HTTPError: HTTP Error 403: FORBIDDEN


Morgan, I'm not sure if this is your fault or not, but I think you're the last one that touched clobberer!
Flags: needinfo?(winter2718)
Blocks: 1156301
Doh, it's because you need to get use a token to clobber. Generate one here: https://api.pub.build.mozilla.org/tokenauth/ with a permission of ' clobberer.post.clobber ' and add that to the request. It'll require modifying the self-serve code unfortunately.
Flags: needinfo?(winter2718)
(In reply to Ben Hearsum [:bhearsum] from comment #0)
> I discovered this while testing buildbot bridge support for canceling jobs
> through Taskcluster, which depends on self-serve cancellation. It looks like
> something related to clobbering is broken, which causes the agent to bail:
> 198168 2015-05-14 08:27:58,602 Received {u'action': u'cancel_build',
> u'body': {u'bid': 70961237, u'when': 1431617278, u'request_id': 1313128},
> u'who': u'buildbot-bridge'}
> 198169 2015-05-14 08:27:58,603 Loading masters from
> https://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/
> production-masters.json
> 198170 2015-05-14 08:27:58,839 Loading branches from
> https://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/
> production-branches.json
> 198171 2015-05-14 08:27:58,992 cancelling build by buildbot-bridge of
> 70961237
> 198172 2015-05-14 08:27:59,025 build is running, going to cancel it!
> 198173 2015-05-14 08:27:59,025 getting slavename from 
> http://buildbot-master82.bb.releng.scl3.mozilla.com:8001/builders/
> OS%20X%2010.7%20alder%20build/builds/3
> 198174 2015-05-14 08:27:59,034 Clobbering bld-lion-r5-090 OS X 10.7 alder
> build at https://api.pub.build.mozilla.org/clobberer/clobber/by-builder
> [{'slave': 'bld-lion-r5-090', 'buildername': 'OS X 10.7 alder build'}]
> 198175 2015-05-14 08:27:59,054 Couldn't cancel build
> 198176 Traceback (most recent call last):
> 198177   File
> "/builds/selfserve-agent/lib/python2.7/site-packages/buildapi/scripts/
> selfserve_agent.py", line 445, in do_cancel_build
> 198178     msg = self._cancel_build(build.claimed_by_name,
> build.buildername, build.number, who, "Cancelled via self-serve")
> 198179   File
> "/builds/selfserve-agent/lib/python2.7/site-packages/buildapi/scripts/
> selfserve_agent.py", line 173, in _cancel_build
> 198180     self._clobber_slave(slavename, builder_name)
> 198181   File
> "/builds/selfserve-agent/lib/python2.7/site-packages/buildapi/scripts/
> selfserve_agent.py", line 138, in _clobber_slave
> 198182     urllib2.urlopen(req, json.dumps(data))
> 198183   File "/tools/python27/lib/python2.7/urllib2.py", line 126, in
> urlopen
> 198184     return _opener.open(url, data, timeout)
> 198185   File "/tools/python27/lib/python2.7/urllib2.py", line 406, in open
> 198186     response = meth(req, response)
> 198187   File "/tools/python27/lib/python2.7/urllib2.py", line 519, in
> http_response
> 198188     'http', request, response, code, msg, hdrs)
> 198189   File "/tools/python27/lib/python2.7/urllib2.py", line 444, in error
> 198190     return self._call_chain(*args)
> 198191   File "/tools/python27/lib/python2.7/urllib2.py", line 378, in
> _call_chain
> 198192     result = func(*args)
> 198193   File "/tools/python27/lib/python2.7/urllib2.py", line 527, in
> http_error_default
> 198194     raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
> 198195 HTTPError: HTTP Error 403: FORBIDDEN
> 
> 
> Morgan, I'm not sure if this is your fault or not, but I think you're the
> last one that touched clobberer!

I should also make it clear, this is my fault. I added the token auth and didn't think about this client. Sorry!
Assignee: nobody → winter2718
Attached patch clobberauth.diffSplinter Review
Verified to work with a good token. I'll need to add the token via puppet in a subsequent patch.
Attachment #8606994 - Flags: review?(bhearsum)
Attachment #8607079 - Flags: review?(bhearsum)
Attachment #8607079 - Flags: review?(bhearsum) → review+
Comment on attachment 8606994 [details] [diff] [review]
clobberauth.diff

Review of attachment 8606994 [details] [diff] [review]:
-----------------------------------------------------------------

Thanks for fixing this up!
Attachment #8606994 - Flags: review?(bhearsum) → review+
Attachment #8606994 - Flags: checked-in+
Attachment #8607079 - Flags: checked-in+
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Blocks: 1166307
Just tested this in production - it works!
Status: RESOLVED → VERIFIED
Morgan, it looks like you issued a user token here -- that's tied to your user account, and would be automatically disabled if you left or lost one of the required permissions.  That wouldn't be good!

Can you issue a new permanent token for this purpose, and switch it out in hiera?
Status: VERIFIED → REOPENED
Resolution: FIXED → ---
Using a permanent token fails with this. The reason being, clobberer tries to attach an e-mail to each clobber request (for the "who" field). I can create a default in the clobberer app itself, or we can add an e-mail field to the TokenUser.

Traceback (most recent call last):
File "/data/www/relengapi/virtualenv/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request rv = self.dispatch_request()
File "/data/www/relengapi/virtualenv/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request return self.view_functions[rule.endpoint](**req.view_args)
File "/data/www/relengapi/virtualenv/lib/python2.7/site-packages/newrelic-2.46.0.37/newrelic/hooks/framework_flask.py", line 40, in _nr_wrapper_handler_ return wrapped(*args, **kwargs)
File "/data/www/relengapi/virtualenv/lib/python2.7/site-packages/relengapi/lib/api.py", line 103, in replacement result = wrapped(*args, **kwargs)
File "/data/www/relengapi/virtualenv/lib/python2.7/site-packages/relengapi/lib/permissions.py", line 88, in req return wrapped(*args, **kwargs)
File "/data/www/relengapi/virtualenv/lib/python2.7/site-packages/relengapi/blueprints/clobberer/__init__.py", line 126, in clobber_by_builder slave=clobber.slave
File "/data/www/relengapi/virtualenv/lib/python2.7/site-packages/relengapi/blueprints/clobberer/__init__.py", line 73, in _add_clobber who = current_user.authenticated_email
File "/data/www/relengapi/virtualenv/lib/python2.7/site-packages/werkzeug/local.py", line 338, in __getattr__ return getattr(self._get_current_object(), name)
AttributeError: 'TokenUser' object has no attribute 'authenticated_email'
Oh, well, it's not doing clobberer any good to record you as the person doing every clobber!

Ideally, we'd find a way to log the person who clicked the button in tbpl, but my understanding is that the chain of causation goes treeherder -> buildapi -> selfserve agent -> relengapi, and that's a loooong way to thread an email address.  So maybe just change `if current_user.anonymous is False` to `if hasattr(current_user, 'authenticated_email')`?
Attachment #8612367 - Flags: review?(dustin) → review+
I replaced the clobberer token with a permanent one. If we've not updated the relengapi deployment lately we'll need to do that asap. Otherwise, we're good to go and can close the bug.(In reply to Dustin J. Mitchell [:dustin] from comment #10)
> Oh, well, it's not doing clobberer any good to record you as the person
> doing every clobber!
> 
> Ideally, we'd find a way to log the person who clicked the button in tbpl,
> but my understanding is that the chain of causation goes treeherder ->
> buildapi -> selfserve agent -> relengapi, and that's a loooong way to thread
> an email address.  So maybe just change `if current_user.anonymous is False`
> to `if hasattr(current_user, 'authenticated_email')`?

I replaced the clobberer token with a permanent one. If we've not updated the relengapi deployment lately we'll need to do that asap. Otherwise, we're good to go and can close the bug.
Verified working in production.
Status: REOPENED → RESOLVED
Closed: 9 years ago9 years ago
Resolution: --- → FIXED
(In reply to Dustin J. Mitchell [:dustin] from comment #10)
> Ideally, we'd find a way to log the person who clicked the button in tbpl,
> but my understanding is that the chain of causation goes treeherder ->
> buildapi -> selfserve agent -> relengapi

There's no treeherder integration with the clobberer (unlike retriggers/cancels for buildapi) - sheriffs use the clobberer links for each repo to take them directly to eg https://api.pub.build.mozilla.org/clobberer/?branch=mozilla-inbound
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.