Closed Bug 1261075 Opened 8 years ago Closed 8 years ago

pushlog_client does not raise exceptions properly

Categories

(Developer Services :: Mercurial: Pushlog, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: armenzg, Assigned: armenzg)

Details

Attachments

(3 files)

During yesterday hg outages I got a bunch of alerts on pulse_actions which did not have much information for me to know what was the root issue.

I would like to log the exception in a better way.
When an exception is raised when querying the pushlog we should at least log it.

Review commit: https://reviewboard.mozilla.org/r/43535/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/43535/
Attachment #8736713 - Flags: review?(gps)
What exceptions were you getting?
Flags: needinfo?(armenzg)
Unfortunately the code muted the exception.

 Mar 31 10:47:07 pulse-actions app/worker2.1: pushlog_client	 DEBUG:	 About to fetch https://hg.mozilla.org/integration/mozilla-inbound/json-pushes?changeset=e8cd4e639e410abc015b6c4211059e700a39c9c3&tipsonly=1 
Mar 31 10:47:08 pulse-actions app/worker2.1: pushlog_client	 DEBUG:	 Push info: {u'74399': {u'changesets': [u'e8cd4e639e410abc015b6c4211059e700a39c9c3'], u'date': 1459321529, u'user': u'gachen@mozilla.com'}} 
Mar 31 10:47:08 pulse-actions app/worker2.1: Traceback (most recent call last): 
Mar 31 10:47:08 pulse-actions app/worker2.1:   File "pulse_actions/worker.py", line 79, in run_pulse 
Mar 31 10:47:08 pulse-actions app/worker2.1:     pulse.listen() 
Mar 31 10:47:08 pulse-actions app/worker2.1:   File "/app/.heroku/python/lib/python2.7/site-packages/mozillapulse/consumers.py", line 151, in listen 
Mar 31 10:47:08 pulse-actions app/worker2.1:     self._drain_events_loop() 
Mar 31 10:47:08 pulse-actions app/worker2.1:   File "/app/.heroku/python/lib/python2.7/site-packages/mozillapulse/consumers.py", line 198, in _drain_events_loop 
Mar 31 10:47:08 pulse-actions app/worker2.1:     self.connection.drain_events(timeout=self.timeout) 
Mar 31 10:47:08 pulse-actions app/worker2.1:   File "/app/.heroku/python/lib/python2.7/site-packages/kombu/connection.py", line 275, in drain_events 
Mar 31 10:47:08 pulse-actions app/worker2.1:     return self.transport.drain_events(self.connection, **kwargs) 
Mar 31 10:47:08 pulse-actions app/worker2.1:   File "/app/.heroku/python/lib/python2.7/site-packages/kombu/transport/pyamqp.py", line 95, in drain_events 
Mar 31 10:47:08 pulse-actions app/worker2.1:     return connection.drain_events(**kwargs) 
Mar 31 10:47:08 pulse-actions app/worker2.1:   File "/app/.heroku/python/lib/python2.7/site-packages/amqp/connection.py", line 326, in drain_events 
Mar 31 10:47:08 pulse-actions app/worker2.1:     return amqp_method(channel, args, content) 
Mar 31 10:47:08 pulse-actions app/worker2.1:   File "/app/.heroku/python/lib/python2.7/site-packages/amqp/channel.py", line 1909, in _basic_deliver 
Mar 31 10:47:08 pulse-actions app/worker2.1:     fun(msg) 
Mar 31 10:47:08 pulse-actions app/worker2.1:   File "/app/.heroku/python/lib/python2.7/site-packages/kombu/messaging.py", line 598, in _receive_callback 
Mar 31 10:47:08 pulse-actions app/worker2.1:     return on_m(message) if on_m else self.receive(decoded, message) 
Mar 31 10:47:08 pulse-actions app/worker2.1:   File "/app/.heroku/python/lib/python2.7/site-packages/kombu/messaging.py", line 564, in receive 
Mar 31 10:47:08 pulse-actions app/worker2.1:     [callback(body, message) for callback in callbacks] 
Mar 31 10:47:08 pulse-actions app/worker2.1:   File "pulse_actions/worker.py", line 68, in handler_with_dry_run 
Mar 31 10:47:08 pulse-actions app/worker2.1:     event_handler(data, message, dry_run) 
Mar 31 10:47:08 pulse-actions app/worker2.1:   File "/app/pulse_actions/handlers/route_functions.py", line 13, in route 
Mar 31 10:47:08 pulse-actions app/worker2.1:     treeherder_buildbot.on_buildbot_event(data, message, dry_run) 
Mar 31 10:47:08 pulse-actions app/worker2.1:   File "/app/pulse_actions/handlers/treeherder_buildbot.py", line 80, in on_buildbot_event 
Mar 31 10:47:08 pulse-actions app/worker2.1:     dry_run=dry_run 
Mar 31 10:47:08 pulse-actions app/worker2.1:   File "/app/.heroku/python/lib/python2.7/site-packages/mozci/mozci.py", line 595, in manual_backfill 
Mar 31 10:47:08 pulse-actions app/worker2.1:     return_revision_list=True) 
Mar 31 10:47:08 pulse-actions app/worker2.1:   File "/app/.heroku/python/lib/python2.7/site-packages/mozhginfo/pushlog_client.py", line 134, in query_pushes_by_specified_revision_range 
Mar 31 10:47:08 pulse-actions app/worker2.1:     raise PushlogError('Unable to retrieve pushlog data. ' 
Mar 31 10:47:08 pulse-actions app/worker2.1: PushlogError: Unable to retrieve pushlog data. Please check repo_url and revision specified.
Attachment #8736714 - Flags: review?(gps)
Attachment #8736821 - Flags: review?(gps)
I've added a generic exception raising test.
Flags: needinfo?(armenzg)
Comment on attachment 8736713 [details]
MozReview Request: Bug 1261075 - Log exceptions

https://reviewboard.mozilla.org/r/43535/#review41075
Attachment #8736713 - Flags: review?(gps) → review+
Comment on attachment 8736714 [details]
MozReview Request: Bump version

https://reviewboard.mozilla.org/r/43537/#review41079
Attachment #8736714 - Flags: review?(gps) → review+
Attachment #8736821 - Flags: review?(gps) → review+
Comment on attachment 8736821 [details]
MozReview Request: Add tests for exception being raised for query_pushes_by_specified_revision_range()

https://reviewboard.mozilla.org/r/43565/#review41081

::: pylib/mozhginfo/mozhginfo/pushlog_client.py:133
(Diff revision 1)
>          push = query_push_by_revision(repo_url, revision)
>          pushid = int(push.id)
>          start_id = pushid - before
>          end_id = pushid + after
>          push_list = query_pushes_by_pushid_range(repo_url, start_id, end_id)
> -    except e:
> +    except Exception, e:

While you're here, you should change to `except Exception as e:` (this is the new exception syntax that isn't compatible with python 2.4/2.5, which should no longer be supported).
I fixed that and released a new package.
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: