Closed Bug 1206910 Opened 9 years ago Closed 9 years ago

hg pushes fail with bzpost exception "KeyError: 'bugs'"

Categories

(Developer Services :: Mercurial: bzpost, defect)

Product:
Developer Services
Component:
Mercurial: bzpost
Type:
defect
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: dustin, Unassigned)

References

Details

Attachments

(1 file)

MozReview Request: Bug 1206910: mozreview test; r?dustin
40 bytes, text/x-review-board-request
dustin
: review+
Details 
I get this about every other push:

pushing to ssh://hg.mozilla.org/try
searching for changes
remote: adding changesets                                                                                                                                                                              
remote: adding manifests
remote: adding file changes
remote: added 2 changesets with 12 changes to 12 files (+1 heads)
remote: recorded push in pushlog
remote: replication of changegroup data completed successfully in 13.6s
remote: 
remote: View your changes here:
remote:   https://hg.mozilla.org/try/rev/6a9c524ab069
remote:   https://hg.mozilla.org/try/rev/ddc2799f4a0e
remote: 
remote: Follow the progress of your build on Treeherder:
remote:   https://treeherder.mozilla.org/#/jobs?repo=try&revision=ddc2799f4a0e
** Unknown exception encountered with possibly-broken third-party extension qimportbz
** which supports versions unknown of Mercurial.
** Please disable qimportbz and try your action again.
** If that fixes the bug please report it to https://bugzilla.mozilla.org/enter_bug.cgi?product=Developer%20Services&component=Mercurial%3A%20qimportbz
** Python 2.7.10 (default, May 30 2015, 12:01:35) [GCC 4.8.4]
** Mercurial Distributed SCM (version 3.3.2)
** Extensions loaded: graphlog, strip, mq, bzexport, bzpost, bundleclone, push-to-try, mqext, qimportbz, color, rebase, purge, histedit, record, pager, shelve, progress, reviewboard, firefoxtree
Traceback (most recent call last):
  File "/usr/lib/python-exec/python2.7/hg", line 43, in <module>
    mercurial.dispatch.run()
  File "/usr/lib64/python2.7/site-packages/mercurial/dispatch.py", line 28, in run
    sys.exit((dispatch(request(sys.argv[1:])) or 0) & 255)
  File "/usr/lib64/python2.7/site-packages/mercurial/dispatch.py", line 71, in dispatch
    ret = _runcatch(req)
  File "/usr/lib64/python2.7/site-packages/mercurial/dispatch.py", line 140, in _runcatch
    return _dispatch(req)
  File "/usr/lib64/python2.7/site-packages/mercurial/dispatch.py", line 863, in _dispatch
    cmdpats, cmdoptions)
  File "/usr/lib64/python2.7/site-packages/mercurial/dispatch.py", line 624, in runcommand
    ret = _runcommand(ui, options, cmd, d)
  File "/usr/lib64/python2.7/site-packages/mercurial/extensions.py", line 194, in wrap
    return wrapper(origfn, *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/hgext/pager.py", line 158, in pagecmd
    return orig(ui, options, cmd, cmdfunc)
  File "/usr/lib64/python2.7/site-packages/mercurial/extensions.py", line 194, in wrap
    return wrapper(origfn, *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/hgext/color.py", line 491, in colorcmd
    return orig(ui_, opts, cmd, cmdfunc)
  File "/usr/lib64/python2.7/site-packages/mercurial/dispatch.py", line 954, in _runcommand
    return checkargs()
  File "/usr/lib64/python2.7/site-packages/mercurial/dispatch.py", line 925, in checkargs
    return cmdfunc()
  File "/usr/lib64/python2.7/site-packages/mercurial/dispatch.py", line 860, in <lambda>
    d = lambda: util.checksignature(func)(ui, *args, **cmdoptions)
  File "/usr/lib64/python2.7/site-packages/mercurial/util.py", line 711, in check
    return func(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/mercurial/extensions.py", line 149, in wrap
    util.checksignature(origfn), *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/mercurial/util.py", line 711, in check
    return func(*args, **kwargs)
  File "/home/dustin/code/moz/.mozbuild/version-control-tools/hgext/firefoxtree/__init__.py", line 353, in pushcommand
    return orig(ui, repo, dest=dest, **opts)
  File "/usr/lib64/python2.7/site-packages/mercurial/util.py", line 711, in check
    return func(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/mercurial/extensions.py", line 149, in wrap
    util.checksignature(origfn), *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/mercurial/util.py", line 711, in check
    return func(*args, **kwargs)
  File "/home/dustin/code/moz/.mozbuild/version-control-tools/hgext/reviewboard/client.py", line 203, in pushcommand
    return orig(ui, repo, *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/mercurial/util.py", line 711, in check
    return func(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/mercurial/extensions.py", line 149, in wrap
    util.checksignature(origfn), *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/mercurial/util.py", line 711, in check
    return func(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/hgext/mq.py", line 3483, in mqcommand
    return orig(ui, repo, *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/mercurial/util.py", line 711, in check
    return func(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/mercurial/commands.py", line 5115, in push
    bookmarks=opts.get('bookmark', ()))
  File "/usr/lib64/python2.7/site-packages/mercurial/extensions.py", line 194, in wrap
    return wrapper(origfn, *args, **kwargs)
  File "/home/dustin/code/moz/.mozbuild/version-control-tools/hgext/firefoxtree/__init__.py", line 189, in push
    **kwargs)
  File "/usr/lib64/python2.7/site-packages/mercurial/extensions.py", line 194, in wrap
    return wrapper(origfn, *args, **kwargs)
  File "/home/dustin/code/moz/.mozbuild/version-control-tools/hgext/reviewboard/client.py", line 222, in wrappedpush
    newbranch=newbranch, **kwargs)
  File "/usr/lib64/python2.7/site-packages/mercurial/exchange.py", line 227, in push
    _pushbookmark(pushop)
  File "/usr/lib64/python2.7/site-packages/mercurial/extensions.py", line 194, in wrap
    return wrapper(origfn, *args, **kwargs)
  File "/home/dustin/code/moz/.mozbuild/version-control-tools/hgext/reviewboard/client.py", line 468, in wrappedpushbookmark
    result = orig(pushop)
  File "/usr/lib64/python2.7/site-packages/mercurial/extensions.py", line 194, in wrap
    return wrapper(origfn, *args, **kwargs)
  File "/home/dustin/code/moz/.mozbuild/version-control-tools/hgext/bzpost/__init__.py", line 166, in wrappedpushbookmark
    comments = bug.get_comments()
  File "/home/dustin/code/moz/.mozbuild/version-control-tools/pylib/Bugsy/bugsy/bug.py", line 250, in get_comments
    in res['bugs'][bug]['comments']]
KeyError: 'bugs'

the push completes, but things explode afterward.

hg 3.3.2.  version-control-tools up to date
Looks like the problem is in bzpost's use of Bugsy.

The qimportbz mention is only because Mercurial blames any enabled extensions that haven't declared version compatibility in the extension config.
Component: Mercurial: qimportbz → Mercurial: bzpost
Possibly related to recent enabling of 2FA?
This is arguably a bug in Bugsy for raising a KeyError (I would think a Bugsy-specific exception or an empty data structure should be returned, depending on the cause of the missing key). But it's also a bug in bzpost for not catching exceptions in Bugsy.

I'm also really curious what the actual Bugzilla response is. If you can reliably reproduce this, could you enter a PDB debugger before the bug.get_comments() call in hgext/bzpost/__init__.py and see what's going on? Or possibly hack up pylib/Bugsy/bugsy/bug.py to print the raw response body or something.

Also, Bugsy stuffs auth foo in a session object. All HTTP requests should automagically send credentials if the Bugsy client was created with credentials. And bzpost does pass available credentials to Bugsy.

To make matters even stranger, I /think/ I recall Dustin telling me that these failures were happening on public bugs. Weird.

dustin: is it too much to ask you to capture some HTTP requests and responses to help us isolate the failure?
Flags: needinfo?(dustin)
It's https (and full of my credentials) so I don't think I can capture it.  And it only seems to happen on repositories with consequences for pushing (try, mozilla-inbound), so experimental pushes to try to replicate sound like a bad idea.

Note that the bug # in the referenced commit above was 189892, which was definitely the wrong bug, but is public.  Here's another instance from my scrollback:

pushing to ssh://hg.mozilla.org/integration/mozilla-inbound
searching for changes
remote: adding changesets
remote: adding manifests
remote: adding file changes
remote: added 1 changesets with 12 changes to 12 files
remote: recorded push in pushlog
remote: replication of changegroup data completed successfully in 2.4s
remote:
remote: View your change here:
remote:   https://hg.mozilla.org/integration/mozilla-inbound/rev/da63c513fe8c
remote:
remote: Follow the progress of your build on Treeherder:
remote:   https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=da63c513fe8c
** Unknown exception encountered with possibly-broken third-party extension qimportbz
** which supports versions unknown of Mercurial.
** Please disable qimportbz and try your action again.
** If that fixes the bug please report it to https://bugzilla.mozilla.org/enter_bug.cgi?product=Developer%20Services&component=Mercurial%3A%20qimportbz
** Python 2.7.10 (default, May 30 2015, 12:01:35) [GCC 4.8.4]
** Mercurial Distributed SCM (version 3.3.2)
** Extensions loaded: graphlog, strip, mq, bzexport, bzpost, bundleclone, push-to-try, mqext, qimportbz, color, rebase, purge, histedit, record, pager, shelve, progress, reviewboard, firefoxtree
Traceback (most recent call last):
  File "/usr/lib/python-exec/python2.7/hg", line 43, in <module>
    mercurial.dispatch.run()
  File "/usr/lib64/python2.7/site-packages/mercurial/dispatch.py", line 28, in run
    sys.exit((dispatch(request(sys.argv[1:])) or 0) & 255)
  File "/usr/lib64/python2.7/site-packages/mercurial/dispatch.py", line 71, in dispatch
    ret = _runcatch(req)
  File "/usr/lib64/python2.7/site-packages/mercurial/dispatch.py", line 140, in _runcatch
    return _dispatch(req)
  File "/usr/lib64/python2.7/site-packages/mercurial/dispatch.py", line 863, in _dispatch
    cmdpats, cmdoptions)
  File "/usr/lib64/python2.7/site-packages/mercurial/dispatch.py", line 624, in runcommand
    ret = _runcommand(ui, options, cmd, d)
  File "/usr/lib64/python2.7/site-packages/mercurial/extensions.py", line 194, in wrap
    return wrapper(origfn, *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/hgext/pager.py", line 158, in pagecmd
    return orig(ui, options, cmd, cmdfunc)
  File "/usr/lib64/python2.7/site-packages/mercurial/extensions.py", line 194, in wrap
    return wrapper(origfn, *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/hgext/color.py", line 491, in colorcmd
    return orig(ui_, opts, cmd, cmdfunc)
  File "/usr/lib64/python2.7/site-packages/mercurial/dispatch.py", line 954, in _runcommand
    return checkargs()
  File "/usr/lib64/python2.7/site-packages/mercurial/dispatch.py", line 925, in checkargs
    return cmdfunc()
  File "/usr/lib64/python2.7/site-packages/mercurial/dispatch.py", line 860, in <lambda>
    d = lambda: util.checksignature(func)(ui, *args, **cmdoptions)
  File "/usr/lib64/python2.7/site-packages/mercurial/util.py", line 711, in check
    return func(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/mercurial/extensions.py", line 149, in wrap
    util.checksignature(origfn), *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/mercurial/util.py", line 711, in check
    return func(*args, **kwargs)
  File "/home/dustin/code/moz/.mozbuild/version-control-tools/hgext/firefoxtree/__init__.py", line 353, in pushcommand
    return orig(ui, repo, dest=dest, **opts)
  File "/usr/lib64/python2.7/site-packages/mercurial/util.py", line 711, in check
    return func(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/mercurial/extensions.py", line 149, in wrap
    util.checksignature(origfn), *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/mercurial/util.py", line 711, in check
    return func(*args, **kwargs)
  File "/home/dustin/code/moz/.mozbuild/version-control-tools/hgext/reviewboard/client.py", line 203, in pushcommand
    return orig(ui, repo, *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/mercurial/util.py", line 711, in check
    return func(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/mercurial/extensions.py", line 149, in wrap
    util.checksignature(origfn), *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/mercurial/util.py", line 711, in check
    return func(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/hgext/mq.py", line 3483, in mqcommand
    return orig(ui, repo, *args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/mercurial/util.py", line 711, in check
    return func(*args, **kwargs)
  File "/usr/lib64/python2.7/site-packages/mercurial/commands.py", line 5115, in push
    bookmarks=opts.get('bookmark', ()))
  File "/usr/lib64/python2.7/site-packages/mercurial/extensions.py", line 194, in wrap
    return wrapper(origfn, *args, **kwargs)
  File "/home/dustin/code/moz/.mozbuild/version-control-tools/hgext/firefoxtree/__init__.py", line 189, in push
    **kwargs)
  File "/usr/lib64/python2.7/site-packages/mercurial/extensions.py", line 194, in wrap
    return wrapper(origfn, *args, **kwargs)
  File "/home/dustin/code/moz/.mozbuild/version-control-tools/hgext/reviewboard/client.py", line 222, in wrappedpush
    newbranch=newbranch, **kwargs)
  File "/usr/lib64/python2.7/site-packages/mercurial/exchange.py", line 227, in push
    _pushbookmark(pushop)
  File "/usr/lib64/python2.7/site-packages/mercurial/extensions.py", line 194, in wrap
    return wrapper(origfn, *args, **kwargs)
  File "/home/dustin/code/moz/.mozbuild/version-control-tools/hgext/reviewboard/client.py", line 468, in wrappedpushbookmark
    result = orig(pushop)
  File "/usr/lib64/python2.7/site-packages/mercurial/extensions.py", line 194, in wrap
    return wrapper(origfn, *args, **kwargs)
  File "/home/dustin/code/moz/.mozbuild/version-control-tools/hgext/bzpost/__init__.py", line 178, in wrappedpushbookmark
    comments = bug.get_comments()
  File "/home/dustin/code/moz/.mozbuild/version-control-tools/pylib/Bugsy/bugsy/bug.py", line 250, in get_comments
    in res['bugs'][bug]['comments']]
KeyError: 'bugs'

again, public (and correct!) bugs this time.

An experimental push to my user repo didn't trigger any such error
Flags: needinfo?(dustin)
I meant to enter a Python debugger and to set a breakpoint at https://hg.mozilla.org/hgcustom/version-control-tools/file/b1dcdbd87406/pylib/Bugsy/bugsy/bug.py#l247. res.text or res.json() should tell us what's going on here.
diff --git a/pylib/Bugsy/bugsy/bug.py b/pylib/Bugsy/bugsy/bug.py
--- a/pylib/Bugsy/bugsy/bug.py
+++ b/pylib/Bugsy/bugsy/bug.py
@@ -240,16 +240,17 @@ class Bug(object):
     def get_comments(self):
         """
             Obtain comments for this bug.
 
             Returns a list of Comment instances.
         """
         bug = unicode(self._bug['id'])
         res = self._bugsy.request('bug/%s/comment' % bug).json()
+        print res.text
 
         return [Comment(**comments) for comments
                 in res['bugs'][bug]['comments']]
 
     def add_comment(self, comment):
         """
             Adds a comment to a bug. If a bug does not have a bug ID then
             you need call `put` on the :class:`Bugsy` class.

->

AttributeError: 'dict' object has no attribute 'text'

oops :)
tanvi: switch to API Keys. `mach mercurial-setup` will guide you through this.
Tanvi, your error is unrelated to both this bug and qimportbz - please read exceptions fully :-)
Summary: hg pushes fail with long traceback → hg pushes fail with bzpost exception "KeyError: 'bugs'"
I am guessing that we didnt get a 200 error here and got an error. I can't see to reproduce so am flying blind on the bugsy side
I'm facing the same error on every push. This is what I get when I print str(res) (comment#7):

{u'code': 67399, u'message': u"The requested method 'Bug.comments' was not found.", u'documentation': u'http://www.bugzilla.org/docs/tip/en/html/api/', u'error': True}
:sebastian

Thank you for that, I will get Bugsy fixed ASAP.
I have fixed Bugsy locally. Need a few more tweaks and then will release
Depends on: 1212856
With the new version of Bugsy the error message is more helpful:
> bugsy.errors.BugsyException: Message: The requested method 'Bug.comments' was not found.

But I still have no idea why this error is happening every time I push to try or fx-team.
If that causes a nonzero exit status, then it will still mess up my scripts :(
Is that Bugsy fix in version-control-tools?  I updated v-c-t this morning and just got the long traceback again.
The bugsy fix is in, see bug 1214615 showing the better Bugsy message, however the long traceback still needs to be fixed.
Worth noting that this also prevents 'hg commit' from marking csets as published.
I can confirm that bugsy is now handling this correctly:

  bugsy.errors.BugsyException: Message: The requested method 'Bug.comments' was not found.

So now I'm getting a different exception, which doesn't really help me very much.  I'm still having to manually fix up the phases in my repos on every commit, and this messes up scripts I have that do `hg push`, since it spuriously returns a nonzero exit status.  Can we get this fixed please??
Severity: normal → major
(In reply to Dustin J. Mitchell [:dustin] from comment #21)
> I can confirm that bugsy is now handling this correctly:
> 
>   bugsy.errors.BugsyException: Message: The requested method 'Bug.comments'
> was not found.
> 
> So now I'm getting a different exception, which doesn't really help me very
> much.  I'm still having to manually fix up the phases in my repos on every
> commit, and this messes up scripts I have that do `hg push`, since it
> spuriously returns a nonzero exit status.  Can we get this fixed please??

https://bugzilla.mozilla.org/show_bug.cgi?id=1214615 was raised before and was down to bad API key. If that doesnt help then we may need to get the BMO team to look into in a new bug since Bug.comments is a problem coming out of BMO
My API key is good -- I use it for mozreview (in other repos) all the time.
(In reply to Dustin J. Mitchell [:dustin] from comment #23)
> My API key is good -- I use it for mozreview (in other repos) all the time.

My API key worked fine for mozreview too - but raised this exception when pushing to some other repos. The new API key I generated then worked with all repositories (actually bugzilla) and reviewboard.
Indeed, that does help.  Which is a little crazy - this is my fourth API key, as they keep breaking.  But that's a BZ bug so this one is closed: resolution: keep generating new BZ API keys until everything that uses it works at the same time.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Bug 1206910: mozreview test; r?dustin
Attachment #8683643 - Flags: review?(dustin)
Attachment #8683643 - Flags: review?(dustin) → review+
Comment on attachment 8683643 [details]
MozReview Request: Bug 1206910: mozreview test; r?dustin

https://reviewboard.mozilla.org/r/24399/#review21863

mozreview works too! wow!
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: