1187522 - "Unknown exception encountered with possibly-broken third-party extension crecord" on push due to UnicodeDecodeError

Status: RESOLVED FIXED

(Developer Services :: Mercurial: bzpost, defect)

Description

[aleth [:aleth]]

I doubt crecord is actually responsible though. Maybe it's related to the patch author of the commit that bzpost struggled with having a character with an accent in his name?

    Are you sure you want to push to remote? (y/n): y
    pushing to ssh://hg.mozilla.org/comm-central/
    no revisions specified to push; using . to avoid pushing multiple heads
    searching for changes
    remote: adding changesets                                   
    remote: adding manifests
    remote: adding file changes
    remote: added 3 changesets with 5 changes to 5 files
    remote: comm-central-thunderbird is CLOSED! Reason: bug 1186782
    remote: But you included the magic words.  Hope you had permission!
    remote: Trying to insert into pushlog.
    remote: Inserted into the pushlog db successfully.
    remote:
    remote: View your changes here:
    remote:   https://hg.mozilla.org/comm-central/rev/7ef320f10b21
    remote:   https://hg.mozilla.org/comm-central/rev/65cab1c8cef3
    remote:   https://hg.mozilla.org/comm-central/rev/a77149516698
    remote:
    remote: Follow the progress of your build on Treeherder:
    remote:   https://treeherder.mozilla.org/#/jobs?repo=comm-central&revision=a77149516698
    recording push at https://bugzilla.mozilla.org/show_bug.cgi?id=1136396
    recording push at https://bugzilla.mozilla.org/show_bug.cgi?id=1186414
    ** Unknown exception encountered with possibly-broken third-party extension crecord
    ** which supports versions unknown of Mercurial.
    ** Please disable crecord and try your action again.
    ** If that fixes the bug please report it to the extension author.
    ** Python 2.7.6 (default, Sep  9 2014, 15:04:36) [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)]
    ** Mercurial Distributed SCM (version 3.4.2)
    ** Extensions loaded: pager, strip, mq, mqext, color, progress, rebase, crecord, shelve, purge, bzexport, qimportbz, relink, histedit, reviewboard, bzpost, firefoxtree, churn, bundleclone, push-to-try
    Traceback (most recent call last):
      File "/usr/local/bin/hg", line 43, in <module>
        mercurial.dispatch.run()
      File "/usr/local/Cellar/mercurial/3.4.2/lib/python2.7/site-packages/mercurial/dispatch.py", line 29, in run
        sys.exit((dispatch(request(sys.argv[1:])) or 0) & 255)
      File "/usr/local/Cellar/mercurial/3.4.2/lib/python2.7/site-packages/mercurial/dispatch.py", line 91, in dispatch
        ret = _runcatch(req)
      File "/usr/local/Cellar/mercurial/3.4.2/lib/python2.7/site-packages/mercurial/dispatch.py", line 160, in _runcatch
        return _dispatch(req)
      File "/usr/local/Cellar/mercurial/3.4.2/lib/python2.7/site-packages/mercurial/dispatch.py", line 885, in _dispatch
        cmdpats, cmdoptions)
      File "/usr/local/Cellar/mercurial/3.4.2/lib/python2.7/site-packages/mercurial/dispatch.py", line 646, in runcommand
        ret = _runcommand(ui, options, cmd, d)
      File "/usr/local/Cellar/mercurial/3.4.2/lib/python2.7/site-packages/mercurial/extensions.py", line 168, in closure
        return func(*(args + a), **kw)
      File "/usr/local/Cellar/mercurial/3.4.2/lib/python2.7/site-packages/hgext/color.py", line 521, in colorcmd
        return orig(ui_, opts, cmd, cmdfunc)
      File "/usr/local/Cellar/mercurial/3.4.2/lib/python2.7/site-packages/mercurial/extensions.py", line 168, in closure
        return func(*(args + a), **kw)
      File "/usr/local/Cellar/mercurial/3.4.2/lib/python2.7/site-packages/hgext/pager.py", line 160, in pagecmd
        return orig(ui, options, cmd, cmdfunc)
      File "/usr/local/Cellar/mercurial/3.4.2/lib/python2.7/site-packages/mercurial/dispatch.py", line 976, in _runcommand
        return checkargs()
      File "/usr/local/Cellar/mercurial/3.4.2/lib/python2.7/site-packages/mercurial/dispatch.py", line 947, in checkargs
        return cmdfunc()
      File "/usr/local/Cellar/mercurial/3.4.2/lib/python2.7/site-packages/mercurial/dispatch.py", line 882, in <lambda>
        d = lambda: util.checksignature(func)(ui, *args, **cmdoptions)
      File "/usr/local/Cellar/mercurial/3.4.2/lib/python2.7/site-packages/mercurial/util.py", line 716, in check
        return func(*args, **kwargs)
      File "/usr/local/Cellar/mercurial/3.4.2/lib/python2.7/site-packages/mercurial/extensions.py", line 168, in closure
        return func(*(args + a), **kw)
      File "/usr/local/Cellar/mercurial/3.4.2/lib/python2.7/site-packages/mercurial/util.py", line 716, in check
        return func(*args, **kwargs)
      File "/Users/helvellyn/.mozbuild/version-control-tools/hgext/firefoxtree/__init__.py", line 353, in pushcommand
        return orig(ui, repo, dest=dest, **opts)
      File "/usr/local/Cellar/mercurial/3.4.2/lib/python2.7/site-packages/mercurial/util.py", line 716, in check
        return func(*args, **kwargs)
      File "/usr/local/Cellar/mercurial/3.4.2/lib/python2.7/site-packages/mercurial/extensions.py", line 168, in closure
        return func(*(args + a), **kw)
      File "/usr/local/Cellar/mercurial/3.4.2/lib/python2.7/site-packages/mercurial/util.py", line 716, in check
        return func(*args, **kwargs)
      File "/Users/helvellyn/.mozbuild/version-control-tools/hgext/reviewboard/client.py", line 182, in pushcommand
        return orig(ui, repo, *args, **kwargs)
      File "/usr/local/Cellar/mercurial/3.4.2/lib/python2.7/site-packages/mercurial/util.py", line 716, in check
        return func(*args, **kwargs)
      File "/usr/local/Cellar/mercurial/3.4.2/lib/python2.7/site-packages/mercurial/extensions.py", line 168, in closure
        return func(*(args + a), **kw)
      File "/usr/local/Cellar/mercurial/3.4.2/lib/python2.7/site-packages/mercurial/util.py", line 716, in check
        return func(*args, **kwargs)
      File "/usr/local/Cellar/mercurial/3.4.2/lib/python2.7/site-packages/hgext/mq.py", line 3505, in mqcommand
        return orig(ui, repo, *args, **kwargs)
      File "/usr/local/Cellar/mercurial/3.4.2/lib/python2.7/site-packages/mercurial/util.py", line 716, in check
        return func(*args, **kwargs)
      File "/usr/local/Cellar/mercurial/3.4.2/lib/python2.7/site-packages/mercurial/commands.py", line 5234, in push
        bookmarks=opts.get('bookmark', ()))
      File "/usr/local/Cellar/mercurial/3.4.2/lib/python2.7/site-packages/mercurial/extensions.py", line 168, in closure
        return func(*(args + a), **kw)
      File "/Users/helvellyn/.mozbuild/version-control-tools/hgext/firefoxtree/__init__.py", line 189, in push
        **kwargs)
      File "/usr/local/Cellar/mercurial/3.4.2/lib/python2.7/site-packages/mercurial/extensions.py", line 168, in closure
        return func(*(args + a), **kw)
      File "/Users/helvellyn/.mozbuild/version-control-tools/hgext/reviewboard/client.py", line 201, in wrappedpush
        newbranch=newbranch, **kwargs)
      File "/usr/local/Cellar/mercurial/3.4.2/lib/python2.7/site-packages/mercurial/exchange.py", line 239, in push
        _pushbookmark(pushop)
      File "/usr/local/Cellar/mercurial/3.4.2/lib/python2.7/site-packages/mercurial/extensions.py", line 168, in closure
        return func(*(args + a), **kw)
      File "/Users/user/.mozbuild/version-control-tools/hgext/bzpost/__init__.py", line 202, in wrappedpushbookmark
        comment = '\n'.join(lines)
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 22: ordinal not in range(128)

Comment 1

[Gregory Szorc [:gps]]

It looks like bzpost isn't unicode safe. Looks like "Florian Quèze" from 65cab1c8cef3 tripped it up.

Should be relatively easy to fix.

Comment 2

[Gregory Szorc [:gps]]

Attachment: MozReview Request: bzpost: properly handle unicode in user and description fields (bug 1187522); r?smacleod

bzpost: properly handle unicode in user and description fields (bug 1187522); r?smacleod

This was more complicated than I thought it would be.

ctx.user() and ctx.description() return a special "local string" class
from mercurial.encoding. This behaves like a str but it is encoded in a
way that the locally configured encoding supports. So, unsupported code
points may render as "?" (or whatever the replacement character is). We
need to call encoding.fromlocal() to obtain the raw bytes (as a str).

Once this was done, I discovered that "baseuri" was a unicode type. This
means that the "lines" list consisted of mixed unicode and str and that
triggered an implicit conversion during the '\n'.join(), which failed
due to the non-ascii. This type is unicode because the module it comes
from is using unicode_literals. So I converted it to a str.

Comment 3

[Steven MacLeod [:smacleod]]

Comment on attachment 8638857 [details]
MozReview Request: bzpost: properly handle unicode in user and description fields (bug 1187522); r?smacleod

https://reviewboard.mozilla.org/r/14149/#review12797

Ship It!

Comment 4

[Gregory Szorc [:gps]]

url:        https://hg.mozilla.org/hgcustom/version-control-tools/rev/a57982fbbfbe46b51d8573ebf6c60070b278cbab
changeset:  a57982fbbfbe46b51d8573ebf6c60070b278cbab
user:       Gregory Szorc <[e-mail]>
date:       Mon Jul 27 11:45:16 2015 -0700
description:
bzpost: properly handle unicode in user and description fields (bug 1187522); r=smacleod

This was more complicated than I thought it would be.

ctx.user() and ctx.description() return a special "local string" class
from mercurial.encoding. This behaves like a str but it is encoded in a
way that the locally configured encoding supports. So, unsupported code
points may render as "?" (or whatever the replacement character is). We
need to call encoding.fromlocal() to obtain the raw bytes (as a str).

Once this was done, I discovered that "baseuri" was a unicode type. This
means that the "lines" list consisted of mixed unicode and str and that
triggered an implicit conversion during the '\n'.join(), which failed
due to the non-ascii. This type is unicode because the module it comes
from is using unicode_literals. So I converted it to a str.