1183656 - "UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 6: ordinal not in range(128)"

Status: VERIFIED FIXED

(Developer Services :: Mercurial: bzexport, defect)

Description

[Nicholas Nethercote [inactive]]

I just tried uploading a patch with bzexport. I set the reviewer to ":dao" and I got the following exception.

> ** Unknown exception encountered with possibly-broken third-party extension bzexport
> ** which supports versions 3.1 of Mercurial.
> ** Please disable bzexport and try your action again.
> ** If that fixes the bug please report it to https://bugzilla.mozilla.org/enter_bug.cgi?product=Developer%20Services&component=Mercurial%3A%20bzexport
> ** Python 2.7.9 (default, Apr  2 2015, 15:33:21) [GCC 4.9.2]
> ** Mercurial Distributed SCM (version 3.1.2)
> ** Extensions loaded: strip, mq, rebase, convert, hgk, extdiff, progress, record, histedit, bzexport, color, transplant, graphlog, relink
> Traceback (most recent call last):
>   File "/usr/bin/hg", line 43, in <module>
>     mercurial.dispatch.run()
>   File "/usr/lib/python2.7/dist-packages/mercurial/dispatch.py", line 28, in run
>     sys.exit((dispatch(request(sys.argv[1:])) or 0) & 255)
>   File "/usr/lib/python2.7/dist-packages/mercurial/dispatch.py", line 69, in dispatch
>     ret = _runcatch(req)
>   File "/usr/lib/python2.7/dist-packages/mercurial/dispatch.py", line 138, in _runcatch
>     return _dispatch(req)
>   File "/usr/lib/python2.7/dist-packages/mercurial/dispatch.py", line 839, in _dispatch
>     cmdpats, cmdoptions)
>   File "/usr/lib/python2.7/dist-packages/mercurial/dispatch.py", line 600, in runcommand
>     ret = _runcommand(ui, options, cmd, d)
>   File "/usr/lib/python2.7/dist-packages/mercurial/extensions.py", line 196, in wrap
>     return wrapper(origfn, *args, **kwargs)
>   File "/usr/lib/python2.7/dist-packages/hgext/color.py", line 433, in colorcmd
>     return orig(ui_, opts, cmd, cmdfunc)
>   File "/usr/lib/python2.7/dist-packages/mercurial/dispatch.py", line 930, in _runcommand
>     return checkargs()
>   File "/usr/lib/python2.7/dist-packages/mercurial/dispatch.py", line 901, in checkargs
>     return cmdfunc()
>   File "/usr/lib/python2.7/dist-packages/mercurial/dispatch.py", line 836, in <lambda>
>     d = lambda: util.checksignature(func)(ui, *args, **cmdoptions)
>   File "/usr/lib/python2.7/dist-packages/mercurial/util.py", line 550, in check
>     return func(*args, **kwargs)
>   File "/usr/lib/python2.7/dist-packages/mercurial/dispatch.py", line 458, in __call__
>     return util.checksignature(self.fn)(ui, *args, **opts)
>   File "/usr/lib/python2.7/dist-packages/mercurial/util.py", line 550, in check
>     return func(*args, **kwargs)
>   File "/usr/lib/python2.7/dist-packages/mercurial/extensions.py", line 151, in wrap
>     util.checksignature(origfn), *args, **kwargs)
>   File "/usr/lib/python2.7/dist-packages/mercurial/util.py", line 550, in check
>     return func(*args, **kwargs)
>   File "/usr/lib/python2.7/dist-packages/hgext/mq.py", line 3391, in mqcommand
>     return orig(ui, repo, *args, **kwargs)
>   File "/usr/lib/python2.7/dist-packages/mercurial/util.py", line 550, in check
>     return func(*args, **kwargs)
>   File "/home/njn/.mozbuild/version-control-tools/hgext/bzexport/__init__.py", line 958, in bzexport
>     users = validate_users(ui, api_server, auth, search_strings, multi_user_prompt, 'reviewer')
>   File "/home/njn/.mozbuild/version-control-tools/hgext/bzexport/__init__.py", line 217, in validate_users
>     user = multi_callback(ui, multi_desc, search_result)
>   File "/home/njn/.mozbuild/version-control-tools/hgext/bzexport/__init__.py", line 161, in multi_user_prompt
>     allow_none=True)
>   File "/home/njn/.mozbuild/version-control-tools/hgext/bzexport/__init__.py", line 124, in prompt_menu
>     message += "  %d. %s\n" % ((i + 1), value.encode('utf-8', 'replace'))
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 6: ordinal not in range(128)

":dao" matches two names:

- Dão Gottwald [:dao] ([e-mail])

- Daosheng Mu[:daoshengmu] ([e-mail])

I think the exception occurred when bzexport tried to ask me which one I wanted. Presumably it's the 'ã' char that's causing the problem.

I re-exported using "[e-mail]" instead and that worked fine.

Comment 1

[Nicholas Nethercote [inactive]]

Note that this is similar to bug 798234.

Comment 2

[Gregory Szorc [:gps]]

There is almost certainly a mix of str and unicode types at play. Python 2 will attempt to .decode('ascii', 'strict') or .encode('ascii', 'strict') to normalize the types. If the values aren't ASCII, you get UnicodeDecodeError or UnicodeEncodeError.

The trick here is ensuring all variables are of the same type so type coercion doesn't occur.

Comment 3

[Gregory Szorc [:gps]]

Also, why are you using bzexport instead of MozReview?

Comment 4

[Nicholas Nethercote [inactive]]

I just hit this again with dbaron, who uses a clock (watch?) emoji in his Bugzilla name.

Comment 5

[Gregory Szorc [:gps]]

The stack trace in the initial comment doesn't match current reality. Please upload a new stack trace. Make sure you are running the latest version-control-tools checkout. Post the changeset SHA-1 along with the stack.

You can get hg to emit a stack by adding --traceback to any command.

Comment 6

[Steve Fink [:sfink] [:s:]]

(In reply to Nicholas Nethercote [:njn] from comment #4)
> I just hit this again with dbaron, who uses a clock (watch?) emoji in his
> Bugzilla name.

This is an error handling error. The real problem is:

Abort: error uploading attachment: REST error on POST to https://bugzilla.mozilla.org/rest/bug/944164/attachment: David Baron [:dbaron] \u231a\ufe0fUTC-7 <[e-mail]> is not currently accepting 'review' requests.

which is about the bugzilla request blocking stuff. But when attempting to report the error, it gets the decoding error. I can reproduce the error by call 

  File "/home/sfink/lib/version-control-tools/hgext/bzexport/__init__.py", line 1200, in bzexport
    raise util.Abort('error uploading attachment: %s' % e.message)
Abort: error uploading attachment: REST error on POST to https://bugzilla.mozilla.org/rest/bug/944164/attachment: David Baron [:dbaron] \u231a\ufe0fUTC-7 <[e-mail]> is not currently accepting 'review' requests.
Traceback (most recent call last):
  File "/home/sfink/bin/hg", line 43, in <module>
    mercurial.dispatch.run()
  File "/home/sfink/src/hg/root-3.7.3/lib64/python2.7/site-packages/mercurial/dispatch.py", line 54, in run
    sys.exit((dispatch(request(sys.argv[1:])) or 0) & 255)
  File "/home/sfink/src/hg/root-3.7.3/lib64/python2.7/site-packages/mercurial/dispatch.py", line 120, in dispatch
    ret = _runcatch(req)
  File "/home/sfink/src/hg/root-3.7.3/lib64/python2.7/site-packages/mercurial/dispatch.py", line 277, in _runcatch
    ui.warn(_("abort: %s\n") % inst)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 129-130: ordinal not in range(128)

Comment 7

[Steve Fink [:sfink] [:s:]]

oops, sorry, I mangled that. I can reproduce the error by calling

  raise util.Abort(u"\u231a")

Comment 8

[Steve Fink [:sfink] [:s:]]

or by running

  "uh oh: %s" % util.Abort(u"\u231a")

Comment 9

[Gregory Szorc [:gps]]

Yeah, Mercurial uses byte strings for everything internally. We're "leaking" a Python unicode type somewhere, leading to an implicit conversion. My guess is the leak is from the parsed JSON response payload (Python's json module likes converting things to unicode type).

You want to take a go at fixing it?

Comment 10

[Nicholas Nethercote [inactive]]

New stack from running |hg --traceback bzexport -e| and using ":dao" for the reviewer:

> Traceback (most recent call last):
>   File "/usr/local/lib/python2.7/dist-packages/mercurial/dispatch.py", line 187, in _runcatch
>     return _dispatch(req)
>   File "/usr/local/lib/python2.7/dist-packages/mercurial/dispatch.py", line 920, in _dispatch
>     cmdpats, cmdoptions)
>   File "/usr/local/lib/python2.7/dist-packages/mercurial/dispatch.py", line 679, in runcommand
>     ret = _runcommand(ui, options, cmd, d)
>   File "/usr/local/lib/python2.7/dist-packages/mercurial/extensions.py", line 183, in closure
>     return func(*(args + a), **kw)
>   File "/usr/local/lib/python2.7/dist-packages/hgext/color.py", line 525, in colorcmd
>     return orig(ui_, opts, cmd, cmdfunc)
>   File "/usr/local/lib/python2.7/dist-packages/mercurial/dispatch.py", line 1051, in _runcommand
>     return checkargs()
>   File "/usr/local/lib/python2.7/dist-packages/mercurial/dispatch.py", line 1011, in checkargs
>     return cmdfunc()
>   File "/usr/local/lib/python2.7/dist-packages/mercurial/dispatch.py", line 917, in <lambda>
>     d = lambda: util.checksignature(func)(ui, *args, **cmdoptions)
>   File "/usr/local/lib/python2.7/dist-packages/mercurial/util.py", line 801, in check
>     return func(*args, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/mercurial/dispatch.py", line 537, in __call__
>     return util.checksignature(self.fn)(ui, *args, **opts)
>   File "/usr/local/lib/python2.7/dist-packages/mercurial/util.py", line 801, in check
>     return func(*args, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/mercurial/extensions.py", line 183, in closure
>     return func(*(args + a), **kw)
>   File "/usr/local/lib/python2.7/dist-packages/mercurial/util.py", line 801, in check
>     return func(*args, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/hgext/mq.py", line 3525, in mqcommand
>     return orig(ui, repo, *args, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/mercurial/util.py", line 801, in check
>     return func(*args, **kwargs)
>   File "/home/njn/.mozbuild/version-control-tools/hgext/bzexport/__init__.py", line 1001, in bzexport
>     users = validate_users(ui, api_server, auth, search_strings, multi_user_prompt, 'reviewer')
>   File "/home/njn/.mozbuild/version-control-tools/hgext/bzexport/__init__.py", line 225, in validate_users
>     user = multi_callback(ui, multi_desc, search_result)
>   File "/home/njn/.mozbuild/version-control-tools/hgext/bzexport/__init__.py", line 169, in multi_user_prompt
>     allow_none=True)
>   File "/home/njn/.mozbuild/version-control-tools/hgext/bzexport/__init__.py", line 132, in prompt_menu
>     message += "  %d. %s\n" % ((i + 1), value.encode('utf-8', 'replace'))
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 6: ordinal not in range(128)
> ** unknown exception encountered, please report by visiting
> ** https://mercurial-scm.org/wiki/BugTracker
> ** Python 2.7.10 (default, Oct 14 2015, 16:09:02) [GCC 5.2.1 20151010]
> ** Mercurial Distributed SCM (version 3.6)
> ** Extensions loaded: strip, mq, rebase, convert, hgk, extdiff, progress, record, histedit, bzexport, color, transplant, graphlog, relink, bundleclone, mqext, qimportbz, bzpost, shelve, reviewboard
> Traceback (most recent call last):
>   File "/usr/local/bin/hg", line 43, in <module>
>     mercurial.dispatch.run()
>   File "/usr/local/lib/python2.7/dist-packages/mercurial/dispatch.py", line 54, in run
>     sys.exit((dispatch(request(sys.argv[1:])) or 0) & 255)
>   File "/usr/local/lib/python2.7/dist-packages/mercurial/dispatch.py", line 116, in dispatch
>     ret = _runcatch(req)
>   File "/usr/local/lib/python2.7/dist-packages/mercurial/dispatch.py", line 187, in _runcatch
>     return _dispatch(req)
>   File "/usr/local/lib/python2.7/dist-packages/mercurial/dispatch.py", line 920, in _dispatch
>     cmdpats, cmdoptions)
>   File "/usr/local/lib/python2.7/dist-packages/mercurial/dispatch.py", line 679, in runcommand
>     ret = _runcommand(ui, options, cmd, d)
>   File "/usr/local/lib/python2.7/dist-packages/mercurial/extensions.py", line 183, in closure
>     return func(*(args + a), **kw)
>   File "/usr/local/lib/python2.7/dist-packages/hgext/color.py", line 525, in colorcmd
>     return orig(ui_, opts, cmd, cmdfunc)
>   File "/usr/local/lib/python2.7/dist-packages/mercurial/dispatch.py", line 1051, in _runcommand
>     return checkargs()
>   File "/usr/local/lib/python2.7/dist-packages/mercurial/dispatch.py", line 1011, in checkargs
>     return cmdfunc()
>   File "/usr/local/lib/python2.7/dist-packages/mercurial/dispatch.py", line 917, in <lambda>
>     d = lambda: util.checksignature(func)(ui, *args, **cmdoptions)
>   File "/usr/local/lib/python2.7/dist-packages/mercurial/util.py", line 801, in check
>     return func(*args, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/mercurial/dispatch.py", line 537, in __call__
>     return util.checksignature(self.fn)(ui, *args, **opts)
>   File "/usr/local/lib/python2.7/dist-packages/mercurial/util.py", line 801, in check
>     return func(*args, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/mercurial/extensions.py", line 183, in closure
>     return func(*(args + a), **kw)
>   File "/usr/local/lib/python2.7/dist-packages/mercurial/util.py", line 801, in check
>     return func(*args, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/hgext/mq.py", line 3525, in mqcommand
>     return orig(ui, repo, *args, **kwargs)
>   File "/usr/local/lib/python2.7/dist-packages/mercurial/util.py", line 801, in check
>     return func(*args, **kwargs)
>   File "/home/njn/.mozbuild/version-control-tools/hgext/bzexport/__init__.py", line 1001, in bzexport
>     users = validate_users(ui, api_server, auth, search_strings, multi_user_prompt, 'reviewer')
>   File "/home/njn/.mozbuild/version-control-tools/hgext/bzexport/__init__.py", line 225, in validate_users
>     user = multi_callback(ui, multi_desc, search_result)
>   File "/home/njn/.mozbuild/version-control-tools/hgext/bzexport/__init__.py", line 169, in multi_user_prompt
>     allow_none=True)
>   File "/home/njn/.mozbuild/version-control-tools/hgext/bzexport/__init__.py", line 132, in prompt_menu
>     message += "  %d. %s\n" % ((i + 1), value.encode('utf-8', 'replace'))
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 6: ordinal not in range(128)


The patch SHA1 is 2ab7245aea0a.

Comment 11

[Nicholas Nethercote [inactive]]

(In reply to Steve Fink [:sfink] [:s:] from comment #6)
> > I just hit this again with dbaron, who uses a clock (watch?) emoji in his
> > Bugzilla name.
> 
> This is an error handling error. The real problem is:
> 
> Abort: error uploading attachment: REST error on POST to
> https://bugzilla.mozilla.org/rest/bug/944164/attachment: David Baron
> [:dbaron] \u231a\ufe0fUTC-7 <[e-mail]> is not currently accepting
> 'review' requests.
> 
> which is about the bugzilla request blocking stuff. But when attempting to
> report the error, it gets the decoding error.

A decoding error also occurs if the person *is* accepting -- comment 10 is from a review request for ":dao". (At least, I assume that Dão is currently accepting review requests... I don't know how to check that easily.)

I don't know if the two cases (accepting reviews and rejecting reviews) have the same codepaths, but the failures certainly seem similar.

Comment 12

[Gregory Szorc [:gps]]

Attachment: bzexport: don't implicitly convert between unicode and str ()

Comment 13

[Gregory Szorc [:gps]]

Attachment: MozReview Request: bzexport: pass str to prompt_menu (bug 1183656); r?njn

prompt_menu expects a str message. search_results['search_string']
is a unicode type. When interpolated, Python 2 unhelpfully implicitly
converts the literal str to a unicode. This later triggers
implicit conversion back to a str assuming the ASCII encoding.

Review commit: https://reviewboard.mozilla.org/r/48895/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/48895/

Comment 14

[Nicholas Nethercote [inactive]]

Comment on attachment 8745175 [details]
MozReview Request: bzexport: pass str to prompt_menu (bug 1183656); r?njn

https://reviewboard.mozilla.org/r/48895/#review45733

Seems plausible, but I'm surprised to be asked for the review. Did you want me to test the new code?

Comment 15

[Gregory Szorc [:gps]]

https://reviewboard.mozilla.org/r/48895/#review45733

It was a trivial patch: I figured you could rubber stamp it :)

Comment 16

[Gregory Szorc [:gps]]

Autolanded.

Comment 17

[Nicholas Nethercote [inactive]]

I just hit this again (using ":mt" as the reviewer, which gave a prompt that included "Martin Törnwall". And then I updated my version-control-tools, tried again, and then it worked. Yay.