1268128 - ISEs for json-automationrelevance

Status: VERIFIED FIXED

(Developer Services :: General, task)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

https://hg.mozilla.org/mozilla-central/json-automationrelevance/ab0044bfa1df858919797bcd6a9aef76a668cd4a

is causing an ISE

Comment 1

[Kendall Libby [:fubar] (he/him)]

hgweb12.dmz.scl3# curl -H "Host: hg.mozilla.org" http://localhost/mozilla-central/json-automationrelevance/ab0044bfa1df858919797bcd6a9aef76a668cd4a

hgweb12.dmz.scl3# grep '127.0.0.1' error_log | cut -d " " -f 11-

mod_wsgi (pid=18421): Exception occurred processing WSGI script '/repo_local/mozilla/webroot_wsgi/hgweb.wsgi'.
Traceback (most recent call last):
  File "/var/hg/venv_hgweb/lib/python2.7/site-packages/mercurial/hgweb/hgweb_mod.py", line 306, in run_wsgi
    for r in self._runwsgi(req, repo):
  File "/var/hg/version-control-tools/hgext/serverlog/__init__.py", line 273, in _runwsgi
    for what in super(hgwebwrapped, self)._runwsgi(req, repo):
  File "/var/hg/venv_hgweb/lib/python2.7/site-packages/mercurial/hgweb/hgweb_mod.py", line 421, in _runwsgi
    content = getattr(webcommands, cmd)(rctx, req, tmpl)
  File "/var/hg/version-control-tools/hgext/hgmo/__init__.py", line 409, in automationrelevancewebcommand
    return json.dumps({'changesets': csets}, indent=2, sort_keys=True)
  File "/usr/lib64/python2.7/json/__init__.py", line 250, in dumps
    sort_keys=sort_keys, **kw).encode(obj)
  File "/usr/lib64/python2.7/json/encoder.py", line 209, in encode
    chunks = list(chunks)
  File "/usr/lib64/python2.7/json/encoder.py", line 434, in _iterencode
    for chunk in _iterencode_dict(o, _current_indent_level):
  File "/usr/lib64/python2.7/json/encoder.py", line 408, in _iterencode_dict
    for chunk in chunks:
  File "/usr/lib64/python2.7/json/encoder.py", line 332, in _iterencode_list
    for chunk in chunks:
  File "/usr/lib64/python2.7/json/encoder.py", line 408, in _iterencode_dict
    for chunk in chunks:
  File "/usr/lib64/python2.7/json/encoder.py", line 390, in _iterencode_dict
    yield _encoder(value)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xfb in position 1: invalid start byte

Comment 2

[Gregory Szorc [:gps]]

I can reproduce this.

Comment 3

[Gregory Szorc [:gps]]

This is getting tripped up by ce9e9e06595a486bbd36732cfef1fa6296e049b1 and its transplant_source extra value, which is binary:

{'author': 'gasolin <gasolin@gasolin-BM6875-BM6675-BP6375>',
 'backsoutnodes': [],
 'bugs': [{'no': '1011023',
           'url': 'https://bugzilla.mozilla.org/show_bug.cgi?id=1011023'}],
 'date': (1460449954.0, -28800),
 'desc': 'Bug 1011023 - Simplify test_bookmarks_restore_notification.js to use add_task. r=mak\n\nMozReview-Commit-ID: Bj1jMdnIbvF',
 'extra': {'branch': 'default',
           'transplant_source': '\x0c\xfb\x98-\x97[\xc3r\xba\x021\xff\xa3n\xf4\x14g4\x8bC'},
 'files': ['toolkit/components/places/tests/unit/test_bookmarks_restore_notification.js'],
 'node': 'ce9e9e06595a486bbd36732cfef1fa6296e049b1',
 'parent': [],
 'pushdate': (1461751093, 25200),
 'pushhead': 'ab0044bfa1df858919797bcd6a9aef76a668cd4a',
 'pushid': 30217,

Apparently that's how the transplant extension rolls :/

Comment 4

[Gregory Szorc [:gps]]

Attachment: MozReview Request: pushlog: convert unicode to str (bug 1268128); r?smacleod

SQLite returns unicode instances. Mercurial deals with str everywhere
internally. Convert the node value to str so we're consistent.

This doesn't fix bug 1268128, but it does contribute to mixed
str and unicode, which is never a good thing.

Review commit: https://reviewboard.mozilla.org/r/49295/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/49295/

Comment 5

[Gregory Szorc [:gps]]

Attachment: MozReview Request: hgmo: handle binary data during json serialization (bug 1268128); r?smacleod

Review commit: https://reviewboard.mozilla.org/r/49297/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/49297/

Comment 6

[Steven MacLeod [:smacleod]]

Comment on attachment 8746194 [details]
MozReview Request: pushlog: convert unicode to str (bug 1268128); r?smacleod

https://reviewboard.mozilla.org/r/49295/#review46483

Comment 7

[Steven MacLeod [:smacleod]]

Comment on attachment 8746195 [details]
MozReview Request: hgmo: handle binary data during json serialization (bug 1268128); r?smacleod

https://reviewboard.mozilla.org/r/49297/#review46485

Comment 8

[Gregory Szorc [:gps]]

Deploying now.

Comment 9

[Gregory Szorc [:gps]]

Was able to get a HTTP 200 from the URL in comment #0. You may still see lingering HTTP 500s until the WSGI processes restart and pick up the new code.