Closed
Bug 1476484
Opened 6 years ago
Closed 6 years ago
Pushing to mozreview fails with "500 - Internal Server Error" when a reviewer's name contains an emoji
Categories
(MozReview Graveyard :: General, defect)
Tracking
(Not tracked)
RESOLVED
FIXED
People
(Reporter: mozbugz, Assigned: glob)
References
Details
Attachments
(1 file)
See bug 1459937, I pushed to it multiple times, and landed it, but it got backed-out. Now I cannot push further updates: > $ hg push review > pushing to https://reviewboard-hg.mozilla.org/autoreview > searching for appropriate review repository > redirecting push to https://reviewboard-hg.mozilla.org/gecko > searching for changes > remote: adding changesets > remote: adding manifests > remote: adding file changes > remote: added 33 changesets with 23 changes to 100 files (+1 heads) > remote: recorded push in pushlog > submitting 3 changesets for review > abort: <!DOCTYPE html> > > <html> > <head> > <title>500 - Internal Server Error | Review Board</title> > </head> > <body> > <h1>Something broke! (Error 500)</h1> > <p> > It appears something broke when you tried to go to here. This is either > a bug in Review Board or a server configuration error. Please report > this to your administrator. > </p> > </body> > </title> Trying again: > $ hg push -v review > pushing to https://reviewboard-hg.mozilla.org/autoreview > searching for appropriate review repository > redirecting push to https://reviewboard-hg.mozilla.org/gecko > searching for changes > all local heads known remotely > no changes found > submitting 3 changesets for review > abort: <!DOCTYPE html> > > <html> > <head> > <title>500 - Internal Server Error | Review Board</title> > [etc.] "all local heads known remotely" -- So it seems the patches have gone through. The patches I'm trying to push are attached to this Try: https://treeherder.mozilla.org/#/jobs?repo=try&revision=7bc98cd79431c2f73745202db179495bb12b55eb Some details, if important: - The first time I pushed the update after backout, it failed because the mozreview was still closed, I reopened it then. - The original patches (before backout) were developed and pushed from Mac, the new ones were updated and push from Windows. (I tried to redo the updates on the Mac, same issue.) - I've tried multiple times, about 16h ago, and again before writing this bug. - In the meantime, I've seen other activity happening on mozreview in other bugs, so I think it's just me! - I tried the trick in bug 1235307, but that didn't work either.
urgh - looks like reviewboard doesn't agree with emoji. 2018-07-18 05:12:17,713 - INFO - - Login attempt (apikey) for user gsquelart@mozilla.com: 2018-07-18 05:12:18,199 - INFO - - Login successful (apikey) for user gsquelart@mozilla.com: 2018-07-18 05:12:18,208 - INFO - - processing BatchReviewRequest for gerald 2018-07-18 05:12:18,214 - INFO - - processing batch submission bz://1459937/gerald to gecko with 3 commits 2018-07-18 05:12:18,220 - INFO - - using squashed review request 255868 2018-07-18 05:12:18,220 - INFO - - bz://1459937/gerald: generating squashed diffset for 255868 2018-07-18 05:12:18,386 - INFO - - bz://1459937/gerald: updated squashed diffset for 255868 2018-07-18 05:12:18,390 - INFO - - bz://1459937/gerald: 3 previous commits; 0 discard on publish; 0 unpublished 2018-07-18 05:12:18,391 - INFO - - bz://1459937/gerald: 0/3 commits mapped exactly 2018-07-18 05:12:18,391 - INFO - - bz://1459937/gerald: 0/3 mapped exactly or to precursors 2018-07-18 05:12:18,400 - INFO - - bz://1459937/gerald: commit ID 8GFO1ZWuZ1b for 02432ef731867be475e05de87639eb492fbec307 found in review request 255870 2018-07-18 05:12:19,619 - INFO - - importing Bugzilla users from query "dbaron": dbaron@dbaron.org 2018-07-18 05:12:19,623 - INFO - - updating first name of 172191:dbaron from David Baron :dbaron: ⌚UTC-7 to David Baron :dbaron: 🏴 ⌚UTC-7 2018-07-18 05:12:19,650 - ERROR - - Exception thrown for user gerald at https://reviewboard.mozilla.org/api/extensions/mozreview.extension.MozReviewExtension/batch-review-requests/ (1366, "Incorrect string value: '\\xF0\\x9F\\x8F\\xB4\\xF3\\xA0...' for column 'first_name' at row 1") Traceback (most recent call last): File "/data/www/reviewboard.mozilla.org/venv/lib/python2.6/site-packages/django/core/handlers/base.py", line 112, in get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/usr/lib64/python2.6/site-packages/newrelic-2.44.0.36/newrelic/hooks/framework_django.py", line 497, in wrapper return wrapped(*args, **kwargs) File "/data/www/reviewboard.mozilla.org/venv/lib/python2.6/site-packages/django/views/decorators/cache.py", line 52, in _wrapped_view_func response = view_func(request, *args, **kwargs) File "/data/www/reviewboard.mozilla.org/venv/lib/python2.6/site-packages/django/views/decorators/vary.py", line 19, in inner_func response = func(*args, **kwargs) File "/data/www/reviewboard.mozilla.org/venv/lib/python2.6/site-packages/djblets/webapi/resources/base.py", line 196, in __call__ request, method, view, api_format=api_format, *args, **kwargs) File "/data/www/reviewboard.mozilla.org/venv/lib/python2.6/site-packages/djblets/webapi/resources/mixins/api_tokens.py", line 65, in call_method_view return view(request, *args, **kwargs) File "/data/www/reviewboard.mozilla.org/venv/lib/python2.6/site-packages/djblets/webapi/resources/base.py", line 449, in post return self.create(*args, **kwargs) File "/data/www/reviewboard.mozilla.org/venv/lib/python2.6/site-packages/djblets/webapi/decorators.py", line 122, in _call return view_func(*args, **kwargs) File "/data/www/reviewboard.mozilla.org/venv/lib/python2.6/site-packages/ReviewBoard-2.5.4.moz-py2.6.egg/reviewboard/webapi/decorators.py", line 139, in _check return view_func(*args, **kwargs) File "/data/www/reviewboard.mozilla.org/venv/lib/python2.6/site-packages/djblets/webapi/decorators.py", line 122, in _call return view_func(*args, **kwargs) File "/data/www/reviewboard.mozilla.org/venv/lib/python2.6/site-packages/djblets/webapi/decorators.py", line 143, in _checklogin return view_func(*args, **kwargs) File "/data/www/reviewboard.mozilla.org/venv/lib/python2.6/site-packages/djblets/webapi/decorators.py", line 122, in _call return view_func(*args, **kwargs) File "/data/www/reviewboard.mozilla.org/venv/lib/python2.6/site-packages/djblets/webapi/decorators.py", line 122, in _call return view_func(*args, **kwargs) File "/data/www/reviewboard.mozilla.org/venv/lib/python2.6/site-packages/djblets/webapi/decorators.py", line 307, in _validate return view_func(*args, **new_kwargs) File "/data/www/reviewboard.mozilla.org/venv/lib/python2.6/site-packages/mozreview-0.1.2a0-py2.6.egg/mozreview/resources/batch_review_request.py", line 351, in create identifier, commits) File "/data/www/reviewboard.mozilla.org/venv/lib/python2.6/site-packages/mozreview-0.1.2a0-py2.6.egg/mozreview/resources/batch_review_request.py", line 627, in _process_submission create_comm_msg_filediff) File "/data/www/reviewboard.mozilla.org/venv/lib/python2.6/site-packages/mozreview-0.1.2a0-py2.6.egg/mozreview/resources/batch_review_request.py", line 982, in update_review_request resolve_reviewers(reviewer_cache, commit.get('requal_reviewers', [])) File "/data/www/reviewboard.mozilla.org/venv/lib/python2.6/site-packages/mozreview-0.1.2a0-py2.6.egg/mozreview/resources/batch_review_request.py", line 929, in resolve_reviewers user = cache.resolve_reviewer(reviewer) File "/data/www/reviewboard.mozilla.org/venv/lib/python2.6/site-packages/mozreview-0.1.2a0-py2.6.egg/mozreview/resources/batch_review_request.py", line 898, in resolve_reviewer self._d[reviewer] = get_user_from_reviewer(self._request, reviewer) File "/data/www/reviewboard.mozilla.org/venv/lib/python2.6/site-packages/mozreview-0.1.2a0-py2.6.egg/mozreview/resources/batch_review_request.py", line 905, in get_user_from_reviewer backend.query_users(reviewer, request) File "/data/www/reviewboard.mozilla.org/venv/lib/python2.6/site-packages/mozreview-0.1.2a0-py2.6.egg/mozreview/bugzilla/auth.py", line 256, in query_users get_or_create_bugzilla_users(users) File "/data/www/reviewboard.mozilla.org/venv/lib/python2.6/site-packages/mozreview-0.1.2a0-py2.6.egg/mozreview/bugzilla/models.py", line 162, in get_or_create_bugzilla_users user.save() File "/data/www/reviewboard.mozilla.org/venv/lib/python2.6/site-packages/django/db/models/base.py", line 545, in save force_update=force_update, update_fields=update_fields) File "/data/www/reviewboard.mozilla.org/venv/lib/python2.6/site-packages/django/db/models/base.py", line 573, in save_base updated = self._save_table(raw, cls, force_insert, force_update, using, update_fields) File "/data/www/reviewboard.mozilla.org/venv/lib/python2.6/site-packages/django/db/models/base.py", line 635, in _save_table forced_update) File "/data/www/reviewboard.mozilla.org/venv/lib/python2.6/site-packages/django/db/models/base.py", line 679, in _do_update return filtered._update(values) > 0 File "/data/www/reviewboard.mozilla.org/venv/lib/python2.6/site-packages/django/db/models/query.py", line 510, in _update return query.get_compiler(self.db).execute_sql(None) File "/data/www/reviewboard.mozilla.org/venv/lib/python2.6/site-packages/django/db/models/sql/compiler.py", line 980, in execute_sql cursor = super(SQLUpdateCompiler, self).execute_sql(result_type) File "/data/www/reviewboard.mozilla.org/venv/lib/python2.6/site-packages/django/db/models/sql/compiler.py", line 786, in execute_sql cursor.execute(sql, params) File "/data/www/reviewboard.mozilla.org/venv/lib/python2.6/site-packages/django/db/backends/util.py", line 53, in execute return self.cursor.execute(sql, params) File "/data/www/reviewboard.mozilla.org/venv/lib/python2.6/site-packages/django/db/utils.py", line 99, in __exit__ six.reraise(dj_exc_type, dj_exc_value, traceback) File "/data/www/reviewboard.mozilla.org/venv/lib/python2.6/site-packages/django/db/backends/util.py", line 53, in execute return self.cursor.execute(sql, params) File "/data/www/reviewboard.mozilla.org/venv/lib/python2.6/site-packages/django/db/backends/mysql/base.py", line 124, in execute return self.cursor.execute(query, args) File "/usr/lib64/python2.6/site-packages/newrelic-2.44.0.36/newrelic/hooks/database_dbapi2.py", line 22, in execute *args, **kwargs) File "/data/www/reviewboard.mozilla.org/venv/lib/python2.6/site-packages/MySQLdb/cursors.py", line 205, in execute self.errorhandler(self, exc, value) File "/data/www/reviewboard.mozilla.org/venv/lib/python2.6/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler raise errorclass, errorvalue OperationalError: (1366, "Incorrect string value: '\\xF0\\x9F\\x8F\\xB4\\xF3\\xA0...' for column 'first_name' at row 1")
Assignee: nobody → glob
Summary: Pushing update to reopened mozreview fails with "500 - Internal Server Error" → Pushing update to reopened mozreview fails with "500 - Internal Server Error" when a reviewer's name contains an emoji
simplest thing would be to strip emoji from usernames from https://stackoverflow.com/a/49146722/953: > emoji_re = re.compile( > "[" > u"\U0001F600-\U0001F64F" # emoticons > u"\U0001F300-\U0001F5FF" # symbols & pictographs > u"\U0001F680-\U0001F6FF" # transport & map symbols > u"\U0001F1E0-\U0001F1FF" # flags (iOS) > u"\U00002702-\U000027B0" > u"\U000024C2-\U0001F251" > "]+", > flags=re.UNICODE, > )
reviewboard itself handles emoji correctly - as an admin i can add emoji to a user's first_name field. however as getting the dev env updated to fully test this will be an epic amount of pain, i think stripping emoji will suffice.
Summary: Pushing update to reopened mozreview fails with "500 - Internal Server Error" when a reviewer's name contains an emoji → Pushing to mozreview fails with "500 - Internal Server Error" when a reviewer's name contains an emoji
|
||
Comment 5•6 years ago
|
||
Comment on attachment 8992847 [details] mozreview: strip emoji from bmo names (bug 1476484) r=smacleod Steven MacLeod [:smacleod] has approved the revision. https://phabricator.services.mozilla.com/D2216
Attachment #8992847 -
Flags: review+
Pushed by bjones@mozilla.com: https://hg.mozilla.org/hgcustom/version-control-tools/rev/8cda14d84914 mozreview: strip emoji from bmo names r=smacleod
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
didn't fix the problem:
> updating first name of 172191:dbaron from David Baron :dbaron: ⌚UTC-7 to David Baron :dbaron: ⌚UTC-7
(1366, "Incorrect string value: '\\xF3\\xA0\\x81\\xB5\\xF3\\xA0...' for column 'first_name' at row 1")Status: RESOLVED → REOPENED
Resolution: FIXED → ---
after more digging.. when i was testing emoji functionality before bmo switched i was using our development environment which is production like in most senses; however it's using sqlite instead of mysql. (and production is the only rb environment; no stage). unlike mysql, sqlite has a sane utf8 implementation. production rb's tables are utf8 not utf8bm4, however like bmo of old emoji in names result in silent truncation at that point. this "incorrect string value" is something different. i've tested in production using 🏴 and ⌚ and it works (with 🏴 being stripped as expected). (time passes) pulling dbaron's real name from the bmo database reveals something interesting - there's quite a few extra bytes there: 'David Baron :dbaron: \xf0\x9f\x8f\xb4\xf3\xa0\x81\xb5\xf3\xa0\x81\xb3\xf3\xa0\x81\xa3\xf3\xa0\x81\xa1\xf3\xa0\x81\xbf \xe2\x8c\x9aUTC-7' which is: 1F3F4 WAVING BLACK FLAG E0075 TAG LATIN SMALL LETTER U E0073 TAG LATIN SMALL LETTER S E0063 TAG LATIN SMALL LETTER C E0061 TAG LATIN SMALL LETTER A E007F CANCEL TAG 231A WATCH these aren't explicitly stripped by my regex, and evidently mysql outright chokes on them instead of silently truncating. dbaron - i've edited your name in bmo to drop the TAG bytes.
Status: REOPENED → RESOLVED
Closed: 6 years ago → 6 years ago
Resolution: --- → FIXED
note this should have been https://emojipedia.org/flag-for-california-usca/
|
||
Comment 10•6 years ago
|
||
Pushed by bjones@mozilla.com: https://hg.mozilla.org/hgcustom/version-control-tools/rev/ea61d7371e8d mozreview: strip TAGs from bmo names r=me
|
Assignee | |
Comment 11•6 years ago
|
||
pushed and deployed an update to the re to also strip TAGs restored dbaron's US-CA flag to his real name (even though it's just a black flag for me).
|
||
Comment 12•6 years ago
|
||
It seems likely to me that -any- higher-plane characters, not just emoji and tag letters, will result in similar failure. So if I changed my bugzilla name to "𝓘𝓸𝓷𝓪𝓽𝓱𝓪𝓷 𝗃𝖿𝗄𝗍𝗁𝖺𝗆𝖾 𝓚𝓮𝔀" (i.e. using Mathematical symbols for stylistic effect), I'm guessing the same problem would arise. Maybe the RE should simply strip all supplementary-plane characters u"\U00010000-\U0010FFFF"?
Yeah, I'm still looking for a font that supports US state flags. That doesn't mean I won't use them, though. Someday "David Baron :dbaron: 🏴 ⌚UTC-7" will show a California flag in it... In any case, needinfo for comment 12, because Jonathan's theory seems pretty likely...
Flags: needinfo?(glob)
|
|
Assignee | |
Comment 14•6 years ago
|
||
(In reply to Jonathan Kew (:jfkthame) from comment #12) > It seems likely to me that -any- higher-plane characters, not just emoji and > tag letters, will result in similar failure. So if I changed my bugzilla > name to "𝓘𝓸𝓷𝓪𝓽𝓱𝓪𝓷 𝗃𝖿𝗄𝗍𝗁𝖺𝗆𝖾 𝓚𝓮𝔀" (i.e. using Mathematical > symbols for stylistic effect), I'm guessing the same problem would arise. the regex we're currently using will strip math symbols so it won't fail, but i see your point. for high plane chars what i expect to happen is for the name to be truncated at that char rather than throw an internal server error (reviewboard also exhibits the same mysql issue that plagued bmo). i'm ok with that within the context of reviewboard. a large influencing factor here is we have a limited capacity to experiment here; all testing needs to happen on production and the deployment process involves a long sequence of manual steps (it used to be automated, but reviewboard uses an ancient version of python and django). if we hit an ISE when updating a reviewer's name i'll revisit, but for now i'd prefer to leave things as-is. note it is possible to test this code without touching reviews: update your name in bmo then log out and back into reviewboard. once logged in click on your avatar at the top right to view your probably mangled name.
Flags: needinfo?(glob)
You need to log in
before you can comment on or make changes to this bug.
Description
•