654232 - [traceback] UnicodeDecodeError on /developers/addon/adblock-plus/file/74835/validation

Status: VERIFIED FIXED

(addons.mozilla.org Graveyard :: Developer Pages, defect)

Description

[krupa raj[:krupa]]

steps to reproduce:
1. Load https://addons.allizom.org/en-US/developers/addon/adblock-plus/file/74835/validation

traceback details:
"Traceback (most recent call last): 

File "/data/amo_python/www/preview/zamboni/apps/devhub/views.py", line 547, in json_file_validation v_result = tasks.file_validator(file.id) 

File "/data/amo_python/www/preview/zamboni/vendor/lib/python/celery/task/base.py", line 222, in __call__ return self.run(*args, **kwargs) 

File "/data/amo_python/www/preview/zamboni/vendor/lib/python/celery/decorators.py", line 52, in run return fun(*args, **kwargs) 

File "/data/amo_python/www/preview/zamboni/vendor/src/nuggets/celeryutils.py", line 33, in wrapped return fun(*args, **kw) 

File "/data/amo_python/www/preview/zamboni/apps/amo/decorators.py", line 82, in wrapper return f(*args, **kw) 

File "/data/amo_python/www/preview/zamboni/apps/amo/decorators.py", line 74, in wrapper return f(*args, **kw) 

File "/data/amo_python/www/preview/zamboni/apps/devhub/tasks.py", line 47, in file_validator result = _validator(file.file_path) 

File "/data/amo_python/www/preview/zamboni/apps/devhub/tasks.py", line 70, in _validator spidermonkey=settings.SPIDERMONKEY) 

File "/data/amo_python/www/preview/zamboni/vendor/src/amo-validator/validator/validate.py", line 39, in validate for_appversions=for_appversions) 

File "/data/amo_python/www/preview/zamboni/vendor/src/amo-validator/validator/submain.py", line 54, in phttp://pastebin.mozilla.org/repare_package for_appversions) 

File "/data/amo_python/www/preview/zamboni/vendor/src/amo-validator/validator/submain.py", line 122, in test_package return test_inner_package(err, package_contents, package, for_appversions) 

File "/data/amo_python/www/preview/zamboni/vendor/src/amo-validator/validator/submain.py", line 239, in test_inner_package test_func(err, package_contents, xpi_package) 

File "/data/amo_python/www/preview/zamboni/vendor/src/amo-validator/validator/testcases/content.py", line 130, in test_packed_packages sub_xpi) 

File "/data/amo_python/www/preview/zamboni/vendor/src/amo-validator/validator/submain.py", line 239, in test_inner_package test_func(err, package_contents, xpi_package) 

File "/data/amo_python/www/preview/zamboni/vendor/src/amo-validator/validator/testcases/content.py", line 160, in test_packed_packages data["extension"]) 

File "/data/amo_python/www/preview/zamboni/vendor/src/amo-validator/validator/testcases/markup/markuptester.py", line 101, in process self._feed_parser(line) 

File "/data/amo_python/www/preview/zamboni/vendor/src/amo-validator/validator/testcases/markup/markuptester.py", line 107, in _feed_parser self.feed(line + "\n") 

File "/usr/lib/python2.6/HTMLParser.py", line 108, in feed self.goahead(0) 

File "/usr/lib/python2.6/HTMLParser.py", line 148, in goahead k = self.parse_starttag(i) 

File "/usr/lib/python2.6/HTMLParser.py", line 249, in parse_starttag attrvalue = self.unescape(attrvalue) 

File "/usr/lib/python2.6/HTMLParser.py", line 387, in unescape return re.sub(r"&(#?[xX]?(?:[0-9a-fA-F]+|\w{1,8}));", replaceEntities, s) 

File "/usr/lib/python2.6/re.py", line 151, in sub return _compile(pattern, 0).sub(repl, string, count) 

UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128)

Comment 1

[Matt Basta [:basta]]

It would figure that re is not unicode-safe. I'll tweak this sometime this week.

Comment 2

[Matt Basta [:basta]]

I have a fix, but can't test it because I don't have access to the URL above. Can you attach the XPI?

Comment 3

[krupa raj[:krupa]]

Attachment: add-on

Comment 4

[Matt Basta [:basta]]

This should be fixed:

https://github.com/mozilla/amo-validator/pull/29

Kumar: Could you add this to the regression tests as well?

Comment 6

[Wil Clouser [:clouserw]]

Can we get this closed today?

Comment 7

[Kumar McMillan [:kumar]]

In the pull request above I requested some code changes before this gets merged in.

As for the regression test, I have pushed this to a branch (it fails) and will merge when it passes https://github.com/kumar303/test-amo-validator/compare/master...non-ascii-html

Comment 8

[Matt Basta [:basta]]

Merged:

https://github.com/mozilla/amo-validator/commit/9b33a0b80a75c0d9fe7250d1be868bf5164c57c3

Comment 9

[Kumar McMillan [:kumar]]

Functional test passes https://github.com/mozilla/test-amo-validator/commit/d59df53902779a39b39a596911a912802d7490d6

Comment 10

[Kumar McMillan [:kumar]]

Pushed to zamboni in https://github.com/jbalogh/zamboni-lib/commit/5bf90a2

Comment 11

[krupa raj[:krupa]]

verified fixed @ 
https://addons-next.allizom.org/en-US/developers/addon/adblock-plus/file/74835/validation
and (from the duped bug)
https://addons-next.allizom.org/en-US/developers/upload/2600a4f4368d4a35bc355e98aa3957c3