1091759 - 'invalid continuation byte' errors parsing jsreftest logs

Status: NEW

(Tree Management :: Treeherder, defect, P3)

Description

[William Lachance (:wlach)]

On my dev version of a treeherder instance, I'm seeing the following errors in `grep -i error celery_worker_gevent.log` that seem to be related to log parsing:

[2014-10-17 08:08:09,981: INFO/MainProcess] Task parse-log[5d88b39b-5ea7-4497-980e-23f96a31a643] retry: Retry in 600s: UnicodeDecodeError('utf8', '06:24:10     INFO -  REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/jsreftest/tests/jsreftest.html?test=js1_2/regexp/control_characters.js | string1 = \xe0O\xd0 \xea:i\x10\xa2\xd8 string1.match(string1) wrong value  item 2', 156, 157, 'invalid continuation byte')
[2014-10-17 08:18:31,831: ERROR/MainProcess] Task parse-log[5d88b39b-5ea7-4497-980e-23f96a31a643] raised unexpected: UnicodeDecodeError('utf8', '06:24:10     INFO -  REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/jsreftest/tests/jsreftest.html?test=js1_2/regexp/control_characters.js | string1 = \xe0O\xd0 \xea:i\x10\xa2\xd8 string1.match(string1) wrong value  item 2', 156, 157, 'invalid continuation byte')

Comment 1

[Ed Morley [:emorley]]

Presume this is the same as:
https://rpm.newrelic.com/accounts/677903/applications/4180461/traced_errors/2954943842

Traceback (most recent call last):
File "/usr/lib64/python2.6/site-packages/newrelic-2.40.0.34/newrelic/hooks/application_celery.py", line 66, in wrapper return wrapped(*args, **kwargs)
File "/usr/lib/python2.6/site-packages/celery-3.1.16-py2.6.egg/celery/app/trace.py", line 437, in __protected_call__ return self.run(*args, **kwargs)
File "/data/www/treeherder.mozilla.org/treeherder-service/treeherder/log_parser/tasks.py", line 95, in parse_log parse_log.retry(exc=e, countdown=10*60)
File "/usr/lib/python2.6/site-packages/celery-3.1.16-py2.6.egg/celery/app/task.py", line 660, in retry maybe_reraise()
File "/usr/lib/python2.6/site-packages/celery-3.1.16-py2.6.egg/celery/utils/__init__.py", line 242, in maybe_reraise reraise(exc_info[0], exc_info[1], exc_info[2])
File "/data/www/treeherder.mozilla.org/treeherder-service/treeherder/log_parser/tasks.py", line 47, in parse_log job_guid, check_errors)
File "/data/www/treeherder.mozilla.org/treeherder-service/treeherder/log_parser/utils.py", line 156, in extract_log_artifacts json.dumps(artifact)))
File "/usr/lib64/python2.6/site-packages/simplejson/__init__.py", line 354, in dumps return _default_encoder.encode(obj)
File "/usr/lib64/python2.6/site-packages/simplejson/encoder.py", line 262, in encode chunks = self.iterencode(o, _one_shot=True)
File "/usr/lib64/python2.6/site-packages/simplejson/encoder.py", line 340, in iterencode return _iterencode(o, 0)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe0 in position 156: invalid continuation byte

Comment 2

[Ed Morley [:emorley]]

Attachment: cedar_ubuntu64_hw_test-androidx86-set-1-bm103-tests1-linux-build38.txt.gz

Is still occurring; attaching example log.

[2015-05-12 10:46:43,674: ERROR/Worker-33] Failed to download/parse log for cedar 9a893345a93a0f2c306226e1f289dd7b1e997286 (http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/cedar-android-x86/1431441887/cedar_ubuntu64_hw_test-androidx86-set-1-bm103-tests1-linux-build38.txt.gz): 'utf8' codec can't decode byte 0xe0 in position 156: invalid continuation byte

[2015-05-12 10:46:43,680: ERROR/MainProcess] Task parse-log[6cbf0976-8215-4405-b1e1-30ac41886463] raised unexpected: UnicodeDecodeError('utf8', '09:45:42     INFO -  REFTEST TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/jsreftest/tests/jsreftest.html?test=js1_2/regexp/control_characters.js | string1 = \xe0O\xd0 \xea:i\x10\xa2\xd8 string1.match(string1) wrong value  item 2', 156, 157, 'invalid continuation byte')
Traceback (most recent call last):
  File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/newrelic-2.50.0.39/newrelic/hooks/application_celery.py", line 66, in wrapper
    return wrapped(*args, **kwargs)
  File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/celery/app/trace.py", line 437, in __protected_call__
    return self.run(*args, **kwargs)
  File "/data/www/treeherder.allizom.org/treeherder-service/treeherder/log_parser/tasks.py", line 34, in parse_log
    check_errors
  File "/data/www/treeherder.allizom.org/treeherder-service/treeherder/log_parser/utils.py", line 281, in post_log_artifacts
    _retry(e)
  File "/data/www/treeherder.allizom.org/treeherder-service/treeherder/log_parser/utils.py", line 256, in _retry
    retry_task.retry(exc=e, countdown=(1 + retry_task.request.retries) * 60)
  File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/celery/app/task.py", line 660, in retry
    maybe_reraise()
  File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/celery/utils/__init__.py", line 242, in maybe_reraise
    reraise(exc_info[0], exc_info[1], exc_info[2])
  File "/data/www/treeherder.allizom.org/treeherder-service/treeherder/log_parser/utils.py", line 274, in post_log_artifacts
    job_guid, check_errors)
  File "/data/www/treeherder.allizom.org/treeherder-service/treeherder/log_parser/utils.py", line 172, in extract_text_log_artifacts
    json.dumps(artifact)))
  File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/simplejson/__init__.py", line 370, in dumps
    return _default_encoder.encode(obj)
  File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/simplejson/encoder.py", line 269, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/data/www/treeherder.allizom.org/venv/lib/python2.7/site-packages/simplejson/encoder.py", line 348, in iterencode
    return _iterencode(o, 0)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xe0 in position 156: invalid continuation byte