Closed Bug 1483850 Opened Last year Closed Last year

[mozprocess] Intermittent UnicodeEncodeError: 'ascii' codec can't encode characters in position 81-83: ordinal not in range(128)

Categories

(Testing :: Mozbase, defect, P5)

Version 3
defect

Tracking

(firefox-esr52 unaffected, firefox-esr60 unaffected, firefox61 unaffected, firefox62 unaffected, firefox63 fixed)

RESOLVED FIXED
mozilla63
Tracking Status
firefox-esr52 --- unaffected
firefox-esr60 --- unaffected
firefox61 --- unaffected
firefox62 --- unaffected
firefox63 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: whimboo)

References

Details

(Keywords: intermittent-failure, regression)

Filed by: hskupin [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=194199539&repo=mozilla-central

https://queue.taskcluster.net/v1/task/VmVmQz5NRFaPJUR-uD7J5g/runs/0/artifacts/public/logs/live_backing.log

There are a couple of those errors. Here one example:

23:37:39     INFO -  Traceback (most recent call last):
23:37:39     INFO -    File "c:\mozilla-build\python\Lib\threading.py", line 801, in __bootstrap_inner
23:37:39     INFO -      self.run()
23:37:39     INFO -    File "c:\mozilla-build\python\Lib\threading.py", line 754, in run
23:37:39     INFO -      self.__target(*self.__args, **self.__kwargs)
23:37:39     INFO -    File "Z:\task_1534373372\build\venv\lib\site-packages\mozprocess\processhandler.py", line 1031, in _read
23:37:39     INFO -      callback(line.rstrip())
23:37:39     INFO -    File "Z:\task_1534373372\build\venv\lib\site-packages\mozprocess\processhandler.py", line 947, in __call__
23:37:39     INFO -      e(*args, **kwargs)
23:37:39     INFO -    File "Z:\task_1534373372\build\venv\lib\site-packages\mozprocess\processhandler.py", line 1079, in __call__
23:37:39     INFO -      self.stream.write(line.decode('iso8859-1') + '\n')
23:37:39     INFO -  UnicodeEncodeError: 'ascii' codec can't encode characters in position 81-83: ordinal not in range(128)
We should not release the version 1.0.0 before those failures have been fixed.
Flags: needinfo?(dave.hunt)
Pavel: Could you look into writing a failing test for this and see if you can fix it for both Python 2.7 and 3.5? Let me know if you have any questions. We should treat this as a priority so we can release mozprocess v1.0.0.
Flags: needinfo?(dave.hunt) → needinfo?(slepushkin)
1) I think, we are hitting this - https://bugzilla.mozilla.org/show_bug.cgi?id=991866 (RESOLVED INCOMPLETE)
2) I've tried to write a test using cyrillic and simplified Chinese in UTF-8 (in test_process_reader.py) - test worked, error has not reproduced (but I am not on Windows)

3) My first idea for fix would be
replace 
>        try:
>            self.stream.write(line.decode() + '\n')
>        except UnicodeDecodeError:
>            # TODO: Workaround for bug #991866 to make sure we can display when
>            # when normal UTF-8 display is failing
>            self.stream.write(line.decode('iso8859-1') + '\n')

with

>       self.stream.write(line.decode(errors='replace') + '\n')

but people in 991866 rejected it and without test - I don't know why.

Don't think this is very helpful ,sorry.
Flags: needinfo?(slepushkin)
Note that Windows is very special when it comes to Unicode. So to reproduce such a problem you should have a Windows machine or VM.
The actual work should now happen on bug 1428713. Marking this regression bug as fixed.
Status: NEW → RESOLVED
Closed: Last year
Keywords: regression
Resolution: --- → FIXED
Target Milestone: --- → mozilla63
Assignee: nobody → hskupin
You need to log in before you can comment on or make changes to this bug.