[mozprocess] UnicodeEncodeError if os.environ has non-ascii characters

RESOLVED FIXED in Firefox 48


4 years ago
3 years ago


(Reporter: mikeh, Assigned: ahal)


Dependency tree / graph

Firefox Tracking Flags

(firefox48 fixed)



(3 attachments)

I'm seeing this today trying to locally run:

# ./mach mochitest-remote dom/camera/test/

This worked yesterday -- all that's changed is I updated my trees and rebuilt the emulator.

FWIW, u'\u279c'[1] is a character in my command prompt, but it always has been.

1. http://www.fileformat.info/info/unicode/char/279C/index.htm
The problem goes away if I change my PROMPT envvar:

export PROMPT=#

Comment 2

4 years ago
Please paste the traceback :).

This problem can usually be fixed by finding the offending string and explicitly encoding it:
my_str.encode('utf-8', 'replace')
I'll get you an update as soon as the dependent bug is fixed. :)
Depends on: 1154002
Created attachment 8592884 [details]
Unicode encode error backtrace

As requested.
Flags: needinfo?(ahalberstadt)
(I guess the question is: why is the script slurping my entire environment? This is likely to cause random problems for lots of people, no?)

Comment 6

4 years ago
Mozrunner is probably setting env=os.environ somewhere. This can be useful if people want to set environment that affects the subprocess (i.e gecko). Did this used to work for you? If so I imagine this is a regression from bug 1050561 (ugh, I hate taking unicode patches).
Flags: needinfo?(ahalberstadt)

Comment 7

4 years ago
Yep, confirmed that bug 1050561 caused the regression.
Depends on: 1050561


4 years ago
Component: Mochitest → Mozbase
Summary: UnicodeEncodeError: 'ascii' codec can't encode character u'\u279c' in position 20: ordinal not in range(128) → [mozprocess] UnicodeEncodeError if os.environ has non-ascii characters

Comment 8

4 years ago
Created attachment 8592929 [details] [diff] [review]
Fix UnicodeEncodeError

This patch fixes the problem, but it might re-expose the problem in bug 1050561. Frank, could you test this out to make sure it doesn't unfix bug 1050561 please?
Flags: needinfo?(bugzilla)
Comment on attachment 8592929 [details] [diff] [review]
Fix UnicodeEncodeError

With a bit of delay I got bad news: This seems to expose the other bug again.
Output of "mozmake mochitest-browser-chrome" after the skipped tests:
dir: addon-sdk/test
Traceback (most recent call last):
  File "_tests/testing/mochitest/runtests.py", line 2623, in <module>
  File "_tests/testing/mochitest/runtests.py", line 2617, in main
    return_code = mochitest.runTests(options)
  File "_tests/testing/mochitest/runtests.py", line 2120, in runTests
    result = self.runMochitests(options, tests_in_dir, onLaunch)
  File "_tests/testing/mochitest/runtests.py", line 2040, in runMochitests
    result = self.doTests(options, onLaunch, testsToRun)
  File "_tests/testing/mochitest/runtests.py", line 2175, in doTests
    self.manifest = self.buildProfile(options)
  File "_tests/testing/mochitest/runtests.py", line 1432, in buildProfile
    certificateStatus = self.fillCertificateDB(options)
  File "_tests/testing/mochitest/runtests.py", line 1301, in fillCertificateDB
    [certutil, "-N", "-d", certdbPath, "-f", pwfilePath], env=toolsEnv)
  File "_tests/testing/mochitest/runtests.py", line 273, in call
  File "f:\mozilla\tree-hg\comm-central\mozilla\testing\mozbase\mozprocess\mozprocess\processhandler.py", line 728,
in run
    self.proc = self.Process([self.cmd] + self.args, **args)
  File "f:\mozilla\tree-hg\comm-central\mozilla\testing\mozbase\mozprocess\mozprocess\processhandler.py", line 115,
in __init__
    universal_newlines, startupinfo, creationflags)
  File "c:\mozilla-build\python\Lib\subprocess.py", line 711, in __init__
    errread, errwrite)
  File "f:\mozilla\tree-hg\comm-central\mozilla\testing\mozbase\mozprocess\mozprocess\processhandler.py", line 268,
in _execute_child
  File "f:\mozilla\tree-hg\comm-central\mozilla\testing\mozbase\mozprocess\mozprocess\winprocess.py", line 145, in _
    self._as_parameter_ = LPCWSTR("\0".join(values))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 9: ordinal not in range(128)
f:/mozilla/tree-hg/comm-central/mozilla/../suite/build.mk:30: recipe for target 'mochitest-browser-chrome' failed
Flags: needinfo?(bugzilla)
Duplicate of this bug: 1152626

Comment 11

3 years ago
Sorry, this slipped through the cracks. Thanks for tweeting about it mhoye!

The root of the problem is that some things are expecting unicode strings, and others are expecting byte strings. I think the best way forward here is to move the fix from bug 1050561 out of processhandler.py and into winprocess.py. I'll get a new patch up.


3 years ago
Assignee: nobody → ahalberstadt

Comment 12

3 years ago
I believe this patch fixes both this and bug 1050561. Here's a try run:

Comment 13

3 years ago
Created attachment 8732153 [details]
MozReview Request: Bug 1152428 - [mozprocess] Fix UnicodeEncodeError when non-ascii characters are in the environment, r?wlach

Review commit: https://reviewboard.mozilla.org/r/41023/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/41023/
Attachment #8732153 - Flags: review?(wlachance)
Comment on attachment 8732153 [details]
MozReview Request: Bug 1152428 - [mozprocess] Fix UnicodeEncodeError when non-ascii characters are in the environment, r?wlach


This looks fine! Thanks.
Attachment #8732153 - Flags: review?(wlachance) → review+

Comment 16

3 years ago
Last Resolved: 3 years ago
status-firefox48: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla48
You need to log in before you can comment on or make changes to this bug.