Closed Bug 1152428 Opened 5 years ago Closed 4 years ago

[mozprocess] UnicodeEncodeError if os.environ has non-ascii characters

Categories

(Testing :: Mozbase, defect)

x86_64
Linux
defect
Not set

Tracking

(firefox48 fixed)

RESOLVED FIXED
mozilla48
Tracking Status
firefox48 --- fixed

People

(Reporter: mikeh, Assigned: ahal)

References

Details

Attachments

(3 files)

I'm seeing this today trying to locally run:

# ./mach mochitest-remote dom/camera/test/

This worked yesterday -- all that's changed is I updated my trees and rebuilt the emulator.

FWIW, u'\u279c'[1] is a character in my command prompt, but it always has been.

1. http://www.fileformat.info/info/unicode/char/279C/index.htm
The problem goes away if I change my PROMPT envvar:

export PROMPT=#
See Also: → 1154002
Please paste the traceback :).

This problem can usually be fixed by finding the offending string and explicitly encoding it:
my_str.encode('utf-8', 'replace')
I'll get you an update as soon as the dependent bug is fixed. :)
Depends on: 1154002
As requested.
Flags: needinfo?(ahalberstadt)
(I guess the question is: why is the script slurping my entire environment? This is likely to cause random problems for lots of people, no?)
Mozrunner is probably setting env=os.environ somewhere. This can be useful if people want to set environment that affects the subprocess (i.e gecko). Did this used to work for you? If so I imagine this is a regression from bug 1050561 (ugh, I hate taking unicode patches).
Flags: needinfo?(ahalberstadt)
Yep, confirmed that bug 1050561 caused the regression.
Depends on: 1050561
Component: Mochitest → Mozbase
Summary: UnicodeEncodeError: 'ascii' codec can't encode character u'\u279c' in position 20: ordinal not in range(128) → [mozprocess] UnicodeEncodeError if os.environ has non-ascii characters
This patch fixes the problem, but it might re-expose the problem in bug 1050561. Frank, could you test this out to make sure it doesn't unfix bug 1050561 please?
Flags: needinfo?(bugzilla)
Comment on attachment 8592929 [details] [diff] [review]
Fix UnicodeEncodeError

With a bit of delay I got bad news: This seems to expose the other bug again.
Output of "mozmake mochitest-browser-chrome" after the skipped tests:
dir: addon-sdk/test
Traceback (most recent call last):
  File "_tests/testing/mochitest/runtests.py", line 2623, in <module>
    main()
  File "_tests/testing/mochitest/runtests.py", line 2617, in main
    return_code = mochitest.runTests(options)
  File "_tests/testing/mochitest/runtests.py", line 2120, in runTests
    result = self.runMochitests(options, tests_in_dir, onLaunch)
  File "_tests/testing/mochitest/runtests.py", line 2040, in runMochitests
    result = self.doTests(options, onLaunch, testsToRun)
  File "_tests/testing/mochitest/runtests.py", line 2175, in doTests
    self.manifest = self.buildProfile(options)
  File "_tests/testing/mochitest/runtests.py", line 1432, in buildProfile
    certificateStatus = self.fillCertificateDB(options)
  File "_tests/testing/mochitest/runtests.py", line 1301, in fillCertificateDB
    [certutil, "-N", "-d", certdbPath, "-f", pwfilePath], env=toolsEnv)
  File "_tests/testing/mochitest/runtests.py", line 273, in call
    process.run()
  File "f:\mozilla\tree-hg\comm-central\mozilla\testing\mozbase\mozprocess\mozprocess\processhandler.py", line 728,
in run
    self.proc = self.Process([self.cmd] + self.args, **args)
  File "f:\mozilla\tree-hg\comm-central\mozilla\testing\mozbase\mozprocess\mozprocess\processhandler.py", line 115,
in __init__
    universal_newlines, startupinfo, creationflags)
  File "c:\mozilla-build\python\Lib\subprocess.py", line 711, in __init__
    errread, errwrite)
  File "f:\mozilla\tree-hg\comm-central\mozilla\testing\mozbase\mozprocess\mozprocess\processhandler.py", line 268,
in _execute_child
    winprocess.EnvironmentBlock(env),
  File "f:\mozilla\tree-hg\comm-central\mozilla\testing\mozbase\mozprocess\mozprocess\winprocess.py", line 145, in _
_init__
    self._as_parameter_ = LPCWSTR("\0".join(values))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 9: ordinal not in range(128)
f:/mozilla/tree-hg/comm-central/mozilla/../suite/build.mk:30: recipe for target 'mochitest-browser-chrome' failed
Flags: needinfo?(bugzilla)
Duplicate of this bug: 1152626
Sorry, this slipped through the cracks. Thanks for tweeting about it mhoye!

The root of the problem is that some things are expecting unicode strings, and others are expecting byte strings. I think the best way forward here is to move the fix from bug 1050561 out of processhandler.py and into winprocess.py. I'll get a new patch up.
Assignee: nobody → ahalberstadt
Status: NEW → ASSIGNED
I believe this patch fixes both this and bug 1050561. Here's a try run:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=c1f7162ea6af
Comment on attachment 8732153 [details]
MozReview Request: Bug 1152428 - [mozprocess] Fix UnicodeEncodeError when non-ascii characters are in the environment, r?wlach

https://reviewboard.mozilla.org/r/41023/#review37637

This looks fine! Thanks.
Attachment #8732153 - Flags: review?(wlachance) → review+
https://hg.mozilla.org/mozilla-central/rev/f1482a4600fc
Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla48
You need to log in before you can comment on or make changes to this bug.