Closed Bug 1152428 Opened 9 years ago Closed 8 years ago

[mozprocess] UnicodeEncodeError if os.environ has non-ascii characters

Categories

(Testing :: Mozbase, defect)

x86_64
Linux
defect
Not set
normal

Tracking

(firefox48 fixed)

RESOLVED FIXED
mozilla48
Tracking Status
firefox48 --- fixed

People

(Reporter: mikeh, Assigned: ahal)

References

Details

Attachments

(3 files)

I'm seeing this today trying to locally run:

# ./mach mochitest-remote dom/camera/test/

This worked yesterday -- all that's changed is I updated my trees and rebuilt the emulator.

FWIW, u'\u279c'[1] is a character in my command prompt, but it always has been.

1. http://www.fileformat.info/info/unicode/char/279C/index.htm
The problem goes away if I change my PROMPT envvar:

export PROMPT=#
See Also: → 1154002
Please paste the traceback :).

This problem can usually be fixed by finding the offending string and explicitly encoding it:
my_str.encode('utf-8', 'replace')
I'll get you an update as soon as the dependent bug is fixed. :)
Depends on: 1154002
As requested.
Flags: needinfo?(ahalberstadt)
(I guess the question is: why is the script slurping my entire environment? This is likely to cause random problems for lots of people, no?)
Mozrunner is probably setting env=os.environ somewhere. This can be useful if people want to set environment that affects the subprocess (i.e gecko). Did this used to work for you? If so I imagine this is a regression from bug 1050561 (ugh, I hate taking unicode patches).
Flags: needinfo?(ahalberstadt)
Yep, confirmed that bug 1050561 caused the regression.
Depends on: 1050561
Component: Mochitest → Mozbase
Summary: UnicodeEncodeError: 'ascii' codec can't encode character u'\u279c' in position 20: ordinal not in range(128) → [mozprocess] UnicodeEncodeError if os.environ has non-ascii characters
This patch fixes the problem, but it might re-expose the problem in bug 1050561. Frank, could you test this out to make sure it doesn't unfix bug 1050561 please?
Flags: needinfo?(bugzilla)
Comment on attachment 8592929 [details] [diff] [review]
Fix UnicodeEncodeError

With a bit of delay I got bad news: This seems to expose the other bug again.
Output of "mozmake mochitest-browser-chrome" after the skipped tests:
dir: addon-sdk/test
Traceback (most recent call last):
  File "_tests/testing/mochitest/runtests.py", line 2623, in <module>
    main()
  File "_tests/testing/mochitest/runtests.py", line 2617, in main
    return_code = mochitest.runTests(options)
  File "_tests/testing/mochitest/runtests.py", line 2120, in runTests
    result = self.runMochitests(options, tests_in_dir, onLaunch)
  File "_tests/testing/mochitest/runtests.py", line 2040, in runMochitests
    result = self.doTests(options, onLaunch, testsToRun)
  File "_tests/testing/mochitest/runtests.py", line 2175, in doTests
    self.manifest = self.buildProfile(options)
  File "_tests/testing/mochitest/runtests.py", line 1432, in buildProfile
    certificateStatus = self.fillCertificateDB(options)
  File "_tests/testing/mochitest/runtests.py", line 1301, in fillCertificateDB
    [certutil, "-N", "-d", certdbPath, "-f", pwfilePath], env=toolsEnv)
  File "_tests/testing/mochitest/runtests.py", line 273, in call
    process.run()
  File "f:\mozilla\tree-hg\comm-central\mozilla\testing\mozbase\mozprocess\mozprocess\processhandler.py", line 728,
in run
    self.proc = self.Process([self.cmd] + self.args, **args)
  File "f:\mozilla\tree-hg\comm-central\mozilla\testing\mozbase\mozprocess\mozprocess\processhandler.py", line 115,
in __init__
    universal_newlines, startupinfo, creationflags)
  File "c:\mozilla-build\python\Lib\subprocess.py", line 711, in __init__
    errread, errwrite)
  File "f:\mozilla\tree-hg\comm-central\mozilla\testing\mozbase\mozprocess\mozprocess\processhandler.py", line 268,
in _execute_child
    winprocess.EnvironmentBlock(env),
  File "f:\mozilla\tree-hg\comm-central\mozilla\testing\mozbase\mozprocess\mozprocess\winprocess.py", line 145, in _
_init__
    self._as_parameter_ = LPCWSTR("\0".join(values))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 9: ordinal not in range(128)
f:/mozilla/tree-hg/comm-central/mozilla/../suite/build.mk:30: recipe for target 'mochitest-browser-chrome' failed
Flags: needinfo?(bugzilla)
Sorry, this slipped through the cracks. Thanks for tweeting about it mhoye!

The root of the problem is that some things are expecting unicode strings, and others are expecting byte strings. I think the best way forward here is to move the fix from bug 1050561 out of processhandler.py and into winprocess.py. I'll get a new patch up.
Assignee: nobody → ahalberstadt
Status: NEW → ASSIGNED
I believe this patch fixes both this and bug 1050561. Here's a try run:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=c1f7162ea6af
Comment on attachment 8732153 [details]
MozReview Request: Bug 1152428 - [mozprocess] Fix UnicodeEncodeError when non-ascii characters are in the environment, r?wlach

https://reviewboard.mozilla.org/r/41023/#review37637

This looks fine! Thanks.
Attachment #8732153 - Flags: review?(wlachance) → review+
https://hg.mozilla.org/mozilla-central/rev/f1482a4600fc
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla48
You need to log in before you can comment on or make changes to this bug.