[mozprocess] UnicodeEncodeError if os.environ has non-ascii characters

RESOLVED FIXED in Firefox 48

Status

Testing
Mozbase
RESOLVED FIXED
3 years ago
2 years ago

People

(Reporter: mikeh, Assigned: ahal)

Tracking

unspecified
mozilla48
x86_64
Linux
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox48 fixed)

Details

MozReview Requests

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(3 attachments)

I'm seeing this today trying to locally run:

# ./mach mochitest-remote dom/camera/test/

This worked yesterday -- all that's changed is I updated my trees and rebuilt the emulator.

FWIW, u'\u279c'[1] is a character in my command prompt, but it always has been.

1. http://www.fileformat.info/info/unicode/char/279C/index.htm
The problem goes away if I change my PROMPT envvar:

export PROMPT=#
See Also: → bug 1154002
Please paste the traceback :).

This problem can usually be fixed by finding the offending string and explicitly encoding it:
my_str.encode('utf-8', 'replace')
I'll get you an update as soon as the dependent bug is fixed. :)
Depends on: 1154002
Created attachment 8592884 [details]
Unicode encode error backtrace

As requested.
Flags: needinfo?(ahalberstadt)
(I guess the question is: why is the script slurping my entire environment? This is likely to cause random problems for lots of people, no?)
Mozrunner is probably setting env=os.environ somewhere. This can be useful if people want to set environment that affects the subprocess (i.e gecko). Did this used to work for you? If so I imagine this is a regression from bug 1050561 (ugh, I hate taking unicode patches).
Flags: needinfo?(ahalberstadt)
Yep, confirmed that bug 1050561 caused the regression.
Depends on: 1050561
Component: Mochitest → Mozbase
Summary: UnicodeEncodeError: 'ascii' codec can't encode character u'\u279c' in position 20: ordinal not in range(128) → [mozprocess] UnicodeEncodeError if os.environ has non-ascii characters
Created attachment 8592929 [details] [diff] [review]
Fix UnicodeEncodeError

This patch fixes the problem, but it might re-expose the problem in bug 1050561. Frank, could you test this out to make sure it doesn't unfix bug 1050561 please?
Flags: needinfo?(bugzilla)
Comment on attachment 8592929 [details] [diff] [review]
Fix UnicodeEncodeError

With a bit of delay I got bad news: This seems to expose the other bug again.
Output of "mozmake mochitest-browser-chrome" after the skipped tests:
dir: addon-sdk/test
Traceback (most recent call last):
  File "_tests/testing/mochitest/runtests.py", line 2623, in <module>
    main()
  File "_tests/testing/mochitest/runtests.py", line 2617, in main
    return_code = mochitest.runTests(options)
  File "_tests/testing/mochitest/runtests.py", line 2120, in runTests
    result = self.runMochitests(options, tests_in_dir, onLaunch)
  File "_tests/testing/mochitest/runtests.py", line 2040, in runMochitests
    result = self.doTests(options, onLaunch, testsToRun)
  File "_tests/testing/mochitest/runtests.py", line 2175, in doTests
    self.manifest = self.buildProfile(options)
  File "_tests/testing/mochitest/runtests.py", line 1432, in buildProfile
    certificateStatus = self.fillCertificateDB(options)
  File "_tests/testing/mochitest/runtests.py", line 1301, in fillCertificateDB
    [certutil, "-N", "-d", certdbPath, "-f", pwfilePath], env=toolsEnv)
  File "_tests/testing/mochitest/runtests.py", line 273, in call
    process.run()
  File "f:\mozilla\tree-hg\comm-central\mozilla\testing\mozbase\mozprocess\mozprocess\processhandler.py", line 728,
in run
    self.proc = self.Process([self.cmd] + self.args, **args)
  File "f:\mozilla\tree-hg\comm-central\mozilla\testing\mozbase\mozprocess\mozprocess\processhandler.py", line 115,
in __init__
    universal_newlines, startupinfo, creationflags)
  File "c:\mozilla-build\python\Lib\subprocess.py", line 711, in __init__
    errread, errwrite)
  File "f:\mozilla\tree-hg\comm-central\mozilla\testing\mozbase\mozprocess\mozprocess\processhandler.py", line 268,
in _execute_child
    winprocess.EnvironmentBlock(env),
  File "f:\mozilla\tree-hg\comm-central\mozilla\testing\mozbase\mozprocess\mozprocess\winprocess.py", line 145, in _
_init__
    self._as_parameter_ = LPCWSTR("\0".join(values))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc2 in position 9: ordinal not in range(128)
f:/mozilla/tree-hg/comm-central/mozilla/../suite/build.mk:30: recipe for target 'mochitest-browser-chrome' failed
Flags: needinfo?(bugzilla)
Duplicate of this bug: 1152626
Sorry, this slipped through the cracks. Thanks for tweeting about it mhoye!

The root of the problem is that some things are expecting unicode strings, and others are expecting byte strings. I think the best way forward here is to move the fix from bug 1050561 out of processhandler.py and into winprocess.py. I'll get a new patch up.
Assignee: nobody → ahalberstadt
Status: NEW → ASSIGNED
I believe this patch fixes both this and bug 1050561. Here's a try run:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=c1f7162ea6af
Created attachment 8732153 [details]
MozReview Request: Bug 1152428 - [mozprocess] Fix UnicodeEncodeError when non-ascii characters are in the environment, r?wlach

Review commit: https://reviewboard.mozilla.org/r/41023/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/41023/
Attachment #8732153 - Flags: review?(wlachance)
Comment on attachment 8732153 [details]
MozReview Request: Bug 1152428 - [mozprocess] Fix UnicodeEncodeError when non-ascii characters are in the environment, r?wlach

https://reviewboard.mozilla.org/r/41023/#review37637

This looks fine! Thanks.
Attachment #8732153 - Flags: review?(wlachance) → review+

Comment 16

2 years ago
bugherder
https://hg.mozilla.org/mozilla-central/rev/f1482a4600fc
Status: ASSIGNED → RESOLVED
Last Resolved: 2 years ago
status-firefox48: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla48
You need to log in before you can comment on or make changes to this bug.