810435 - mach xpcshell-test fails with TypeError: environment can only contain strings

Status: RESOLVED FIXED

(Testing :: XPCShell Harness, defect)

Description

[Marco Bonardo [:mak]]

I have this failure when trying to run xpcshell-tests, environment is win7 x64 and the common mozilla-build

TypeError: environment can only contain strings

  File "c:\mozilla\mozilla-inbound\testing/xpcshell/mach_commands.py", line 137, in run_xpcshell_test
    xpcshell.run_test(**params)
  File "c:\mozilla\mozilla-inbound\testing/xpcshell/mach_commands.py", line 65, in run_test
    self._run_xpcshell_harness(**args)
  File "c:\mozilla\mozilla-inbound\testing/xpcshell/mach_commands.py", line 111, in _run_xpcshell_harness
    xpcshell.runTests(**args)
  File "c:\mozilla\mozilla-inbound\testing/xpcshell\runxpcshelltests.py", line 801, in runTests
    stdout=pStdout, stderr=pStderr, env=self.env, cwd=testdir)
  File "c:\mozilla\mozilla-inbound\testing/xpcshell\runxpcshelltests.py", line 310, in launchProcess
    env=env, cwd=cwd)
  File "c:\mozilla\mozilla-build\python\lib\subprocess.py", line 679, in __init__
    errread, errwrite)
  File "c:\mozilla\mozilla-build\python\lib\subprocess.py", line 893, in _execute_child
    startupinfo)

I tried to figure out where this comes from, what looks like is that here os.path.join generates a unicode string for 'xpcshell'

http://mxr.mozilla.org/mozilla-central/source/testing/xpcshell/mach_commands.py#82

then here we use xpcshell as xrePath

http://mxr.mozilla.org/mozilla-central/source/testing/xpcshell/runxpcshelltests.py#117

then here we add xrePath to the current $PATH and set it to env['PATH'] that is now unicode

http://mxr.mozilla.org/mozilla-central/source/testing/xpcshell/runxpcshelltests.py#139

finally we pass env to Popen, that complains

http://mxr.mozilla.org/mozilla-central/source/testing/xpcshell/runxpcshelltests.py#309

My workaround was just to:

'xpcshell': os.path.join(self.bindir, 'xpcshell').encode('utf-8'),

but I suppose there is a better solution.

Comment 1

[Gregory Szorc [:gps]]

Nice detective work!

I hate this Python bug so much. The problem magically goes away in Python 2.7.3. Since you are on Windows, I'm guessing you are using the Python provided by MozillaBuild, which is Python 2.7.2.

The mach command should likely convert the unicode strings to regular strings before it passes them into runxpcshelltests.py. I think this is the preferred solution because runxpcshelltests.py is not Unicode aware and mach is.

Comment 2

[Marco Bonardo [:mak]]

Yes I use that version of python in mozilla-build.
Actually my workaround works only for a full harness, single tests is even worse (there's likely other data passed through with wrong encoding).

Comment 3

[Gregory Szorc [:gps]]

Attachment: Convert unicode to str

This is necessary evil, I reckon. Once MozillaBuild ships with Python 2.7.3, I may change the mach driver to enforce that point release so we don't have to workaround these unicode bugs.

Comment 4

[Jeff Hammel]

Comment on attachment 681358 [details] [diff] [review]
Convert unicode to str

+        for k in args:
+            v = args[k]

I generally do `for k, v in args.items()`

Comment 5

[Gregory Szorc [:gps]]

(In reply to Jeff Hammel [:jhammel] from comment #4)
> Comment on attachment 681358 [details] [diff] [review]
> Convert unicode to str
> 
> +        for k in args:
> +            v = args[k]
> 
> I generally do `for k, v in args.items()`

This is what happens when I write JavaScript for a week. Ugh.

Comment 6

[Gregory Szorc [:gps]]

https://hg.mozilla.org/mozilla-central/rev/7fe91eedc16c