453909 - genservercerts target fails to compile with UTF-8 chars in environment

Status: RESOLVED WONTFIX

(Firefox Build System :: General, defect)

Description

[Håkan Waara]

Attachment: My mozconfig (firefox debug)

OS X 10.5, building Firefox debug, Python 2.6a3

I'm not sure about the right Bugzilla component, or even what this script does, but my mozilla-central pull from 5 minutes ago now fails.

Could it be that I have installed python 2.6a3? Python 2.6 is soon stable, so the error might be worthwhile to fix then.

Here's the output:

/usr/local/bin/python ../../_profile/pgo/genpgocert.py --gen-server
/Users/hakan/Programmering/mozilla/mozilla-central/ff-debug/build/pgo/automation.py:85: DeprecationWarning: The popen2 module is deprecated.  Use the subprocess module.
  import popen2
pk12util: PKCS12 IMPORT SUCCESSFUL
Traceback (most recent call last):
  File "../../_profile/pgo/genpgocert.py", line 156, in <module>
certificateStatus = createSSLServerCertificate(CERTS_DIR)
  File "../../_profile/pgo/genpgocert.py", line 135, in createSSLServerCertificate
status = runUtil(certutil, ["-S", "-s", "CN=%s" % firstLocation, "-t", "Pu,,", "-c", "pgo temporary ca", "-m", "2", "-8", locationsParam, "-v", "12", "-n", "pgo server certificate", "-d", dbDir, "-z", rndfilePath, "-f", pwfilePath])
  File "../../_profile/pgo/genpgocert.py", line 27, in runUtil
proc = automation.Process(util, args, automation.environment(), inputdata)
  File "/Users/hakan/Programmering/mozilla/mozilla-central/ff-debug/build/pgo/automation.py", line 93, in __init__
cmd = " ".join(cmd)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 25: ordinal not in range(128)
make[4]: *** [libs] Error 1
make[3]: *** [libs_tier_toolkit] Error 2
make[2]: *** [tier_toolkit] Error 2
make[1]: *** [default] Error 2
make: *** [build] Error 2

Comment 1

[(not currently active) Ted Mielczarek]

We added SSL support to mochitest, this script generates the server certs using NSS utils. I assume you have some non-ascii something that is making it goof up (probably your name somewhere :)

You can --disable-tests to get around this for the moment.

Comment 2

[Benjamin Smedberg]

I saw this too, although I didn't re-run autoconf... could that be it?

Comment 3

[Honza Bambas (:mayhemer)]

I would more say it could be problem with IDN hosts in server-location.txt file that the version of Python could not handle. Just guess, will try on my mac.

Comment 4

[Honza Bambas (:mayhemer)]

But... UnicodeDecodeError says character at position 25 could not be decoded. The string being join is the command line. The first items in the array are environment variable=value pairs. On position 25, as I read the code, could not be the list of host but the environment listed.

Please check you env variables if there is not anything wrong and also check different version of python. I suspect that in alpha 3 version could be a bug.

Comment 5

[Håkan Waara]

My environment variables have unicode in them, but that should be OK because my LANG=sv_SE.UTF-8 and LOCALE=UTF-8 etc.  

When I comment out this variable in my .bash_profile, it works:
export DARCS_DONT_ESCAPE_EXTRA="å@"

So does this script require all environment variables to be only ASCII? I would say that is a bug. It will likely bite lots of non-US people.

Comment 6

[Håkan Waara]

If that's of any interest, in the cmd array that variable looks like this:

"DARCS_DONT_ESCAPE_EXTRA='\xe5@' ",

Comment 7

[Honza Bambas (:mayhemer)]

I can reproduce it with Python 2.5.2. 

STR:

export LANG=sv_SE.UTF-8
export LOCALE=UTF-8
export DARCS_DONT_ESCAPE_EXTRA="\303@"
src$ make -C objdir/testing/mochitest

[...]
Traceback (most recent call last):
  File "../../_profile/pgo/genpgocert.py", line 156, in <module>
    certificateStatus = createSSLServerCertificate(CERTS_DIR)
  File "../../_profile/pgo/genpgocert.py", line 135, in createSSLServerCertificate
    status = runUtil(certutil, ["-S", "-s", "CN=%s" % firstLocation, "-t", "Pu,,", "-c", "pgo temporary ca", "-m", "2", "-8", locationsParam, "-v", "12", "-n", "pgo server certificate", "-d", dbDir, "-z", rndfilePath, "-f", pwfilePath])
  File "../../_profile/pgo/genpgocert.py", line 27, in runUtil
    proc = automation.Process(util, args, automation.environment(), inputdata)
  File ".../objdir/build/pgo/automation.py", line 93, in __init__
    cmd = " ".join(cmd)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 25: ordinal not in range(128)
make[1]: *** [libs] Error 1
make: *** [default] Error 2

Comment 8

[Honza Bambas (:mayhemer)]

Attachment: Possible fix

When LOCALE variable is set we convert the value of an environment variable from uncoding defined by LOCALE to ASCII. This fixes the problem according to STR from previous comment.

Comment 9

[Honza Bambas (:mayhemer)]

Attachment: Possible fix

Ops, forget to remove debugging print.

Comment 10

[Benjamin Smedberg]

I don't understand this patch. Why is there any unicode conversion going on at all? `cmd` should be entirely short strings, AFAICT, which shouldn't go through any conversions at all...

Comment 11

[Honza Bambas (:mayhemer)]

(In reply to comment #10)
> I don't understand this patch. Why is there any unicode conversion going on at
> all? `cmd` should be entirely short strings, AFAICT, which shouldn't go through
> any conversions at all...

The problem is in str.join(). The cmd list being joined contains combination of 'str' and 'unicode' objects. Environment variable pairs are unidcode, some others are str. When str.join is executed it does implicit conversion on all items to be ascii (str). When trying to convert unicode object that has a char > 128 exception is thrown. The same exception surprisingly occurs when I instead of " ".join(cmd) do u" ".join(cmd). I would expect the objects are converted to unicode and there would be no problem. 

So, I have one more solution, when putting the env variables to the cmd array I do decode to utf-8 on values:

        for (k, v) in env.iteritems():
          cmd.append(k + "='" + v.decode("utf-8") + "' ")

Whith this change it works, but IMHO it is in principal almost the same as the original fix. We still have to respect the LOCALE variable when it is set.

Comment 12

[Benjamin Smedberg]

Why are environment pairs unicode? They aren't on my setup:

>>> repr(os.environ['LANG'])
"'en_US.UTF-8'"

I don't think os.environ should be returning unicode, unless there's some python configuration step I've missed (and in that case, we should try to disable it)

Comment 13

[Honza Bambas (:mayhemer)]

Changing title.

See bug 459699 for details how building and generation of certificates changed. It is no more generated during full build process but on demand when changes to the host list are made.

Comment 14

[Gregory Szorc [:gps]]

Bug appears to be in automation.py. I suspect this has been fixed elsewhere.