Last Comment Bug 453909 - genservercerts target fails to compile with UTF-8 chars in environment
: genservercerts target fails to compile with UTF-8 chars in environment
Status: NEW
:
Product: Core
Classification: Components
Component: Build Config (show other bugs)
: Trunk
: x86 Mac OS X
: -- major with 1 vote (vote)
: ---
Assigned To: Nobody; OK to take it and work on it
:
: Gregory Szorc [:gps]
Mentors:
Depends on:
Blocks: 428009
  Show dependency treegraph
 
Reported: 2008-09-05 16:49 PDT by Håkan Waara
Modified: 2012-02-03 12:31 PST (History)
4 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---


Attachments
My mozconfig (firefox debug) (265 bytes, text/plain)
2008-09-05 16:49 PDT, Håkan Waara
no flags Details
Possible fix (475 bytes, patch)
2008-09-07 03:11 PDT, Honza Bambas (:mayhemer)
no flags Details | Diff | Splinter Review
Possible fix (437 bytes, patch)
2008-09-07 03:13 PDT, Honza Bambas (:mayhemer)
benjamin: review-
Details | Diff | Splinter Review

Description Håkan Waara 2008-09-05 16:49:04 PDT
Created attachment 337151 [details]
My mozconfig (firefox debug)

OS X 10.5, building Firefox debug, Python 2.6a3

I'm not sure about the right Bugzilla component, or even what this script does, but my mozilla-central pull from 5 minutes ago now fails.

Could it be that I have installed python 2.6a3? Python 2.6 is soon stable, so the error might be worthwhile to fix then.

Here's the output:

/usr/local/bin/python ../../_profile/pgo/genpgocert.py --gen-server
/Users/hakan/Programmering/mozilla/mozilla-central/ff-debug/build/pgo/automation.py:85: DeprecationWarning: The popen2 module is deprecated.  Use the subprocess module.
  import popen2
pk12util: PKCS12 IMPORT SUCCESSFUL
Traceback (most recent call last):
  File "../../_profile/pgo/genpgocert.py", line 156, in <module>
certificateStatus = createSSLServerCertificate(CERTS_DIR)
  File "../../_profile/pgo/genpgocert.py", line 135, in createSSLServerCertificate
status = runUtil(certutil, ["-S", "-s", "CN=%s" % firstLocation, "-t", "Pu,,", "-c", "pgo temporary ca", "-m", "2", "-8", locationsParam, "-v", "12", "-n", "pgo server certificate", "-d", dbDir, "-z", rndfilePath, "-f", pwfilePath])
  File "../../_profile/pgo/genpgocert.py", line 27, in runUtil
proc = automation.Process(util, args, automation.environment(), inputdata)
  File "/Users/hakan/Programmering/mozilla/mozilla-central/ff-debug/build/pgo/automation.py", line 93, in __init__
cmd = " ".join(cmd)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 25: ordinal not in range(128)
make[4]: *** [libs] Error 1
make[3]: *** [libs_tier_toolkit] Error 2
make[2]: *** [tier_toolkit] Error 2
make[1]: *** [default] Error 2
make: *** [build] Error 2
Comment 1 Ted Mielczarek [:ted.mielczarek] 2008-09-05 16:53:23 PDT
We added SSL support to mochitest, this script generates the server certs using NSS utils. I assume you have some non-ascii something that is making it goof up (probably your name somewhere :)

You can --disable-tests to get around this for the moment.
Comment 2 Benjamin Smedberg [:bsmedberg] 2008-09-05 16:59:23 PDT
I saw this too, although I didn't re-run autoconf... could that be it?
Comment 3 Honza Bambas (:mayhemer) 2008-09-06 02:17:07 PDT
I would more say it could be problem with IDN hosts in server-location.txt file that the version of Python could not handle. Just guess, will try on my mac.
Comment 4 Honza Bambas (:mayhemer) 2008-09-06 02:29:43 PDT
But... UnicodeDecodeError says character at position 25 could not be decoded. The string being join is the command line. The first items in the array are environment variable=value pairs. On position 25, as I read the code, could not be the list of host but the environment listed.

Please check you env variables if there is not anything wrong and also check different version of python. I suspect that in alpha 3 version could be a bug.
Comment 5 Håkan Waara 2008-09-06 02:57:51 PDT
My environment variables have unicode in them, but that should be OK because my LANG=sv_SE.UTF-8 and LOCALE=UTF-8 etc.  

When I comment out this variable in my .bash_profile, it works:
export DARCS_DONT_ESCAPE_EXTRA="å@"

So does this script require all environment variables to be only ASCII? I would say that is a bug. It will likely bite lots of non-US people.
Comment 6 Håkan Waara 2008-09-06 03:02:09 PDT
If that's of any interest, in the cmd array that variable looks like this:

"DARCS_DONT_ESCAPE_EXTRA='\xe5@' ",
Comment 7 Honza Bambas (:mayhemer) 2008-09-06 03:06:20 PDT
I can reproduce it with Python 2.5.2. 

STR:

export LANG=sv_SE.UTF-8
export LOCALE=UTF-8
export DARCS_DONT_ESCAPE_EXTRA="\303@"
src$ make -C objdir/testing/mochitest

[...]
Traceback (most recent call last):
  File "../../_profile/pgo/genpgocert.py", line 156, in <module>
    certificateStatus = createSSLServerCertificate(CERTS_DIR)
  File "../../_profile/pgo/genpgocert.py", line 135, in createSSLServerCertificate
    status = runUtil(certutil, ["-S", "-s", "CN=%s" % firstLocation, "-t", "Pu,,", "-c", "pgo temporary ca", "-m", "2", "-8", locationsParam, "-v", "12", "-n", "pgo server certificate", "-d", dbDir, "-z", rndfilePath, "-f", pwfilePath])
  File "../../_profile/pgo/genpgocert.py", line 27, in runUtil
    proc = automation.Process(util, args, automation.environment(), inputdata)
  File ".../objdir/build/pgo/automation.py", line 93, in __init__
    cmd = " ".join(cmd)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 25: ordinal not in range(128)
make[1]: *** [libs] Error 1
make: *** [default] Error 2
Comment 8 Honza Bambas (:mayhemer) 2008-09-07 03:11:03 PDT
Created attachment 337276 [details] [diff] [review]
Possible fix

When LOCALE variable is set we convert the value of an environment variable from uncoding defined by LOCALE to ASCII. This fixes the problem according to STR from previous comment.
Comment 9 Honza Bambas (:mayhemer) 2008-09-07 03:13:51 PDT
Created attachment 337277 [details] [diff] [review]
Possible fix

Ops, forget to remove debugging print.
Comment 10 Benjamin Smedberg [:bsmedberg] 2008-09-09 06:57:14 PDT
I don't understand this patch. Why is there any unicode conversion going on at all? `cmd` should be entirely short strings, AFAICT, which shouldn't go through any conversions at all...
Comment 11 Honza Bambas (:mayhemer) 2008-09-09 15:22:52 PDT
(In reply to comment #10)
> I don't understand this patch. Why is there any unicode conversion going on at
> all? `cmd` should be entirely short strings, AFAICT, which shouldn't go through
> any conversions at all...

The problem is in str.join(). The cmd list being joined contains combination of 'str' and 'unicode' objects. Environment variable pairs are unidcode, some others are str. When str.join is executed it does implicit conversion on all items to be ascii (str). When trying to convert unicode object that has a char > 128 exception is thrown. The same exception surprisingly occurs when I instead of " ".join(cmd) do u" ".join(cmd). I would expect the objects are converted to unicode and there would be no problem. 

So, I have one more solution, when putting the env variables to the cmd array I do decode to utf-8 on values:

        for (k, v) in env.iteritems():
          cmd.append(k + "='" + v.decode("utf-8") + "' ")

Whith this change it works, but IMHO it is in principal almost the same as the original fix. We still have to respect the LOCALE variable when it is set.
Comment 12 Benjamin Smedberg [:bsmedberg] 2008-09-09 16:25:32 PDT
Why are environment pairs unicode? They aren't on my setup:

>>> repr(os.environ['LANG'])
"'en_US.UTF-8'"

I don't think os.environ should be returning unicode, unless there's some python configuration step I've missed (and in that case, we should try to disable it)
Comment 13 Honza Bambas (:mayhemer) 2008-10-21 10:48:23 PDT
Changing title.

See bug 459699 for details how building and generation of certificates changed. It is no more generated during full build process but on demand when changes to the host list are made.

Note You need to log in before you can comment on or make changes to this bug.