genservercerts target fails to compile with UTF-8 chars in environment

RESOLVED WONTFIX

Status

()

Core
Build Config
--
major
RESOLVED WONTFIX
9 years ago
3 months ago

People

(Reporter: Håkan Waara, Unassigned)

Tracking

Trunk
x86
Mac OS X
Points:
---

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments, 1 obsolete attachment)

(Reporter)

Description

9 years ago
Created attachment 337151 [details]
My mozconfig (firefox debug)

OS X 10.5, building Firefox debug, Python 2.6a3

I'm not sure about the right Bugzilla component, or even what this script does, but my mozilla-central pull from 5 minutes ago now fails.

Could it be that I have installed python 2.6a3? Python 2.6 is soon stable, so the error might be worthwhile to fix then.

Here's the output:

/usr/local/bin/python ../../_profile/pgo/genpgocert.py --gen-server
/Users/hakan/Programmering/mozilla/mozilla-central/ff-debug/build/pgo/automation.py:85: DeprecationWarning: The popen2 module is deprecated.  Use the subprocess module.
  import popen2
pk12util: PKCS12 IMPORT SUCCESSFUL
Traceback (most recent call last):
  File "../../_profile/pgo/genpgocert.py", line 156, in <module>
certificateStatus = createSSLServerCertificate(CERTS_DIR)
  File "../../_profile/pgo/genpgocert.py", line 135, in createSSLServerCertificate
status = runUtil(certutil, ["-S", "-s", "CN=%s" % firstLocation, "-t", "Pu,,", "-c", "pgo temporary ca", "-m", "2", "-8", locationsParam, "-v", "12", "-n", "pgo server certificate", "-d", dbDir, "-z", rndfilePath, "-f", pwfilePath])
  File "../../_profile/pgo/genpgocert.py", line 27, in runUtil
proc = automation.Process(util, args, automation.environment(), inputdata)
  File "/Users/hakan/Programmering/mozilla/mozilla-central/ff-debug/build/pgo/automation.py", line 93, in __init__
cmd = " ".join(cmd)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xe5 in position 25: ordinal not in range(128)
make[4]: *** [libs] Error 1
make[3]: *** [libs_tier_toolkit] Error 2
make[2]: *** [tier_toolkit] Error 2
make[1]: *** [default] Error 2
make: *** [build] Error 2
We added SSL support to mochitest, this script generates the server certs using NSS utils. I assume you have some non-ascii something that is making it goof up (probably your name somewhere :)

You can --disable-tests to get around this for the moment.
Blocks: 428009
Component: Build Config → Testing
QA Contact: build-config → testing
I saw this too, although I didn't re-run autoconf... could that be it?
I would more say it could be problem with IDN hosts in server-location.txt file that the version of Python could not handle. Just guess, will try on my mac.
But... UnicodeDecodeError says character at position 25 could not be decoded. The string being join is the command line. The first items in the array are environment variable=value pairs. On position 25, as I read the code, could not be the list of host but the environment listed.

Please check you env variables if there is not anything wrong and also check different version of python. I suspect that in alpha 3 version could be a bug.
(Reporter)

Comment 5

9 years ago
My environment variables have unicode in them, but that should be OK because my LANG=sv_SE.UTF-8 and LOCALE=UTF-8 etc.  

When I comment out this variable in my .bash_profile, it works:
export DARCS_DONT_ESCAPE_EXTRA="å@"

So does this script require all environment variables to be only ASCII? I would say that is a bug. It will likely bite lots of non-US people.
Component: Testing → Build Config
(Reporter)

Comment 6

9 years ago
If that's of any interest, in the cmd array that variable looks like this:

"DARCS_DONT_ESCAPE_EXTRA='\xe5@' ",
I can reproduce it with Python 2.5.2. 

STR:

export LANG=sv_SE.UTF-8
export LOCALE=UTF-8
export DARCS_DONT_ESCAPE_EXTRA="\303@"
src$ make -C objdir/testing/mochitest

[...]
Traceback (most recent call last):
  File "../../_profile/pgo/genpgocert.py", line 156, in <module>
    certificateStatus = createSSLServerCertificate(CERTS_DIR)
  File "../../_profile/pgo/genpgocert.py", line 135, in createSSLServerCertificate
    status = runUtil(certutil, ["-S", "-s", "CN=%s" % firstLocation, "-t", "Pu,,", "-c", "pgo temporary ca", "-m", "2", "-8", locationsParam, "-v", "12", "-n", "pgo server certificate", "-d", dbDir, "-z", rndfilePath, "-f", pwfilePath])
  File "../../_profile/pgo/genpgocert.py", line 27, in runUtil
    proc = automation.Process(util, args, automation.environment(), inputdata)
  File ".../objdir/build/pgo/automation.py", line 93, in __init__
    cmd = " ".join(cmd)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 25: ordinal not in range(128)
make[1]: *** [libs] Error 1
make: *** [default] Error 2
QA Contact: testing → build-config
Created attachment 337276 [details] [diff] [review]
Possible fix

When LOCALE variable is set we convert the value of an environment variable from uncoding defined by LOCALE to ASCII. This fixes the problem according to STR from previous comment.
Assignee: nobody → honzab
Status: NEW → ASSIGNED
Attachment #337276 - Flags: review?(benjamin)
Created attachment 337277 [details] [diff] [review]
Possible fix

Ops, forget to remove debugging print.
Attachment #337276 - Attachment is obsolete: true
Attachment #337277 - Flags: review?(benjamin)
Attachment #337276 - Flags: review?(benjamin)
I don't understand this patch. Why is there any unicode conversion going on at all? `cmd` should be entirely short strings, AFAICT, which shouldn't go through any conversions at all...
(In reply to comment #10)
> I don't understand this patch. Why is there any unicode conversion going on at
> all? `cmd` should be entirely short strings, AFAICT, which shouldn't go through
> any conversions at all...

The problem is in str.join(). The cmd list being joined contains combination of 'str' and 'unicode' objects. Environment variable pairs are unidcode, some others are str. When str.join is executed it does implicit conversion on all items to be ascii (str). When trying to convert unicode object that has a char > 128 exception is thrown. The same exception surprisingly occurs when I instead of " ".join(cmd) do u" ".join(cmd). I would expect the objects are converted to unicode and there would be no problem. 

So, I have one more solution, when putting the env variables to the cmd array I do decode to utf-8 on values:

        for (k, v) in env.iteritems():
          cmd.append(k + "='" + v.decode("utf-8") + "' ")

Whith this change it works, but IMHO it is in principal almost the same as the original fix. We still have to respect the LOCALE variable when it is set.
Why are environment pairs unicode? They aren't on my setup:

>>> repr(os.environ['LANG'])
"'en_US.UTF-8'"

I don't think os.environ should be returning unicode, unless there's some python configuration step I've missed (and in that case, we should try to disable it)
Attachment #337277 - Flags: review?(benjamin) → review-
Changing title.

See bug 459699 for details how building and generation of certificates changed. It is no more generated during full build process but on demand when changes to the host list are made.
Summary: mozilla-central fails to compile (failure in pgo/genpgocert.py) → genservercerts target fails to compile with UTF-8 chars in environment
Assignee: honzab.moz → nobody
Status: ASSIGNED → NEW

Updated

3 months ago
Status: NEW → RESOLVED
Last Resolved: 3 months ago
Resolution: --- → WONTFIX

Comment 14

3 months ago
Bug appears to be in automation.py. I suspect this has been fixed elsewhere.
You need to log in before you can comment on or make changes to this bug.