767249 - Building FF fails if system codepage is 950

Status: RESOLVED WORKSFORME

(Core :: Layout: Text and Fonts, defect)

Description

[Edmund Wong (:ewong)]

Running Vista English (codepage 950) w/ MSVC2010.  Recently pulling trunk, and building FF failed.  (Log is a lot longer, but bugzilla limits the log
that I can dump.)

l:\mozstuff\mozprgs\mc\config\rules.mk:1014:0$ /l/mozstuff/mozprgs/mc/obj-ff/_virtualenv/Scripts/python.exe -O l:/mozstuff/mozprgs/mc/build/cl.py cl -Fohb-unicode.obj -c -D_HAS_EXCEPTIONS=0 -I../../../dist/stl_wrappers  -DPACKAGE_VERSION="\"moz\"" -DPACKAGE_BUGREPORT="\"http://bugzilla.mozilla.org/\"" -DHAVE_OT=1 -DHB_NO_MT -Il:/mozstuff/mozprgs/mc/gfx/harfbuzz/src  -Il:/mozstuff/mozprgs/mc/gfx/harfbuzz/src -I. -I../../../dist/include  -Il:/mozstuff/mozprgs/mc/obj-ff/dist/include/nspr -Il:/mozstuff/mozprgs/mc/obj-ff/dist/include/nss        -TP -nologo -W3 -Gy -Fdgenerated.pdb -wd4800 -we4553 -GR-  -DNDEBUG -DTRIMMED -Zi -UDEBUG -DNDEBUG -O1 -Oy -MD            -FI ../../../dist/include/mozilla-config.h -DMOZILLA_CLIENT -UDEBUG l:/mozstuff/mozprgs/mc/gfx/harfbuzz/src/hb-unicode.cc
l:\mozstuff\mozprgs\mc\config\rules.mk:1014:0: command '/l/mozstuff/mozprgs/mc/obj-ff/_virtualenv/Scripts/python.exe -O l:/mozstuff/mozprgs/mc/build/cl.py cl -Fohb-shape.obj -c -D_HAS_EXCEPTIONS=0 -I../../../dist/stl_wrapperhb-unicode.cc

l:\mozstuff\mozprgs\mc\gfx\harfbuzz\src\hb-private.hh : warning C4819: The file contains a character that cannot be represented in the current code page (950). Save the file in Unicode format to prevent data loss

l:\mozstuff\mozprgs\mc\gfx\harfbuzz\src\hb-private.hh(531) : error C2001: newline in constant

l:\mozstuff\mozprgs\mc\gfx\harfbuzz\src\hb-private.hh(532) : error C2958: the left parenthesis '(' found at 'l:\mozstuff\mozprgs\mc\gfx\harfbuzz\src\hb-private.hh(528)' was not matched correctly

l:\mozstuff\mozprgs\mc\gfx\harfbuzz\src\hb-private.hh : warning C4819: The file contains a character that cannot be represented in the current code page (950). Save the file in Unicode format to prevent data loss

l:\mozstuff\mozprgs\mc\gfx\harfbuzz\src\hb-private.hh : warning C4819: The file contains a character that cannot be represented in the current code page (950). Save the file in Unicode format to prevent data loss

l:\mozstuff\mozprgs\mc\gfx\harfbuzz\src\hb-private.hh : warning C4819: The file contains a character that cannot be represented in the current code page (950). Save the file in Unicode format to prevent data loss

l:\mozstuff\mozprgs\mc\gfx\harfbuzz\src\hb-private.hh : warning C4819: The file contains a character that cannot be represented in the current code page (950). Save the file in Unicode format to prevent data loss

l:\mozstuff\mozprgs\mc\gfx\harfbuzz\src\hb-object-private.hh(62) : warning C4197: 'hb_atomic_int_t' : top-level volatile in cast is ignored

l:\mozstuff\mozprgs\mc\gfx\harfbuzz\src\hb-object-private.hh(134) : warning C4197: 'hb_atomic_int_t' : top-level volatile in cast is ignored

l:/mozstuff/mozprgs/mc/gfx/harfbuzz/src/hb-unicode.cc(103) : warning C4197: 'hb_atomic_int_t' : top-level volatile in cast is ignored

hb-warning.cc

l:\mozstuff\mozprgs\mc\gfx\harfbuzz\src\hb-private.hh : warning C4819: The file contains a character that cannot be represented in the current code page (950). Save the file in Unicode format to prevent data loss

l:\mozstuff\mozprgs\mc\gfx\harfbuzz\src\hb-private.hh(531) : error C2001: newline in constant

l:\mozstuff\mozprgs\mc\gfx\harfbuzz\src\hb-private.hh(532) : error C2958: the left parenthesis '(' found at 'l:\mozstuff\mozprgs\mc\gfx\harfbuzz\src\hb-private.hh(528)' was not matched correctly

l:\mozstuff\mozprgs\mc\gfx\harfbuzz\src\hb-private.hh : warning C4819: The file contains a character that cannot be represented in the current code page (950). Save the file in Unicode format to prevent data loss

s  -DPACKAGE_VERSION="\"moz\"" -DPACKAGE_BUGREPORT="\"http://bugzilla.mozilla.org/\"" -DHAVE_OT=1 -DHB_NO_MT -Il:/mozstuff/mozprgs/mc/gfx/harfbuzz/src  -Il:/mozstuff/mozprgs/mc/gfx/harfbuzz/src -I. -I../../../dist/include  -Il:/mozstuff/mozprgs/mc/obj-ff/dist/include/nspr -Il:/mozstuff/mozprgs/mc/obj-ff/dist/include/nss        -TP -nologo -W3 -Gy -Fdgenerated.pdb -wd4800 -we4553 -GR-  -DNDEBUG -DTRIMMED -Zi -UDEBUG -DNDEBUG -O1 -Oy -MD            -FI ../../../dist/include/mozilla-config.h -DMOZILLA_CLIENT -UDEBUG l:/mozstuff/mozprgs/mc/gfx/harfbuzz/src/hb-shape.cc' failed, return code 2
l:\mozstuff\mozprgs\mc\config\rules.mk:1014:0$ /l/mozstuff/mozprgs/mc/obj-ff/_virtualenv/Scripts/python.exe -O l:/mozstuff/mozprgs/mc/build/cl.py cl -Fohb-warning.obj -c -D_HAS_EXCEPTIONS=0 -I../../../dist/stl_wrappers  -DPACKAGE_VERSION="\"moz\"" -DPACKAGE_BUGREPORT="\"http://bugzilla.mozilla.org/\"" -DHAVE_OT=1 -DHB_NO_MT -Il:/mozstuff/mozprgs/mc/gfx/harfbuzz/src  -Il:/mozstuff/mozprgs/mc/gfx/harfbuzz/src -I. -I../../../dist/include  -Il:/mozstuff/mozprgs/mc/obj-ff/dist/include/nspr -Il:/mozstuff/mozprgs/mc/obj-ff/dist/include/nss        -TP -nologo -W3 -Gy -Fdgenerated.pdb -wd4800 -we4553 -GR-  -DNDEBUG -DTRIMMED -Zi -UDEBUG -DNDEBUG -O1 -Oy -MD            -FI ../../../dist/include/mozilla-config.h -DMOZILLA_CLIENT -UDEBUG l:/mozstuff/mozprgs/mc/gfx/harfbuzz/src/hb-warning.cc
l:\mozstuff\mozprgs\mc\config\rules.mk:1014:0: command '/l/mozstuff/mozprgs/mc/obj-ff/_virtualenv/Scripts/python.exe -O l:/mozstuff/mozprgs/mc/build/cl.py cl -Fohb-unicode.obj -c -D_HAS_EXCEPTIONS=0 -I../../../dist/stl_wrappers  -DPACKAGE_VERSION="\"moz\"" -DPACKAGE_BUGREPORT="\"http://bugzilla.mozilla.org/\"" -DHAVE_OT=1 -DHB_NO_MT -Il:/mozstuff/mozprgs/mc/gfx/harfbuzz/src  -Il:/mozstuff/mozprgs/mc/gfx/harfbuzz/src -I. -I../../../dist/include  -Il:/mozstuff/mozprgs/mc/obj-ff/dist/include/nspr -Il:/mozstuff/mozprgs/mc/obj-ff/dist/include/nss        -TP -nologo -W3 -Gy -Fdgenerated.pdb -wd4800 -we4553 -GR-  -DNDEBUG -DTRIMMED -Zi -UDEBUG -DNDEBUG -O1 -Oy -MD            -FI ../../../dist/include/mozilla-config.h -DMOZILLA_CLIENT -UDEBUG l:/mozstuff/mozprgs/mc/gfx/harfbuzz/src/hb-unicode.cc' failed, return code 2
<mozharfbuzz.lib.desc>: Found error
<mozharfbuzz.lib.desc>: Found error
<mozharfbuzz.lib.desc>: Found error
<mozharfbuzz.lib.desc>: Found error
<mozharfbuzz.lib.desc>: Found error
<mozharfbuzz.lib.desc>: Found error
<mozharfbuzz.lib.desc>: Found error
<mozharfbuzz.lib.desc>: Found error
<mozharfbuzz.lib.desc>: Found error
<mozharfbuzz.lib.desc>: Found error
<mozharfbuzz.lib.desc>: Found error
<mozharfbuzz.lib.desc>: Found error
<mozharfbuzz.lib.desc>: Found error
<mozharfbuzz.lib.desc>: Found error
<mozharfbuzz.lib.desc>: Found error
<mozharfbuzz.lib.desc>: Found error
l:\mozstuff\mozprgs\mc\config\rules.mk:1014:0: command '/l/mozstuff/mozprgs/mc/obj-ff/_virtualenv/Scripts/python.exe -O l:/mozstuff/mozprgs/mc/build/cl.py cl -Fohb-warning.obj -c -D_HAS_EXCEPTIONS=0 -I../../../dist/stl_wrappers  -DPACKAGE_VERSION="\"moz\"" -DPACKAGE_BUGREPORT="\"http://bugzilla.mozilla.org/\"" -DHAVE_OT=1 -DHB_NO_MT -Il:/mozstuff/mozprgs/mc/gfx/harfbuzz/src  -Il:/mozstuff/mozprgs/mc/gfx/harfbuzz/src -I. -I../../../dist/include  -Il:/mozstuff/mozprgs/mc/obj-ff/dist/include/nspr -Il:/mozstuff/mozprgs/mc/obj-ff/dist/include/nss        -TP -nologo -W3 -Gy -Fdgenerated.pdb -wd4800 -we4553 -GR-  -DNDEBUG -DTRIMMED -Zi -UDEBUG -DNDEBUG -O1 -Oy -MD            -FI ../../../dist/include/mozilla-config.h -DMOZILLA_CLIENT -UDEBUG l:/mozstuff/mozprgs/mc/gfx/harfbuzz/src/hb-warning.cc' failed, return code 2
<mozharfbuzz.lib.desc>: Found error
<libs>: Found error
l:\mozstuff\mozprgs\mc\config\makefiles\target_libs.mk:103:0: command 'l:/mozstuff/mozilla-build/python/python.exe l:/mozstuff/mozprgs/mc/build/pymake/pymake/../make.py -C harfbuzz/src libs' failed, return code 2
l:\mozstuff\mozprgs\mc\config\makefiles\target_libs.mk:18:0: command 'l:/mozstuff/mozilla-build/python/python.exe l:/mozstuff/mozprgs/mc/build/pymake/pymake/../make.py -C gfx libs' failed, return code 2
l:\mozstuff\mozprgs\mc\config\rules.mk:607:0: command 'l:/mozstuff/mozilla-build/python/python.exe l:/mozstuff/mozprgs/mc/build/pymake/pymake/../make.py libs_tier_platform' failed, return code 2
l:\mozstuff\mozprgs\mc\config\rules.mk:572:0: command 'l:/mozstuff/mozilla-build/python/python.exe l:/mozstuff/mozprgs/mc/build/pymake/pymake/../make.py  tier_platform' failed, return code 2
l:\mozstuff\mozprgs\mc\client.mk:344:0: command 'l:/mozstuff/mozilla-build/python/python.exe l:/mozstuff/mozprgs/mc/build/pymake/pymake/../make.py -j2 -C l:/mozstuff/mozprgs/mc/obj-ff' failed, return code 2
l:\mozstuff\mozprgs\mc\client.mk:144:0: command 'l:/mozstuff/mozilla-build/python/python.exe l:/mozstuff/mozprgs/mc/build/pymake/pymake/../make.py -f l:/mozstuff/mozprgs/mc/client.mk realbuild' failed, return code 2
Generating LALR tables
warning: interface 'nsIScriptSecurityManager' is scriptable but derives from non-scriptable 'nsIXPCSecurityManager', l:/mozstuff/mozprgs/mc/caps/idl/nsIScriptSecurityManager.idl line 13:0
interface nsIScriptSecurityManager : nsIXPCSecurityManager
^
warning: interface 'nsIScriptSecurityManager' is scriptable but derives from non-scriptable 'nsIXPCSecurityManager', l:/mozstuff/mozprgs/mc/caps/idl/nsIScriptSecurityManager.idl line 13:0
interface nsIScriptSecurityManager : nsIXPCSecurityManager
^
warning: [const] doesn't make sense on builtin types., <class 'xpidl.BuiltinLocation'>
warning: [const] doesn't make sense on builtin types., <class 'xpidl.BuiltinLocation'>
warning: [const] doesn't make sense on builtin types., <class 'xpidl.BuiltinLocation'>
warning: [const] doesn't make sense on builtin types., <class 'xpidl.BuiltinLocation'>
warning: [const] doesn't make sense on builtin types., <class 'xpidl.BuiltinLocation'>
warning: [const] doesn't make sense on builtin types., <class 'xpidl.BuiltinLocation'>
warning: [const] doesn't make sense on builtin types., <class 'xpidl.BuiltinLocation'>
warning: [const] doesn't make sense on builtin types., <class 'xpidl.BuiltinLocation'>
warning: [const] doesn't make sense on builtin types., <class 'xpidl.BuiltinLocation'>
warning: [const] doesn't make sense on builtin types., <class 'xpidl.BuiltinLocation'>

Comment 1

[Jonathan Kew [:jfkthame]]

This seems to be happening because the harfbuzz code includes some string literals in UTF-8, but MSVC is interpreting the source using the system codepage. I thought there was some way to tell the compiler what codepage to use, but I didn't find any such option in a little Googling. :(

The potentially misinterpreted string constants wouldn't affect normal operation, as they're just part of optional debug/tracing output, but in this case unfortunately they're breaking compilation altogether. Maybe we can persuade Behdad to avoid non-ASCII string literals in the source, for the sake of easier portability...

Comment 2

[Simon Montagu :smontagu]

As far as I know, the only way to tell the compiler that the source files are UTF-8 is to use a BOM. This is not an acceptable solution, because the BOM makes mxr assume the file is binary and not index it.

IINM the non-ASCII codepoints in themselves should only cause a warning, not an error, and the error is probably a result of the compiler ignoring the rest of the line, so it's possible it can be reduced back to a warning by recasting the code.

Comment 3

[Behdad Esfahbod]

Ugh.  This is ugly.  Let me fix it somehow.

Would using octal or hex escape sequences acceptable?

Comment 4

[Behdad Esfahbod]

I pushed a fix using octal escapes to master.  Makes "gcc -fexec-charset=ASCII" accept the code.  Didn't test with MSVC.  Let me know if it works.

Comment 5

[wjywbs]

I encountered the same error. My code base is 936. f:\moz\src\gfx\harfbuzz\src\hb-private.hh(531) : error C2001: 常量中有换行符(\n exists in const)
f:\moz\src\gfx\harfbuzz\src\hb-private.hh(532) : error C2958: 左 圆括号“(”(在
“f:\moz\src\gfx\harfbuzz\src\hb-private.hh(528)”中找到)未能正确匹配 (can't match ")" )