846523 - Can't build on Japanese system locale

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Masatoshi Kimura [:emk]]

Build failed with the following error:

 1:11.99 creating config files and headers...
 1:11.99 Traceback (most recent call last):
 1:11.99   File "./config.status", line 855, in <module>
 1:11.99     config_status(**args)
 1:11.99   File "h:\m\mozilla-central\build\ConfigStatus.py", line 130, in confi
g_status
 1:11.99     backend.consume(definitions)
 1:11.99   File "h:\m\mozilla-central\python\mozbuild\mozbuild\backend\base.py",
 line 78, in consume
 1:11.99     for obj in objs:
 1:11.99   File "h:\m\mozilla-central\python\mozbuild\mozbuild\frontend\emitter.
py", line 34, in emit
 1:11.99     for out in output:
 1:11.99   File "h:\m\mozilla-central\python\mozbuild\mozbuild\frontend\reader.p
y", line 599, in read_mozbuild
 1:11.99     sys.exc_info()[2], other_error=e)
 1:11.99 mozbuild.frontend.reader.BuildReaderError: ============================
==
 1:11.99 ERROR PROCESSING MOZBUILD FILE
 1:11.99 ==============================
 1:11.99
 1:11.99 The error occurred while processing the following file:
 1:11.99
 1:11.99     h:/m/mozilla-central\moz.build
 1:11.99
 1:11.99 The error appears to be part of the mozbuild.frontend.reader Python mod
ule itself! It is possible you have stumbled across a legitimate bug.
 1:11.99
 1:11.99 Traceback (most recent call last):
 1:11.99   File "h:\m\mozilla-central\python\mozbuild\mozbuild\frontend\reader.p
y", line 575, in read_mozbuild
 1:11.99     filesystem_absolute=filesystem_absolute, descend=descend):
 1:11.99   File "h:\m\mozilla-central\python\mozbuild\mozbuild\frontend\reader.p
y", line 613, in _read_mozbuild
 1:11.99     sandbox = MozbuildSandbox(self.config, path)
 1:12.01   File "h:\m\mozilla-central\python\mozbuild\mozbuild\frontend\reader.p
y", line 162, in __init__
 1:12.01     v = v.decode('utf-8', 'strict')
 1:12.01   File "h:\m\mozilla-central\obj-i686-pc-mingw32\_virtualenv\lib\encodi
ngs\utf_8.py", line 16, in decode
 1:12.01     return codecs.utf_8_decode(input, errors, True)
 1:12.01 UnicodeDecodeError: 'utf8' codec can't decode byte 0x83 in position 325
7: invalid start byte
 1:12.01
 1:12.01 *** Fix above errors and then restart with               "c:/mozilla-bu
ild/python/python.exe h:/m/mozilla-central/build/pymake/pymake/../make.py -f cli
ent.mk build"
 1:12.03 h:\m\mozilla-central\client.mk:320:0: command 'cd obj-i686-pc-mingw32 &
&  MAKE="c:/mozilla-build/python/python.exe h:/m/mozilla-central/build/pymake/py
make/../make.py"  h:/m/mozilla-central/configure  \
 1:12.03   || ( echo "*** Fix above errors and then restart with\
 1:12.03                \"c:/mozilla-build/python/python.exe h:/m/mozilla-centra
l/build/pymake/pymake/../make.py -f client.mk build\"" && exit 1 )' failed, retu
rn code 1
 1:12.03 h:\m\mozilla-central\client.mk:332:0: command 'c:/mozilla-build/python/
python.exe h:/m/mozilla-central/build/pymake/pymake/../make.py -f h:/m/mozilla-c
entral/client.mk configure' failed, return code 2
 1:12.03 h:\m\mozilla-central\client.mk:160:0: command 'c:/mozilla-build/python/
python.exe h:/m/mozilla-central/build/pymake/pymake/../make.py -f h:/m/mozilla-c
entral/client.mk realbuild' failed, return code 2
 1:12.23 0 compiler warnings present.
Finished building. Built files are in obj-i686-pc-mingw32
2

$

I thought I said "Converting with utf-8 will not work." :(

Comment 1

[Gregory Szorc [:gps]]

(In reply to Masatoshi Kimura [:emk] from comment #0)
> I thought I said "Converting with utf-8 will not work." :(

And I thought you tested my patch queue and confirmed things worked! Where did we go wrong? I wonder if I lost a patch revision?

Comment 2

[Gregory Szorc [:gps]]

This is going to sound scary, but I'm pretty confident a gross hack to fix this is to change "strict" to "replace" in /python/mozbuild/mozbuild/frontend/reader.py:162.

Can you confirm that works?

Comment 3

[Masatoshi Kimura [:emk]]

(In reply to Gregory Szorc [:gps] from comment #2)
> This is going to sound scary, but I'm pretty confident a gross hack to fix
> this is to change "strict" to "replace" in
> /python/mozbuild/mozbuild/frontend/reader.py:162.
> 
> Can you confirm that works?

It may "work", but according to  bug 784841 comment 270, we shouldn't do that.
> So, it's important for us not to get this value wrong (i.e. wrong encoding):
> if it is wrong, header dependencies will be broken and incremental builds
> won't be proper.

I think we need to convert it with mbcs encoding on Windows.

Comment 4

[Gregory Szorc [:gps]]

(In reply to Masatoshi Kimura [:emk] from comment #3)
> It may "work", but according to  bug 784841 comment 270, we shouldn't do
> that.
> > So, it's important for us not to get this value wrong (i.e. wrong encoding):
> > if it is wrong, header dependencies will be broken and incremental builds
> > won't be proper.
> 
> I think we need to convert it with mbcs encoding on Windows.

Oh, I'm aware this is not the ideal way to solve the problem :)

We can get away with it because AFAIK the only Unicode showing up in config.status is from the localized cl.exe string to identify included files. And, since this string is not used by moz.build files (only by scripts executed as part of building), we don't care that the string isn't preserved properly. I'm content with a quick fix now and a proper fix in bug 844509.

Comment 5

[y-ii]

same problem on Chinese system locale

Comment 6

[Gregory Szorc [:gps]]

Attachment: Unicode exception for CL_INCLUDES_PREFIX, v1

This is dirty, but I believe it will get the job done.

This patch is still somewhat strict since it only excludes CL_INCLUDES_PREFIX from non-utf-8. If it isn't loose enough, we can see what other variables are holding us back or possibly exclude all variables (while potentially logging a warning).

Comment 7

[y-ii]

(In reply to Gregory Szorc [:gps] from comment #6)

It does not work. Same error.

Comment 8

[Gregory Szorc [:gps]]

(In reply to y-ii from comment #7)
> (In reply to Gregory Szorc [:gps] from comment #6)
> 
> It does not work. Same error.

Do you know enough about python to set a breakpoint and see what variable it is tripping up on?

Just add the following line in Python to add a breakpoint that when encountered will dump to an interactive debugger:

    import pdb; pdb.set_trace()

Just "p k" and "p v" to print k and v, respectively, and let me know what you get!

Comment 9

[y-ii]

(In reply to Gregory Szorc [:gps] from comment #8)

sorry，i am just a noob. So i don't know if i did the right thing.

without the pathch, "p k" get "CPP","p v" get "cl -E -nologo"

Comment 10

[y-ii]

I think I'm wrong.

I don't know where I can set a breakpoint. 
When I set breakpoint at line"except UnicodeDecodeError:", k "allsubsts",v "ACCEPT_MAR_CHANNEL_ID.............."

Comment 11

[(not currently active) Ted Mielczarek]

Can you attach config.status from your objdir to this bug?

Comment 12

[Gregory Szorc [:gps]]

Please attach the config.status file from your objdir so we can examine it.

Comment 13

[Gregory Szorc [:gps]]

Please also attach your config.status file.

Comment 14

[y-ii]

Attachment: config.status

Comment 15

[Masatoshi Kimura [:emk]]

Attachment: config.status

FYI, I can build the tree with the comment #3 change and it seems that .deps are properly generated.

Comment 16

[Masatoshi Kimura [:emk]]

> comment #3 change
comment #2 change

Comment 17

[Masatoshi Kimura [:emk]]

Comment on attachment 719820 [details] [diff] [review]
Unicode exception for CL_INCLUDES_PREFIX, v1

This patch didn't work.
The build ended up with the following infinite error messages:
191:47.40 mozbuild.frontend.reader.BuildReaderError: ===========================
===
191:47.40 ERROR PROCESSING MOZBUILD FILE
191:47.40 ==============================
191:47.40
191:47.40 The error occurred while processing the following file:
191:47.40
191:47.40     h:/m/mozilla-central\moz.build
191:47.40
191:47.40 The error appears to be part of the mozbuild.frontend.reader Python mo
dule itself! It is possible you have stumbled across a legitimate bug.
191:47.40
191:47.40 Traceback (most recent call last):
191:47.40   File "h:\m\mozilla-central\python\mozbuild\mozbuild\frontend\reader.
py", line 578, in read_mozbuild
191:47.42     filesystem_absolute=filesystem_absolute, descend=descend):
191:47.42   File "h:\m\mozilla-central\python\mozbuild\mozbuild\frontend\reader.
py", line 616, in _read_mozbuild
191:47.42     sandbox = MozbuildSandbox(self.config, path)
191:47.42   File "h:\m\mozilla-central\python\mozbuild\mozbuild\frontend\reader.
py", line 165, in __init__
191:47.42     v = v.decode('utf-8', 'strict')
191:47.42   File "h:\m\mozilla-central\obj-i686-pc-mingw32\_virtualenv\lib\encod
ings\utf_8.py", line 16, in decode
191:47.42     return codecs.utf_8_decode(input, errors, True)
191:47.42 UnicodeDecodeError: 'utf8' codec can't decode byte 0x83 in position 32
57: invalid start byte
191:47.42
191:47.42 h:\m\mozilla-central\config\rules.mk:1198:0: command 'h:/m/mozilla-cen
tral/obj-i686-pc-mingw32/_virtualenv/Scripts/python.exe ../config.status -n' fai
led, return code 1
191:47.42 Error remaking makefiles (ignored)
191:47.72 creating config files and headers...
191:47.72 Traceback (most recent call last):
191:47.72   File "../config.status", line 855, in <module>
191:47.72     config_status(**args)
191:47.72   File "h:\m\mozilla-central\build\ConfigStatus.py", line 130, in conf
ig_status
191:47.72     backend.consume(definitions)
191:47.72   File "h:\m\mozilla-central\python\mozbuild\mozbuild\backend\base.py"
, line 78, in consume
191:47.72     for obj in objs:
191:47.72   File "h:\m\mozilla-central\python\mozbuild\mozbuild\frontend\emitter
.py", line 34, in emit
191:47.72     for out in output:
191:47.72   File "h:\m\mozilla-central\python\mozbuild\mozbuild\frontend\reader.
py", line 602, in read_mozbuild
191:47.72     sys.exc_info()[2], other_error=e)
191:47.72 mozbuild.frontend.reader.BuildReaderError: ===========================
===
191:47.72 ERROR PROCESSING MOZBUILD FILE
191:47.72 ==============================
191:47.72
191:47.72 The error occurred while processing the following file:
191:47.72
191:47.72     h:/m/mozilla-central\moz.build
191:47.72
191:47.72 The error appears to be part of the mozbuild.frontend.reader Python mo
dule itself! It is possible you have stumbled across a legitimate bug.
191:47.73
191:47.73 Traceback (most recent call last):
191:47.73   File "h:\m\mozilla-central\python\mozbuild\mozbuild\frontend\reader.
py", line 578, in read_mozbuild
191:47.73     filesystem_absolute=filesystem_absolute, descend=descend):

Comment 18

[(not currently active) Ted Mielczarek]

Comment on attachment 719820 [details] [diff] [review]
Unicode exception for CL_INCLUDES_PREFIX, v1

Review of attachment 719820 [details] [diff] [review]:
-----------------------------------------------------------------

Sounds like this doesn't fix the problem.

Comment 19

[ICHIMARU Takeshi(a.k.a. Ayakawa)]

Attachment: sample patch

only test on japanese environment.

Comment 20

[Gregory Szorc [:gps]]

Attachment: Accept invalid UTF-8, v1

Let's loosen this up a bit.

If this doesn't work, then ���.

Comment 21

[Masatoshi Kimura [:emk]]

 1:04.35 creating ./config.status
 1:06.32 Reticulating splines...
 1:06.32 Traceback (most recent call last):
 1:06.32   File "./config.status", line 859, in <module>
 1:06.32     config_status(**args)
 1:06.32   File "h:\m\mozilla-central\build\ConfigStatus.py", line 122, in confi
g_status
 1:06.32     summary = backend.consume(definitions)
 1:06.32   File "h:\m\mozilla-central\python\mozbuild\mozbuild\backend\base.py",
 line 149, in consume
 1:06.32     for obj in objs:
 1:06.32   File "h:\m\mozilla-central\python\mozbuild\mozbuild\frontend\emitter.
py", line 39, in emit
 1:06.32     for out in output:
 1:06.32   File "h:\m\mozilla-central\python\mozbuild\mozbuild\frontend\reader.p
y", line 610, in read_mozbuild
 1:06.32     sys.exc_info()[2], other_error=e)
 1:06.32 mozbuild.frontend.reader.BuildReaderError: ============================
==
 1:06.32 ERROR PROCESSING MOZBUILD FILE
 1:06.32 ==============================
 1:06.32
 1:06.32 The error occurred while processing the following file:
 1:06.32
 1:06.32     h:/m/mozilla-central\moz.build
 1:06.32
 1:06.32 The error appears to be part of the mozbuild.frontend.reader Python mod
ule itself! It is possible you have stumbled across a legitimate bug.
 1:06.32
 1:06.32 Traceback (most recent call last):
 1:06.32   File "h:\m\mozilla-central\python\mozbuild\mozbuild\frontend\reader.p
y", line 586, in read_mozbuild
 1:06.32     filesystem_absolute=filesystem_absolute, descend=descend):
 1:06.32   File "h:\m\mozilla-central\python\mozbuild\mozbuild\frontend\reader.p
y", line 625, in _read_mozbuild
 1:06.32     sandbox = MozbuildSandbox(self.config, path)
 1:06.32   File "h:\m\mozilla-central\python\mozbuild\mozbuild\frontend\reader.p
y", line 169, in __init__
 1:06.32     log(self._log, logging.INFO, 'lossy_encoding',
 1:06.32 AttributeError: 'MozbuildSandbox' object has no attribute '_log'
 1:06.33
 1:06.33 *** Fix above errors and then restart with               "c:/mozilla-bu
ild/python/python.exe h:/m/mozilla-central/build/pymake/pymake/../make.py -f cli
ent.mk build"
 1:06.35 h:\m\mozilla-central\client.mk:320:0: command 'cd obj-i686-pc-mingw32 &
&  MAKE="c:/mozilla-build/python/python.exe h:/m/mozilla-central/build/pymake/py
make/../make.py"  h:/m/mozilla-central/configure  \
 1:06.35   || ( echo "*** Fix above errors and then restart with\
 1:06.35                \"c:/mozilla-build/python/python.exe h:/m/mozilla-centra
l/build/pymake/pymake/../make.py -f client.mk build\"" && exit 1 )' failed, retu
rn code 1
 1:06.35 h:\m\mozilla-central\client.mk:332:0: command 'c:/mozilla-build/python/
python.exe h:/m/mozilla-central/build/pymake/pymake/../make.py -f h:/m/mozilla-c
entral/client.mk configure' failed, return code 2
 1:06.36 h:\m\mozilla-central\client.mk:160:0: command 'c:/mozilla-build/python/
python.exe h:/m/mozilla-central/build/pymake/pymake/../make.py -f h:/m/mozilla-c
entral/client.mk realbuild' failed, return code 2

Comment 22

[Masatoshi Kimura [:emk]]

Why don't you take a commet #19 approach? We are using mbcs encoding anyway when config.status is executed directly. Please fix a current problem before worrying about future Python 3 migration.

Comment 23

[Gregory Szorc [:gps]]

locale.getsystemdefaultencodingwhatever() isn't reliable IIRC. While it may happen to work on your system, I wouldn't be surprised if it regressed behavior elsewhere because that function is returning an invalid encoding or one that doesn't match what's in config.status.

My patch didn't work because I didn't actually test it. Shame on me. I'll fix the patch and include a unit test. I'm confident it will then work for you.

Comment 24

[Masatoshi Kimura [:emk]]

Then what about "mbcs encoding on Windows"? (comment #3) config.status declares mbcs for a long time, but nobody complains.

Comment 25

[Gregory Szorc [:gps]]

Attachment: Accept invalid UTF-8, v2

This band aid includes tests and should work. I copied part of the byte sequence from bug 784841 to be sure this works.

Comment 26

[(not currently active) Ted Mielczarek]

Comment on attachment 722511 [details] [diff] [review]
Accept invalid UTF-8, v2

Review of attachment 722511 [details] [diff] [review]:
-----------------------------------------------------------------

::: python/mozbuild/mozbuild/frontend/reader.py
@@ +171,5 @@
> +                        log(self._log, logging.INFO, 'lossy_encoding',
> +                            {'variable': k},
> +                            'Lossy Unicode encoding for {variable}. See bug 844509.')
> +
> +                        v = v.decode('utf-8', 'replace')

We are probably going to have to figure out how to fix this right at some point. :-/

Comment 27

[Gregory Szorc [:gps]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/8920d009778e

Comment 28

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/8920d009778e