1050561 - UnicodeDecodeError when running mochitest (SeaMonkey/Thunderbird only)

Status: RESOLVED FIXED

(Testing :: Mozbase, defect)

Description

[Frank Wein [:mcsmurf]]

This is likely a SeaMonkey/Thunderbird bug only and it may go away when merging c-c with m-c.
Currently one cannot various mochitest suites, e.g. "mozmake mochitest-browser-chrome" for SeaMonkey because of this error:
UnicodeDecodeError: 'ascii' codec can't decode byte 0xad in position 837: ordinal not in range(128)
in testing\mozbase\mozprocess\mozprocess\winprocess.py (probably line 154 or near there, I added some debug code).

This happens because the test environment (env variables) includes this one from configure.in:
'NONASCII=\xad\xad'
(see http://hg.mozilla.org/mozilla-central/annotate/2f198e81ed98//configure.in#l7792)

The code in winprocess.py tries to decode the whole byte string to ascii when appending the unicode string
u'LD_LIBRARY_PATH=f:\\mozilla\\tree-hg\\comm-central\\obj-i686-pc-mingw32\\mozilla\\dist\\bin'

This fails as \xad is not part of ascii standard. Not sure if there is a simple workaround or fix for this problem.

Comment 1

[Masatoshi Kimura [:emk]]

It means mochitest for SeaMonkey will fail on Japanese Windows.

Comment 2

[Masatoshi Kimura [:emk]]

I was unable to build SeaMonkey...

 0:32.81 Embedding manifest from h:/m/comm-central/mail/app/thunderbird.exe.mani
fest
 0:32.82
 0:32.82 h:/m/comm-central/mozilla/mail/app/thunderbird.exe.manifest : general e
rror c1010070: Failed to load and parse the manifest. ????????????????
 0:32.82
 0:32.83 h:/m/comm-central/config/rules.mk:703: recipe for target 'thunderbird.e
xe' failed

Why Thunderbird is being built when building SeaMonkey???

My .mozconfig is:
ac_add_options --enable-application=suite
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/../objdir-sm-release
ac_add_options --enable-winsdk-directx

Comment 3

[Frank Wein [:mcsmurf]]

That's Bug 1040009

Comment 4

[Masatoshi Kimura [:emk]]

Still fails with patches from bug 1040009:

$ ./mozilla/mach build
 0:00.28 c:\mozilla-build\mozmake\mozmake.EXE -f client.mk -s
 0:01.04 Adding client.mk options from h:/m/comm-central/.mozconfig:
 0:01.04     MOZ_OBJDIR=objdir-sm-release
 0:01.04     FOUND_MOZCONFIG := h:/m/comm-central/.mozconfig
 0:01.94 cd objdir-sm-release
 0:01.95 h:/m/comm-central/configure
 0:02.16 loading cache ./config.cache
 0:02.78 h:/m/comm-central/mozilla/configure: line 84: Ambiguous: command not fo
und
 0:02.78 Failed eval'ing the following:
 0:03.22 Ambiguous object directory detected. We detected that both h:\m\comm-ce
ntral\objdir-sm-release and h:\m\comm-central\mozilla\objdir-sm-release could be
 object directories. This is typically caused by having a mozconfig pointing to
a different object directory from the current working directory. To solve this p
roblem, ensure you do not have a default mozconfig in searched paths.
 0:03.24 *** Fix above errors and then restart with\
 0:03.24                "c:/mozilla-build/mozmake/mozmake.EXE -f client.mk build
"
 0:03.25 h:/m/comm-central/client.mk:349: recipe for target 'configure' failed
 0:03.25 mozmake.EXE[1]: *** [configure] Error 1
 0:03.25 client.mk:363: recipe for target 'objdir-sm-release/Makefile' failed
 0:03.25 mozmake.EXE: *** [objdir-sm-release/Makefile] Error 2
 0:03.25 0 compiler warnings present.
2

I also changed .mozconfig as follows, still no luck:
ac_add_options --enable-application=suite
mk_add_options MOZ_OBJDIR=objdir-sm-release
ac_add_options --enable-winsdk-directx

Comment 5

[Masatoshi Kimura [:emk]]

The build system insists on finding manifest from $SRCDIR/mozilla/suite/app rather than $SRCDIR/suite/app:

 0:20.46 Embedding manifest from h:/m/comm-central/suite/app/seamonkey.exe.manif
est
 0:20.49
 0:20.49 h:/m/comm-central/mozilla/suite/app/seamonkey.exe.manifest : general er
ror c1010070: Failed to load and parse the manifest. ????????????????

Comment 6

[Masatoshi Kimura [:emk]]

Even tbpl keeps on failing to build SeaMonkey since 22 July.
http://ftp.mozilla.org/pub/mozilla.org/seamonkey/nightly/latest-comm-central-trunk/
http://ftp.mozilla.org/pub/mozilla.org/seamonkey/nightly/2014-08-12-00-30-04-comm-central-trunk/

I have to conclude that SeaMonkey is virtually dead because no new contributor can build the tree.

Comment 7

[Frank Wein [:mcsmurf]]

BTW: This bug is not for sorting out build problems that are being worked on.

Comment 8

[Masatoshi Kimura [:emk]]

Still fails even with bug 1040009 patches:

 3:35.08 creating ./config.status
 3:35.65 Traceback (most recent call last):
 3:35.65   File "./config.status", line 924, in <module>
 3:35.65     config_status(**args)
 3:35.65   File "h:\m\comm-central\mozilla\python\mozbuild\mozbuild\config_statu
s.py", line 134, in config_status
 3:35.65     emitter = TreeMetadataEmitter(env)
 3:35.65   File "h:\m\comm-central\mozilla\python\mozbuild\mozbuild\frontend\emi
tter.py", line 84, in __init__
 3:35.65     mozinfo.find_and_update_from_json(config.topobjdir)
 3:35.65   File "h:\m\comm-central\mozilla\testing\mozbase\mozinfo\mozinfo\mozin
fo.py", line 153, in find_and_update_from_json
 3:35.65     build = MozbuildObject.from_environment()
 3:35.66   File "h:\m\comm-central\mozilla\python\mozbuild\mozbuild\base.py", li
ne 181, in from_environment
 3:35.66     raise ObjdirMismatchException(topobjdir, _config_topobjdir)
 3:35.66 mozbuild.base.ObjdirMismatchException: Objdir mismatch: h:\m\comm-centr
al\objdir-sm-release != h:\m\comm-central\mozilla\objdir-sm-release
 3:35.67 *** Fix above errors and then restart with\
 3:35.67                "c:/mozilla-build/mozmake/mozmake.EXE -f client.mk build
"
 3:35.68 h:/m/comm-central/client.mk:361: recipe for target 'configure' failed
 3:35.68 mozmake.EXE[1]: *** [configure] Error 1
 3:35.68 client.mk:375: recipe for target 'h:/m/comm-central/objdir-sm-release/M
akefile' failed
 3:35.69 mozmake.EXE: *** [h:/m/comm-central/objdir-sm-release/Makefile] Error 2

 3:35.70 0 compiler warnings present.
2

Comment 9

[Andrew Halberstadt [:ahal]]

Hi Masatoshi, this bug is about a UnicodeDecodeError in mozprocess. If you believe you have build issues that are not covered by bug 1040009, please file a new bug under 'Core/Build Config'.

Frank, I'd guess that the UnicodeDecodeError is happening here:
http://mxr.mozilla.org/mozilla-central/source/testing/mozbase/mozprocess/mozprocess/winprocess.py#142

Changing that line to:
values = ["%s=%s" % (key.decode('utf8', 'replace'), value.decode('utf8', 'replace'))

Will probably fix your issue. That being said, that line might not be the best place to do the decoding. Ideally we'd decode wherever those strings are first input into the program. Could you let me know if that change works and if so I'll look into making a proper patch? Thanks for finding this!

Comment 10

[Masatoshi Kimura [:emk]]

I was trying to build SeaMonkey because I would have to build the tree to run "mozmake mochitest-browser-chrome". If someone else will work on this bug, I will not investigate further.

(In reply to Andrew Halberstadt [:ahal] from comment #9)
> Changing that line to:
> values = ["%s=%s" % (key.decode('utf8', 'replace'), value.decode('utf8',
> 'replace'))
> 
> Will probably fix your issue.

I'm not sure it will decode the environment string correctly on Windows. Windows ANSI codepage is not UTF-8.

Comment 11

[Masatoshi Kimura [:emk]]

Attachment: Decode string when building an environment block to support non-ASCII environment variables

I was finally able to build the tree with MOZ_OBJDIR completely removed!

This patch will decode strings using the 'mbcs' encoding which means the ANSI codepage on Windows. This file is Windows-specific anyway as the name implies.

Comment 12

[Andrew Halberstadt [:ahal]]

Attachment: mozprocess_windows_decode

Thanks, that patch looks workable, but I'd prefer to do the decoding a little earlier on to avoid this type of problem again in the future. Does this patch work for you?

Comment 13

[Masatoshi Kimura [:emk]]

(In reply to Andrew Halberstadt [:ahal] from comment #12)
> Created attachment 8482879 [details] [diff] [review]
> mozprocess_windows_decode
> 
> Thanks, that patch looks workable, but I'd prefer to do the decoding a
> little earlier on to avoid this type of problem again in the future. Does
> this patch work for you?

It worked for me, but I'd rather use sys.getfilesystemencoding() rather than the hard-coded encoding names.
It will give us a consistent result for all platforms.

Comment 14

[Frank Wein [:mcsmurf]]

Thanks for the patch and sorry for the incredible late feedback (real life :), attachment 8482879 [details] [diff] [review] works fine and fixes the problem.

Comment 15

[Philip Chee]

What needs to be done to get this checked in to the tree?

Comment 16

[Andrew Halberstadt [:ahal]]

I haven't tried, but the patch is likely bit-rotted by now. Then it needs to be reviewed, and I'd also like to see a passing try run with the syntax:
try: -a

Comment 17

[Masatoshi Kimura [:emk]]

Please attach a new patch with comments addressed.

Comment 18

[Philip Chee]

According to Google sys.getfilesystemencoding() can return None. On *nix it can return anything e.g. "ANSI_X3.4-1968"

Comment 19

[Philip Chee]

Attachment: v1.patch using sys.getfilesystemencoding()

Is this the right approach? I've never used Python before.

(In reply to Andrew Halberstadt [:ahal] from comment #16)
> I haven't tried, but the patch is likely bit-rotted by now. Then it needs to
> be reviewed, and I'd also like to see a passing try run with the syntax:
Who would be a suitable reviewer?

> try: -a
The try server doesn't do anything when I do try: -a

This try server job uses: "try: -b do -p all -u all -t none"
https://treeherder.mozilla.org/#/jobs?repo=try&revision=10c84ba38901

Comment 20

[Andrew Halberstadt [:ahal]]

Comment on attachment 8587430 [details] [diff] [review]
v1.patch using sys.getfilesystemencoding()

Review of attachment 8587430 [details] [diff] [review]:
-----------------------------------------------------------------

Close, but if sys.getfilesystemencoding() returns None, then that will fail. I'd change it to:
sys.getfilesystemencoding() or 'utf-8'.

Comment 21

[Philip Chee]

Attachment: v2.patch using sys.getfilesystemencoding() or utf-8

try-server results:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=86029fe3ce2a

> Close, but if sys.getfilesystemencoding() returns None, then that will fail.
> I'd change it to:
> sys.getfilesystemencoding() or 'utf-8'.
Done.

Comment 22

[Andrew Halberstadt [:ahal]]

Comment on attachment 8588070 [details] [diff] [review]
v2.patch using sys.getfilesystemencoding() or utf-8

Review of attachment 8588070 [details] [diff] [review]:
-----------------------------------------------------------------

Encoding changes always scare me, but I guess this looks good assuming your try run was all green.

Comment 23

[Philip Chee]

> Encoding changes always scare me, but I guess this looks good assuming your
> try run was all green.
Thanks. Managed to get a all green try run:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=b7e8d67b2e37

http://hg.mozilla.org/integration/mozilla-inbound/rev/acb01dce9ffe

Comment 24

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/acb01dce9ffe

Comment 25

[Frank Wein [:mcsmurf]]

Comment on attachment 8588070 [details] [diff] [review]
v2.patch using sys.getfilesystemencoding() or utf-8

Yeah, it works! Thanks to everyone being involved here.