Closed Bug 1050561 Opened 6 years ago Closed 5 years ago

UnicodeDecodeError when running mochitest (SeaMonkey/Thunderbird only)

Categories

(Testing :: Mozbase, defect)

x86_64
Windows 7
defect
Not set

Tracking

(firefox40 fixed)

RESOLVED FIXED
mozilla40
Tracking Status
firefox40 --- fixed

People

(Reporter: mcsmurf, Assigned: philip.chee)

References

Details

Attachments

(3 files, 1 obsolete file)

This is likely a SeaMonkey/Thunderbird bug only and it may go away when merging c-c with m-c.
Currently one cannot various mochitest suites, e.g. "mozmake mochitest-browser-chrome" for SeaMonkey because of this error:
UnicodeDecodeError: 'ascii' codec can't decode byte 0xad in position 837: ordinal not in range(128)
in testing\mozbase\mozprocess\mozprocess\winprocess.py (probably line 154 or near there, I added some debug code).

This happens because the test environment (env variables) includes this one from configure.in:
'NONASCII=\xad\xad'
(see http://hg.mozilla.org/mozilla-central/annotate/2f198e81ed98//configure.in#l7792)

The code in winprocess.py tries to decode the whole byte string to ascii when appending the unicode string
u'LD_LIBRARY_PATH=f:\\mozilla\\tree-hg\\comm-central\\obj-i686-pc-mingw32\\mozilla\\dist\\bin'

This fails as \xad is not part of ascii standard. Not sure if there is a simple workaround or fix for this problem.
Component: Build Config → Mozbase
Product: Core → Testing
QA Contact: hskupin
Blocks: 948405
It means mochitest for SeaMonkey will fail on Japanese Windows.
I was unable to build SeaMonkey...

 0:32.81 Embedding manifest from h:/m/comm-central/mail/app/thunderbird.exe.mani
fest
 0:32.82
 0:32.82 h:/m/comm-central/mozilla/mail/app/thunderbird.exe.manifest : general e
rror c1010070: Failed to load and parse the manifest. ????????????????
 0:32.82
 0:32.83 h:/m/comm-central/config/rules.mk:703: recipe for target 'thunderbird.e
xe' failed

Why Thunderbird is being built when building SeaMonkey???

My .mozconfig is:
ac_add_options --enable-application=suite
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/../objdir-sm-release
ac_add_options --enable-winsdk-directx
That's Bug 1040009
Still fails with patches from bug 1040009:

$ ./mozilla/mach build
 0:00.28 c:\mozilla-build\mozmake\mozmake.EXE -f client.mk -s
 0:01.04 Adding client.mk options from h:/m/comm-central/.mozconfig:
 0:01.04     MOZ_OBJDIR=objdir-sm-release
 0:01.04     FOUND_MOZCONFIG := h:/m/comm-central/.mozconfig
 0:01.94 cd objdir-sm-release
 0:01.95 h:/m/comm-central/configure
 0:02.16 loading cache ./config.cache
 0:02.78 h:/m/comm-central/mozilla/configure: line 84: Ambiguous: command not fo
und
 0:02.78 Failed eval'ing the following:
 0:03.22 Ambiguous object directory detected. We detected that both h:\m\comm-ce
ntral\objdir-sm-release and h:\m\comm-central\mozilla\objdir-sm-release could be
 object directories. This is typically caused by having a mozconfig pointing to
a different object directory from the current working directory. To solve this p
roblem, ensure you do not have a default mozconfig in searched paths.
 0:03.24 *** Fix above errors and then restart with\
 0:03.24                "c:/mozilla-build/mozmake/mozmake.EXE -f client.mk build
"
 0:03.25 h:/m/comm-central/client.mk:349: recipe for target 'configure' failed
 0:03.25 mozmake.EXE[1]: *** [configure] Error 1
 0:03.25 client.mk:363: recipe for target 'objdir-sm-release/Makefile' failed
 0:03.25 mozmake.EXE: *** [objdir-sm-release/Makefile] Error 2
 0:03.25 0 compiler warnings present.
2

I also changed .mozconfig as follows, still no luck:
ac_add_options --enable-application=suite
mk_add_options MOZ_OBJDIR=objdir-sm-release
ac_add_options --enable-winsdk-directx
The build system insists on finding manifest from $SRCDIR/mozilla/suite/app rather than $SRCDIR/suite/app:

 0:20.46 Embedding manifest from h:/m/comm-central/suite/app/seamonkey.exe.manif
est
 0:20.49
 0:20.49 h:/m/comm-central/mozilla/suite/app/seamonkey.exe.manifest : general er
ror c1010070: Failed to load and parse the manifest. ????????????????
Even tbpl keeps on failing to build SeaMonkey since 22 July.
http://ftp.mozilla.org/pub/mozilla.org/seamonkey/nightly/latest-comm-central-trunk/
http://ftp.mozilla.org/pub/mozilla.org/seamonkey/nightly/2014-08-12-00-30-04-comm-central-trunk/

I have to conclude that SeaMonkey is virtually dead because no new contributor can build the tree.
BTW: This bug is not for sorting out build problems that are being worked on.
Still fails even with bug 1040009 patches:

 3:35.08 creating ./config.status
 3:35.65 Traceback (most recent call last):
 3:35.65   File "./config.status", line 924, in <module>
 3:35.65     config_status(**args)
 3:35.65   File "h:\m\comm-central\mozilla\python\mozbuild\mozbuild\config_statu
s.py", line 134, in config_status
 3:35.65     emitter = TreeMetadataEmitter(env)
 3:35.65   File "h:\m\comm-central\mozilla\python\mozbuild\mozbuild\frontend\emi
tter.py", line 84, in __init__
 3:35.65     mozinfo.find_and_update_from_json(config.topobjdir)
 3:35.65   File "h:\m\comm-central\mozilla\testing\mozbase\mozinfo\mozinfo\mozin
fo.py", line 153, in find_and_update_from_json
 3:35.65     build = MozbuildObject.from_environment()
 3:35.66   File "h:\m\comm-central\mozilla\python\mozbuild\mozbuild\base.py", li
ne 181, in from_environment
 3:35.66     raise ObjdirMismatchException(topobjdir, _config_topobjdir)
 3:35.66 mozbuild.base.ObjdirMismatchException: Objdir mismatch: h:\m\comm-centr
al\objdir-sm-release != h:\m\comm-central\mozilla\objdir-sm-release
 3:35.67 *** Fix above errors and then restart with\
 3:35.67                "c:/mozilla-build/mozmake/mozmake.EXE -f client.mk build
"
 3:35.68 h:/m/comm-central/client.mk:361: recipe for target 'configure' failed
 3:35.68 mozmake.EXE[1]: *** [configure] Error 1
 3:35.68 client.mk:375: recipe for target 'h:/m/comm-central/objdir-sm-release/M
akefile' failed
 3:35.69 mozmake.EXE: *** [h:/m/comm-central/objdir-sm-release/Makefile] Error 2

 3:35.70 0 compiler warnings present.
2
Hi Masatoshi, this bug is about a UnicodeDecodeError in mozprocess. If you believe you have build issues that are not covered by bug 1040009, please file a new bug under 'Core/Build Config'.

Frank, I'd guess that the UnicodeDecodeError is happening here:
http://mxr.mozilla.org/mozilla-central/source/testing/mozbase/mozprocess/mozprocess/winprocess.py#142

Changing that line to:
values = ["%s=%s" % (key.decode('utf8', 'replace'), value.decode('utf8', 'replace'))

Will probably fix your issue. That being said, that line might not be the best place to do the decoding. Ideally we'd decode wherever those strings are first input into the program. Could you let me know if that change works and if so I'll look into making a proper patch? Thanks for finding this!
Flags: needinfo?(bugzilla)
I was trying to build SeaMonkey because I would have to build the tree to run "mozmake mochitest-browser-chrome". If someone else will work on this bug, I will not investigate further.

(In reply to Andrew Halberstadt [:ahal] from comment #9)
> Changing that line to:
> values = ["%s=%s" % (key.decode('utf8', 'replace'), value.decode('utf8',
> 'replace'))
> 
> Will probably fix your issue.

I'm not sure it will decode the environment string correctly on Windows. Windows ANSI codepage is not UTF-8.
I was finally able to build the tree with MOZ_OBJDIR completely removed!

This patch will decode strings using the 'mbcs' encoding which means the ANSI codepage on Windows. This file is Windows-specific anyway as the name implies.
Assignee: nobody → VYV03354
Status: UNCONFIRMED → ASSIGNED
Ever confirmed: true
Attachment #8481752 - Flags: review?(ahalberstadt)
Thanks, that patch looks workable, but I'd prefer to do the decoding a little earlier on to avoid this type of problem again in the future. Does this patch work for you?
Flags: needinfo?(VYV03354)
Attachment #8481752 - Flags: review?(ahalberstadt)
(In reply to Andrew Halberstadt [:ahal] from comment #12)
> Created attachment 8482879 [details] [diff] [review]
> mozprocess_windows_decode
> 
> Thanks, that patch looks workable, but I'd prefer to do the decoding a
> little earlier on to avoid this type of problem again in the future. Does
> this patch work for you?

It worked for me, but I'd rather use sys.getfilesystemencoding() rather than the hard-coded encoding names.
It will give us a consistent result for all platforms.
Flags: needinfo?(VYV03354)
Thanks for the patch and sorry for the incredible late feedback (real life :), attachment 8482879 [details] [diff] [review] works fine and fixes the problem.
Flags: needinfo?(bugzilla)
What needs to be done to get this checked in to the tree?
Flags: needinfo?(ahalberstadt)
Flags: needinfo?(VYV03354)
I haven't tried, but the patch is likely bit-rotted by now. Then it needs to be reviewed, and I'd also like to see a passing try run with the syntax:
try: -a
Flags: needinfo?(ahalberstadt)
Please attach a new patch with comments addressed.
Flags: needinfo?(VYV03354)
According to Google sys.getfilesystemencoding() can return None. On *nix it can return anything e.g. "ANSI_X3.4-1968"
Is this the right approach? I've never used Python before.

(In reply to Andrew Halberstadt [:ahal] from comment #16)
> I haven't tried, but the patch is likely bit-rotted by now. Then it needs to
> be reviewed, and I'd also like to see a passing try run with the syntax:
Who would be a suitable reviewer?

> try: -a
The try server doesn't do anything when I do try: -a

This try server job uses: "try: -b do -p all -u all -t none"
https://treeherder.mozilla.org/#/jobs?repo=try&revision=10c84ba38901
Attachment #8587430 - Flags: feedback?(bugzilla)
Attachment #8587430 - Flags: feedback?(ahalberstadt)
Comment on attachment 8587430 [details] [diff] [review]
v1.patch using sys.getfilesystemencoding()

Review of attachment 8587430 [details] [diff] [review]:
-----------------------------------------------------------------

Close, but if sys.getfilesystemencoding() returns None, then that will fail. I'd change it to:
sys.getfilesystemencoding() or 'utf-8'.
Attachment #8587430 - Flags: feedback?(ahalberstadt)
try-server results:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=86029fe3ce2a

> Close, but if sys.getfilesystemencoding() returns None, then that will fail.
> I'd change it to:
> sys.getfilesystemencoding() or 'utf-8'.
Done.
Assignee: VYV03354 → philip.chee
Attachment #8587430 - Attachment is obsolete: true
Attachment #8587430 - Flags: feedback?(bugzilla)
Attachment #8588070 - Flags: review?(ahalberstadt)
Attachment #8588070 - Flags: feedback?(bugzilla)
Comment on attachment 8588070 [details] [diff] [review]
v2.patch using sys.getfilesystemencoding() or utf-8

Review of attachment 8588070 [details] [diff] [review]:
-----------------------------------------------------------------

Encoding changes always scare me, but I guess this looks good assuming your try run was all green.
Attachment #8588070 - Flags: review?(ahalberstadt) → review+
> Encoding changes always scare me, but I guess this looks good assuming your
> try run was all green.
Thanks. Managed to get a all green try run:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=b7e8d67b2e37

http://hg.mozilla.org/integration/mozilla-inbound/rev/acb01dce9ffe
https://hg.mozilla.org/mozilla-central/rev/acb01dce9ffe
Status: ASSIGNED → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla40
Comment on attachment 8588070 [details] [diff] [review]
v2.patch using sys.getfilesystemencoding() or utf-8

Yeah, it works! Thanks to everyone being involved here.
Attachment #8588070 - Flags: feedback?(bugzilla) → feedback+
You need to log in before you can comment on or make changes to this bug.