1800171 - non-ASCII characters of hg outputs are not displayed correctly with MozillaBuild 4.0.2

Status: RESOLVED FIXED

(Firefox Build System :: MozillaBuild, defect)

Description

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)(away 4/27 - 5/6)]

In MozillaBuild 4.0.2 (I use the terminal in vscode), hg outputs gets broken if it contains Japanese characters like:

$ hg --version
Mercurial - 蛻・淵讒区・邂｡逅・ヤ繝ｼ繝ｫ(繝舌・繧ｸ繝ｧ繝ｳ 6.2.3)
(隧ｳ邏ｰ縺ｯ https://mercurial-scm.org 繧貞盾辣ｧ)

Copyright (C) 2005-2022 Olivia Mackall and others
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

However, I got result of hg log -r central correctly:

$ hg log -r central
リビジョン:   742345:963d3b67f524
fxtree:      central
ユーザ:       Tim Giles <[e-mail]>
日付:         Thu Nov 10 14:43:54 2022 +0000
要約:         Bug 1797509 - Add help text for determining harness specific arguments. r=ahal

I don't know why these command gets different result.

Comment 1

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)(away 4/27 - 5/6)]

I got same result in "Windows Terminal".

Comment 2

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)(away 4/27 - 5/6)]

IIRC, before MozillaBuild 4.0.2, Mercurial was not localized on Windows.

Comment 4

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

Does the problem go away on MozillaBuild 4.0.1?
What happens if you downgrade your version of Mercurial in MozillaBuild 4.0.2? (python3 -m pip install mercurial==6.1)
Can you also try changing the registry value at HKCU\Console\VirtualTerminalLevel (delete it, or set it to 0)?

Glandium and I are seeing different results, so I'm just trying to get more info on the behavior you're seeing.

Comment 5

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)(away 4/27 - 5/6)]

Thank you for the investigation. Oddly, I cannot reproduce this today (before testing what you said, I checked current behavior). I rebooted my system yesterday, it could've "fixed" this issue.

Sorry for the bug spams.

-> WFM

Comment 6

[Mike Hommey [:glandium]]

Is it displaying English or correct Japanese now?

Comment 7

[Brian Birtles (:birtles, Feb 26~ parental leave)]

I still reproduces for me today:

$ hg --version
Mercurial - 蛻・淵讒区・邂｡逅・ヤ繝ｼ繝ｫ(繝舌・繧ｸ繝ｧ繝ｳ 6.2.3)
(隧ｳ邏ｰ縺ｯ https://mercurial-scm.org 繧貞盾辣ｧ)

Copyright (C) 2005-2022 Olivia Mackall and others
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Comment 8

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)(away 4/27 - 5/6)]

(In reply to Mike Hommey [:glandium] from comment #6)

Is it displaying English or correct Japanese now?

Correct Japanese text now.

Comment 9

[Mike Hommey [:glandium]]

did you switch your console to the utf-8 codepage?

I'm reopening, BTW, because I can still reproduce, and so does Brian. As a workaround, you can remove hg.exe in $mozilla-build/python3/Scripts, but please note that has other downsides. Another workaround is to remove the locales from $mozilla-build/python3/lib/site-packages/mercurial/locale

Comment 10

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)(away 4/27 - 5/6)]

No, I did nothing special at least for settings of python, mercurial, etc which are used for developing m-c.

Comment 11

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)(away 4/27 - 5/6)]

Oh, restarting vscode causes this bug back...

Comment 12

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

Can you try deleting the locale folder at $mozilla-build/python3/lib/site-packages/mercurial/locale and seeing if that resolves it? It looks like pip didn't provide the locale folder for mercurial before 6.2.3. It does look like there's a bad interaction between MSYS2 and the locales, but that might have always been there, and I'm not too keen on digging into what the cause is at this time (that definitely sound like a rabbit hole). If removing the locale folder 'fixes' the problem, we're essentially just going back to what previous versions of MozillaBuild provided. Which means we can just remove the locale during packageit.py when creating the MozillaBuild installer, and that should prevent this problem in future releases.

Ideally we do find and fix the root cause eventually, but I think for now that's the pragmatic solution to this.

Comment 13

[Masayuki Nakano [:masayuki] (he/him)(JST, +0900)(away 4/27 - 5/6)]

Right. Renaming locale makes the output written in English. I think that it's no problem to make Mercurial work only with English because English ability is required in the further process like review in Phabricator.

Comment 14

[Alex Hochheiden [:ahochheiden] (needinfo? me)]

Attachment: Bug 1800171 - Remove 'locale' directory after pip install of `mercurial` to workaround an `MSYS2` issue r?#build

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/mozilla-build/rev/6ccb9ec541fb
Remove 'locale' directory after pip install of mercurial to workaround an MSYS2 issue r=firefox-build-system-reviewers,glandium

Comment 16

[Masatoshi Kimura [:emk]]

When will this fix be released? I hit this bug when I bought a new machine. Renaming locale worked around the problem.