Open Bug 1223325 Opened 9 years ago Updated 2 years ago

Windows build failed because of windows user name containing non-ASCII characters

Categories

(Firefox Build System :: General, defect)

Product:
Firefox Build System
Component:
General
Platform:
x86_64
Windows 7
Type:
defect

Tracking

(Not tracked)

Status:
UNCONFIRMED

People

(Reporter: izh1979, Unassigned)

Details

User Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:41.0) Gecko/20100101 Firefox/41.0 SeaMonkey/2.38
Build ID: 20150925151141

Steps to reproduce:

Hi.

I'm trying to build SeaMonkey on Windows-7/64 using msvc2013.
In the latest release it fails with Unicode-related errors because my windows' user name consists of non-English letters (and I can't change it). One release ago it was fine.

I suppose, this bug is similar to https://bugzilla.mozilla.org/show_bug.cgi?id=1152428


Actual results:

    $ mozmake -f client.mk
    client.mk:205: n:/projects/mozilla/obj-comm-release-x64/.mozconfig.mk: No such file or directory
    mkdir -p 'n:/projects/mozilla/obj-comm-release-x64/'
    > n:/projects/mozilla/obj-comm-release-x64/.mozconfig.mk
    Adding client.mk options from n:/projects/mozilla/comm-release/.mozconfig:
        MOZ_OBJDIR=n:/projects/mozilla/obj-comm-release-x64
        MOZ_MAKE_FLAGS=-j4
        CONFIG_GUESS=x86_64-pc-mingw32
        FOUND_MOZCONFIG := n:/projects/mozilla/comm-release/.mozconfig
    mozmake[1]: Entering directory 'n:/projects/mozilla/comm-release'
    Generating n:/projects/mozilla/comm-release/configure using autoconf
    cd n:/projects/mozilla/comm-release; /usr/local/bin/autoconf-2.13
    Generating n:/projects/mozilla/comm-release/mozilla/configure using autoconf
    cd n:/projects/mozilla/comm-release/mozilla; /usr/local/bin/autoconf-2.13
    Generating n:/projects/mozilla/comm-release/mozilla/js/src/configure using autoconf
    cd n:/projects/mozilla/comm-release/mozilla/js/src; /usr/local/bin/autoconf-2.13
    cd n:/projects/mozilla/obj-comm-release-x64
    n:/projects/mozilla/comm-release/configure
    creating cache ./config.cache
    n:/projects/mozilla/comm-release/mozilla/configure: line 88: Error: command not found
    n:/projects/mozilla/comm-release/mozilla/configure: line 90: [environment,: command not found
    n:/projects/mozilla/comm-release/mozilla/configure: line 92: The: command not found
    n:/projects/mozilla/comm-release/mozilla/configure: line 93: a: command not found
    n:/projects/mozilla/comm-release/mozilla/configure: line 95: You: command not found
    n:/projects/mozilla/comm-release/mozilla/configure: line 97: If: command not found
    n:/projects/mozilla/comm-release/mozilla/configure: line 98: message.: command not found
    n:/projects/mozilla/comm-release/mozilla/configure: line 100: The: command not found
    n:/projects/mozilla/comm-release/mozilla/configure: eval: line 107: unexpected EOF while looking for matching `''
    n:/projects/mozilla/comm-release/mozilla/configure: eval: line 110: syntax error: unexpected end of file
    Failed eval'ing the following:
    Error running mach:
     
        ['environment', '--format=configure']
     
    The error occurred in code that was called by the mach command. This is either
    a bug in the called code itself or in the way that mach is calling it.
     
    You should consider filing a bug for this issue.
     
    If filing a bug, please include the full output of mach, including this error
    message.
     
    The details of the failure are as follows:
     
    UnicodeDecodeError: 'ascii' codec can't decode byte 0xfd in position 10: ordinal not in range(128)
     
      File "n:/projects/mozilla/comm-release/mozilla/.\build/mach_bootstrap.py", line 265, in pre_dispatch_handler
        state_dir = get_state_dir()[0]
      File "n:/projects/mozilla/comm-release/mozilla/.\build/mach_bootstrap.py", line 201, in get_state_dir
        state_user_dir = os.path.expanduser('~/.mozbuild')
      File "c:\mozilla-build\python\lib\ntpath.py", line 311, in expanduser
        return userhome + path[i:]
    *** Fix above errors and then restart with\
                   "c:/mozilla-build/mozmake/mozmake -f client.mk build"
    n:/projects/mozilla/comm-release/client.mk:361: recipe for target 'configure' failed
    mozmake[1]: *** [configure] Error 1
    mozmake[1]: Leaving directory 'n:/projects/mozilla/comm-release'
    client.mk:375: recipe for target 'n:/projects/mozilla/obj-comm-release-x64/Makefile' failed
    mozmake: *** [n:/projects/mozilla/obj-comm-release-x64/Makefile] Error 2 


Expected results:

SeaMonkey should be built.
OS: Unspecified → Windows 7
Hardware: Unspecified → x86_64
I'm trying to build cloning sources from http://hg.mozilla.org/releases/comm-release/
Is this still a problem with current VS2015 comm-esr52 releases and mozbuild 2.2?
Flags: needinfo?(izh1979)
(In reply to Frank-Rainer Grahl (:frg) from comment #2)
> Is this still a problem with current VS2015 comm-esr52 releases and mozbuild
> 2.2?

Yes. I've installed 2.2 version of mozbuild (without 3.x) and got following:

---8<-------------------------------------------------------------------------------
?¤¬Ё-Ёбва вRа@QWERTY2 /n/seamonkey/comm-esr52/mozilla
$ mach bootstrap
Error running mach:

    ['bootstrap']

The error occurred in mach itself. This is likely a bug in mach itself or a
fundamental problem with a loaded module.

Please consider filing a bug against mach by going to the URL:

    https://bugzilla.mozilla.org/enter_bug.cgi?product=Core&component=mach


If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 7: ordinal not in range(128)

  File "n:\seamonkey\comm-esr52\mozilla\python/mach\mach\main.py", line 352, in run
    return self._run(argv)
  File "n:\seamonkey\comm-esr52\mozilla\python/mach\mach\main.py", line 386, in _run
    self.load_settings(self.settings_paths)
  File "n:\seamonkey\comm-esr52\mozilla\python/mach\mach\main.py", line 544, in load_settings
    files = map(find_in_dir, self.settings_paths)
  File "n:\seamonkey\comm-esr52\mozilla\python/mach\mach\main.py", line 540, in find_in_dir
    path = os.path.join(base, name)
  File "d:\mozilla-build\python\lib\ntpath.py", line 85, in join
    result_path = result_path + p_path
---8<-------------------------------------------------------------------------------

The garbage before @QWERTY2 in the command prompt corresponding to the word "Администратор" (Administrator -- default windows admin user) in wrong encoding. The console on Windows is typically in DOS (CP866) encoding while apps outputs strings typically in Windows (CP1251) encoding, or vice-verse. But anyway, with proper escaping these build scripts should handle any sequence of characters.
Flags: needinfo?(izh1979)
So the problem is that the user name in Windows is not in UTF-8, nor in some multibyte encoding, but actually in CP866 or CP1251 where non-English letters are placed in the upper half of code table (i.e with the values in the range 128-255).
The problem is that self.settings_paths array consisting of:
['c:/Users/\x80\xa4\xac\xa8\xad\xa8\xe1\xe2\xe0\xa0\xe2\xae\xe0/.mozbuild', 'n:\\seamonkey\\comm-esr52\\mozilla']
So the function cannot handle \x## hexadecimal escapes.
The hexadecimal string is for "Администратор" in single-byte CP866 encoding: https://en.wikipedia.org/wiki/Code_page_866
Could you try with a comm-beta build and VS2017. This is likely a backend problem and I think I saw some fixes there. The resulting SeaMonkey build is mostly unusable but it should compile fine.
With comm-beta, VS2017 and mozilla-build 3.1.1 the error stays the same.
I suppose it needs to convert paths from local OEM encoding to UTF8 prior to operating with them.
According to the reporter also occurs trying to build Fx 60 so probably a general build problem.
Product: SeaMonkey → Core
We don't support building in directories with pretty much non-ascii names (sadly).

If you aren't getting an obvious error stating that, please give us steps to reproduce so we can fix that.

(I'm not sure if comment #0 is still accurate because it is ~2 years old.)
(In reply to Gregory Szorc [:gps] from comment #12)
> We don't support building in directories with pretty much non-ascii names
> (sadly).
> 
> If you aren't getting an obvious error stating that, please give us steps to
> reproduce so we can fix that.
> 
> (I'm not sure if comment #0 is still accurate because it is ~2 years old.)

Yes, the comment #0 is outdated. But the comment #3 is still actual.

The problem is that Mozilla build environment tries for some reason to use user profile directories. The build directory itself doesn't contain any non-ASCII characters. It's n:\seamonkey\ for sources and d:\mozilla-build\ for build scripts. The problem is with user profile directory because of Russian windows user name. Windows doesn't impose ASCII restrictions on user names and it's a frequently situation to use Russian user names.

It would be good if i would be possible to use some command-line option or config file parameter to prevent build scripts from looking in user profile directories. What's the purpose of using it? I would prefer to specify some separate folder explicitly if needed.
The self.settings_paths in N:\seamonkey\comm-beta\mozilla\python\mach\mach\main.py is
['c:/Users/\x80\xa4\xac\xa8\xad\xa8\xe1\xe2\xe0\xa0\xe2\xae\xe0/.mozbuild', 'n:\\seamonkey\\comm-beta\\mozilla']
That's why self.load_settings(self.settings_paths) fails.

If it is possible to override it with specifying some different directory for keeping settings, it would help.
(Or it should convert Windows user name to UTF-8 from local encoding prior to constructing a path.)
By adding the lines below to start-shell-msvc2017-x64.bat I could make it go further:
SET USERNAME=IZh
SET HOME=/d/mozilla-build/home/%USERNAME%
SET USERPROFILE=/d/mozilla-build/home/%USERNAME%
Product: Core → Firefox Build System
Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.