1785976 - Building fail in korean Windows 10

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Taejin Jang]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/104.0.0.0 Safari/537.36

Steps to reproduce:

Windows 10 Korean
./mach build

Actual results:

 1:15.15 Traceback (most recent call last):
 1:15.15   File "c:\mozilla-source\mozilla-source\third_party\python\pyyaml\lib3\yaml\reader.py", line 89, in peek
 1:15.15     return self.buffer[self.pointer+index]
 1:15.15 IndexError: string index out of range
 1:15.15 During handling of the above exception, another exception occurred:
 1:15.15 Traceback (most recent call last):
 1:15.15   File "C:\mozilla-build\python3\lib\runpy.py", line 197, in _run_module_as_main
 1:15.15     return _run_code(code, main_globals, None,
 1:15.15   File "C:\mozilla-build\python3\lib\runpy.py", line 87, in _run_code
 1:15.15     exec(code, run_globals)
 1:15.15   File "c:\mozilla-source\mozilla-source\python\mozbuild\mozbuild\action\file_generate.py", line 156, in <module>
 1:15.15     sys.exit(log_build_task(main, sys.argv[1:]))
 1:15.15   File "c:\mozilla-source\mozilla-source\python\mozbuild\mozbuild\action\util.py", line 18, in log_build_task
 1:15.16     return f(*args, **kwargs)
 1:15.16   File "c:\mozilla-source\mozilla-source\python\mozbuild\mozbuild\action\file_generate.py", line 100, in main
 1:15.16     ret = module.__dict__[method](
 1:15.16   File "C:/mozilla-source/mozilla-source/toolkit/components/nimbus/generate/generate_feature_manifest.py", line 96, in generate_platform_feature_manifest
 1:15.16     data = yaml.safe_load(yaml_input)
 1:15.16   File "c:\mozilla-source\mozilla-source\third_party\python\pyyaml\lib3\yaml\__init__.py", line 162, in safe_load
 1:15.16     return load(stream, SafeLoader)
 1:15.16   File "c:\mozilla-source\mozilla-source\third_party\python\pyyaml\lib3\yaml\__init__.py", line 114, in load
 1:15.16     return loader.get_single_data()
 1:15.16   File "c:\mozilla-source\mozilla-source\third_party\python\pyyaml\lib3\yaml\constructor.py", line 49, in get_single_data
 1:15.16     node = self.get_single_node()
 1:15.16   File "c:\mozilla-source\mozilla-source\third_party\python\pyyaml\lib3\yaml\composer.py", line 36, in get_single_node
 1:15.16     document = self.compose_document()
 1:15.16   File "c:\mozilla-source\mozilla-source\third_party\python\pyyaml\lib3\yaml\composer.py", line 55, in compose_document
 1:15.16     node = self.compose_node(None, None)
 1:15.16   File "c:\mozilla-source\mozilla-source\third_party\python\pyyaml\lib3\yaml\composer.py", line 84, in compose_node
 1:15.16     node = self.compose_mapping_node(anchor)
 1:15.16   File "c:\mozilla-source\mozilla-source\third_party\python\pyyaml\lib3\yaml\composer.py", line 133, in compose_mapping_node
 1:15.16     item_value = self.compose_node(node, item_key)
 1:15.16   File "c:\mozilla-source\mozilla-source\third_party\python\pyyaml\lib3\yaml\composer.py", line 84, in compose_node
 1:15.16     node = self.compose_mapping_node(anchor)
 1:15.16   File "c:\mozilla-source\mozilla-source\third_party\python\pyyaml\lib3\yaml\composer.py", line 133, in compose_mapping_node
 1:15.16     item_value = self.compose_node(node, item_key)
 1:15.16   File "c:\mozilla-source\mozilla-source\third_party\python\pyyaml\lib3\yaml\composer.py", line 84, in compose_node
 1:15.16     node = self.compose_mapping_node(anchor)
 1:15.16   File "c:\mozilla-source\mozilla-source\third_party\python\pyyaml\lib3\yaml\composer.py", line 133, in compose_mapping_node
 1:15.16     item_value = self.compose_node(node, item_key)
 1:15.16   File "c:\mozilla-source\mozilla-source\third_party\python\pyyaml\lib3\yaml\composer.py", line 84, in compose_node
 1:15.16     node = self.compose_mapping_node(anchor)
 1:15.17   File "c:\mozilla-source\mozilla-source\third_party\python\pyyaml\lib3\yaml\composer.py", line 127, in compose_mapping_node
 1:15.17     while not self.check_event(MappingEndEvent):
 1:15.17   File "c:\mozilla-source\mozilla-source\third_party\python\pyyaml\lib3\yaml\parser.py", line 98, in check_event
 1:15.17     self.current_event = self.state()
 1:15.17   File "c:\mozilla-source\mozilla-source\third_party\python\pyyaml\lib3\yaml\parser.py", line 428, in parse_block_mapping_key
 1:15.17     if self.check_token(KeyToken):
 1:15.17   File "c:\mozilla-source\mozilla-source\third_party\python\pyyaml\lib3\yaml\scanner.py", line 116, in check_token
 1:15.17     self.fetch_more_tokens()
 1:15.17   File "c:\mozilla-source\mozilla-source\third_party\python\pyyaml\lib3\yaml\scanner.py", line 255, in fetch_more_tokens
 1:15.17     return self.fetch_plain()
 1:15.17   File "c:\mozilla-source\mozilla-source\third_party\python\pyyaml\lib3\yaml\scanner.py", line 679, in fetch_plain
 1:15.17     self.tokens.append(self.scan_plain())
 1:15.17   File "c:\mozilla-source\mozilla-source\third_party\python\pyyaml\lib3\yaml\scanner.py", line 1290, in scan_plain
 1:15.17     ch = self.peek(length)
 1:15.17   File "c:\mozilla-source\mozilla-source\third_party\python\pyyaml\lib3\yaml\reader.py", line 91, in peek
 1:15.17     self.update(index+1)
 1:15.17   File "c:\mozilla-source\mozilla-source\third_party\python\pyyaml\lib3\yaml\reader.py", line 153, in update
 1:15.17     self.update_raw()
 1:15.17   File "c:\mozilla-source\mozilla-source\third_party\python\pyyaml\lib3\yaml\reader.py", line 178, in update_raw
 1:15.17     data = self.stream.read(size)
 1:15.17 UnicodeDecodeError: 'cp949' codec can't decode byte 0xe2 in position 7348: illegal multibyte sequence
 1:15.20 mozmake[3]: *** [backend.mk:972: toolkit/components/nimbus/lib/.deps/NimbusFeatureManifest.h.stub] Error 1
 1:15.20 mozmake[3]: *** Waiting for unfinished jobs....
 1:22.69 mozmake[2]: *** [C:/mozilla-source/mozilla-source/config/recurse.mk:34: export] Error 2
 1:22.70 mozmake[1]: *** [C:/mozilla-source/mozilla-source/config/rules.mk:361: default] Error 2
 1:22.70 mozmake: *** [client.mk:63: build] Error 2
 1:22.72 1 compiler warnings present.

Expected results:

Build Successful

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The Bugbug bot thinks this bug should belong to the 'Firefox Build System::General' component, and is moving the bug to that component. Please correct in case you think the bot is wrong.

Comment 2

[Taejin Jang]

The building system is using python yaml library in generate_feature_manifest.py.
But, the input file's character set is utf-8, but korean windows system is EUC-KR or CP949.
So I think adding charset option is the resolving solution.

def generate_feature_manifest(fd, input_file):
    write_fm_js_headers(fd)
    nimbus_dir_path = pathlib.Path(input_file).parent
    try:
        with open(input_file, "r") as yaml_input:
            data = yaml.safe_load(yaml_input)
            with pathlib.Path(nimbus_dir_path, FEATURE_MANIFEST_SCHEMA).open() as f:
                schema = json.load(f)
                for feature_entry in data:
                    try:
                        jsonschema.validate(data[feature_entry], schema)
                    except Exception as e:
                        print("Error while validating FeatureManifest.yaml")
                        print("On key: {}".format(feature_entry))
                        print("Input file: {}".format(input_file))
                        raise e
            fd.write("const FeatureManifest = {};".format(json.dumps(data)))
    except (IOError) as e:
        print("{}: error:\n  {}\n".format(input_file, e))
        sys.exit(1)

Comment 3

[Taejin Jang]

Building was successed by adding encoding='utf-8' above open function.

https://searchfox.org/mozilla-central/source/toolkit/components/nimbus/generate/generate_feature_manifest.py#40,95

Comment 4

[Alex Hochheiden [:ahochheiden] [he/him]]

Attachment: Bug 1785976 - Make encoding explicitly `utf-8` to resolve an issue on Korean Windows r?barret

Comment 5

[Taejin Jang]

Line 95 is not changed yet.

Comment 6

[Alex Hochheiden [:ahochheiden] [he/him]]

Ah, good call. I didn't see the highlight since it was off-screen.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/99b382b3dd1e
Make encoding explicitly `utf-8` to resolve an issue on Korean Windows r=barret

Comment 8

[Marian-Vasile Laza]

https://hg.mozilla.org/mozilla-central/rev/99b382b3dd1e