1797573 - Mots fails to decode the "Á" in "Emilio Cobos Álvarez"

Status: RESOLVED FIXED

(Conduit :: mots, defect, P2)

Description

[Chris Martin [:cmartin] - OOO until Nov 9]

As mentioned in Issue 35, mots is failing to decode the "Á" in "Emilio Cobos Álvarez" on Windows with Python 3.10.4.

I checked with a hex editor, and it is encoded correctly as C3 81, which corresponds to code point U+0xC1 "Latin Capital Letter a with Acute". So somehow the script just isn't handling it properly.

Here is the error dump:

$ mots module ls
2022-10-26 13:09:08,501 cli        ERROR    'charmap' codec can't decode byte 0x81 in position 5508: character maps to <undefined>
Traceback (most recent call last):
  File "C:\Users\cmartin\AppData\Local\Programs\Python\Python310\lib\site-packages\mots\cli.py", line 255, in main
    args.func(args)
  File "C:\Users\cmartin\AppData\Local\Programs\Python\Python310\lib\site-packages\mots\cli.py", line 39, in ls
    file_config.load()
  File "C:\Users\cmartin\AppData\Local\Programs\Python\Python310\lib\site-packages\mots\config.py", line 70, in load
    self.config = yaml.load(f)
  File "C:\Users\cmartin\AppData\Local\Programs\Python\Python310\lib\site-packages\ruamel\yaml\main.py", line 432, in load
    constructor, parser = self.get_constructor_parser(stream)
  File "C:\Users\cmartin\AppData\Local\Programs\Python\Python310\lib\site-packages\ruamel\yaml\main.py", line 483, in get_constructor_parser
    self.reader.stream = stream
  File "C:\Users\cmartin\AppData\Local\Programs\Python\Python310\lib\site-packages\ruamel\yaml\reader.py", line 133, in stream
    self.determine_encoding()
  File "C:\Users\cmartin\AppData\Local\Programs\Python\Python310\lib\site-packages\ruamel\yaml\reader.py", line 193, in determine_encoding
    self.update_raw()
  File "C:\Users\cmartin\AppData\Local\Programs\Python\Python310\lib\site-packages\ruamel\yaml\reader.py", line 288, in update_raw
    data = self.stream.read(size)
  File "C:\Users\cmartin\AppData\Local\Programs\Python\Python310\lib\encodings\cp1252.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x81 in position 5508: character maps to <undefined>

Comment 1

[Chris Martin [:cmartin] - OOO until Nov 9]

Just from the stack dump, it looks like it chose the old "cp1252" as the code page, so it's trying to decode it as Code Page 1252 instead of UTF-8. I'm guessing that somewhere this "ruamel-yaml" library needs to be told what encoding to use so it doesn't try to guess (incorrectly)

Comment 2

[Zeid [:zeid]]

:cmartin -- thanks for the bug report. Since this is happening when loading the config, I assume it will impact most if not all commands, not just the one you mentioned. Will look into this.

Comment 3

[Chris Martin [:cmartin] - OOO until Nov 9]

Ah... It looks like the change you need to make is to add encoding="utf-8" to mots/config.py line 69

Comment 4

[Zeid [:zeid]]

(In reply to Chris Martin [:cmartin] from comment #3)

Ah... It looks like the change you need to make is to add encoding="utf-8" to mots/config.py line 69

You are welcome to create a pull request! I would like to finish adding the Windows tests and reproduce this in a bug before we merge the change though.

Comment 5

[Chris Martin [:cmartin] - OOO until Nov 9]

Fair enough! Done

Comment 6

[Zeid [:zeid]]

Attachment: GitHub Pull Request