1615223 - 'charmap' codec can't decode byte 0x88 in position 189: character maps to <undefined>

Status: RESOLVED FIXED

(Developer Infrastructure :: Source Documentation, defect)

Description

[Jan Honza Odvarko [:Honza] (always need-info? me)]

I am experiencing the following error when running mach doc (on Windows 10 machine)
I am running in on fresh m-c clone.

# Sphinx version: 2.2.2
# Python version: 3.6.5 (CPython)
# Docutils version: 0.15.2 release
# Jinja2 version: 2.10.3
# Last messages:

# Loaded extensions:
Traceback (most recent call last):
  File "c:\src\mozilla.org\mozilla-central\obj-x86_64-pc-mingw32\_virtualenvs\moztreedocs-OSceHKx-\Lib\site-packages\sphinx\cmd\build.py", line 275, in build_main
    args.tags, args.verbosity, args.jobs, args.keep_going)
  File "c:\src\mozilla.org\mozilla-central\obj-x86_64-pc-mingw32\_virtualenvs\moztreedocs-OSceHKx-\Lib\site-packages\sphinx\application.py", line 243, in __init__
    self.setup_extension(extension)
  File "c:\src\mozilla.org\mozilla-central\obj-x86_64-pc-mingw32\_virtualenvs\moztreedocs-OSceHKx-\Lib\site-packages\sphinx\application.py", line 402, in setup_extension
    self.registry.load_extension(self, extname)
  File "c:\src\mozilla.org\mozilla-central\obj-x86_64-pc-mingw32\_virtualenvs\moztreedocs-OSceHKx-\Lib\site-packages\sphinx\registry.py", line 487, in load_extension
    metadata = setup(app)
  File "c:\src\mozilla.org\mozilla-central\python/mozbuild\mozbuild\sphinx.py", line 183, in setup
    manager.generate_docs(app)
  File "c:\src\mozilla.org\mozilla-central\tools\moztreedocs\__init__.py", line 96, in generate_docs
    self._synchronize_docs()
  File "c:\src\mozilla.org\mozilla-central\tools\moztreedocs\__init__.py", line 164, in _synchronize_docs
    self._process_markdown(m, source_path, os.path.join(".", target))
  File "c:\src\mozilla.org\mozilla-central\tools\moztreedocs\__init__.py", line 127, in _process_markdown
    post = frontmatter.load(f)
  File "c:\src\mozilla.org\mozilla-central\obj-x86_64-pc-mingw32\_virtualenvs\moztreedocs-OSceHKx-\Lib\site-packages\frontmatter\__init__.py", line 107, in load
    text = fd.read()
  File "c:\mozilla-build\python3\lib\encodings\cp1250.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]
UnicodeDecodeError: 'charmap' codec can't decode byte 0x88 in position 189: character maps to <undefined>

Encoding error:
'charmap' codec can't decode byte 0x88 in position 189: character maps to <undefined>
./mach: failed to generate documentation:
c:\src\mozilla.org\mozilla-central: sphinx return code 2

What could be wrong?

Honza

Comment 1

[Sylvestre Ledru [:Sylvestre]]

I guess it is a bug in the frontmatter lib
Catching the exception in tools/moztreedocs/init.py and continue the operation might work.

As a temp workaround

diff --git a/tools/moztreedocs/__init__.py b/tools/moztreedocs/__init__.py
--- a/tools/moztreedocs/__init__.py
+++ b/tools/moztreedocs/__init__.py
@@ -117,16 +117,18 @@ class _SphinxManager(object):
             sphinx.ext.apidoc.main(argv=args)
 
     def _process_markdown(self, m, markdown_file, dest):
         """
         When dealing with a markdown file, we check if we have a front matter.
         If this is the case, we read the information, create a temporary file,
         reuse the front matter info into the md file
         """
+        m.add_link(markdown_file, dest)
+        return
         with open(markdown_file, "r") as f:
             # Load the front matter header
             post = frontmatter.load(f)
             if len(post.keys()) > 0:
                 # Has a front matter, use it
                 with tempfile.NamedTemporaryFile("w", delete=False) as fh:
                     # Use the frontmatter title
                     fh.write(post["title"] + "\n")

might work

Comment 2

[Jan Honza Odvarko [:Honza] (always need-info? me)]

Yep, works for me, thanks Sylvestre!

Honza

Comment 3

[Jan Honza Odvarko [:Honza] (always need-info? me)]

Ah, so there is one more problem at the end of the mach doc command

copying static files... ... done
copying extra files... done
dumping search index in English (code: en)... done
dumping object inventory... done
build succeeded, 1351 warnings.

The HTML pages are in obj-x86_64-pc-mingw32\docs\html.

Generated documentation:
c:/src/mozilla.org/mozilla-central/obj-x86_64-pc-mingw32\docs\html
Post processing HTML files
Error running mach:

    ['doc']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

UnicodeDecodeError: 'charmap' codec can't decode byte 0x98 in position 9466: character maps to <undefined>

  File "c:\src\mozilla.org\mozilla-central\tools/moztreedocs/mach_commands.py", line 106, in build_docs
    self._post_process_html(savedir)
  File "c:\src\mozilla.org\mozilla-central\tools/moztreedocs/mach_commands.py", line 187, in _post_process_html
    filedata = file.read()
  File "c:\mozilla-build\python3\lib\encodings\cp1250.py", line 23, in decode
    return codecs.charmap_decode(input,self.errors,decoding_table)[0]

Comment 4

[Jan Honza Odvarko [:Honza] (always need-info? me)]

Please see my previous comment. Any workaround for that?

Thanks!
Honza

Comment 5

[Andrew Halberstadt [:ahal]]

This looks like the same root issue as the one in https://bugzilla.mozilla.org/show_bug.cgi?id=1606970.

Comment 6

[Sylvestre Ledru [:Sylvestre]]

This should do (the post processing is for mermaid)

diff --git a/tools/moztreedocs/mach_commands.py b/tools/moztreedocs/mach_commands.py
--- a/tools/moztreedocs/mach_commands.py
+++ b/tools/moztreedocs/mach_commands.py
@@ -169,16 +169,17 @@ class Documentation(MachCommandBase):
             docdir,
             savedir,
         ]
         if jobs:
             args.extend(['-j', jobs])
         return sphinx.cmd.build.build_main(args)
 
     def _post_process_html(self, savedir):
+        return
         """
             Perform some operations on the generated html to fix some URL
         """
         MERMAID_VERSION = "8.4.4"
         for root, _, files in os.walk(savedir):
             for file in files:
                 if file.endswith(".html"):
                     p = os.path.join(root, file)

Comment 7

[Jan Honza Odvarko [:Honza] (always need-info? me)]

Attachment: Build proces output

Great, thanks!

The `mach doc` command is now running at the end and I can see docs running on `http://localhost:5500/`

But, when I change and existing *.rst file (or *.md file) I can't see the change on `http://localhost:5500/`

For example, I am changing these files:
* .\mozilla-central\gfx\docs\AdvancedLayers.rst => http://localhost:5500/gfx/AdvancedLayers.html
* .\mozilla-central\devtools\docs\bugs-issues.md => http://localhost:5500/devtools/bugs-issues.html

When saving the change I can see the following output from the build process (see the attachment)
But, the page itself doesn't change...

(the change is there if I run `mach doc` again but, it takes long time)

Honza

Comment 8

[Sylvestre Ledru [:Sylvestre]]

I don't know how this part of mach doc works. It has been magic too me when files are updated or not. Sorry :)

Comment 9

[Molly Howell (she/her) (no longer active)]

Attachment: Bug 1615223 - Fix multiple character mapping errors during documentation generation on Windows. r=#firefox-source-docs-reviewers

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/94b82b06b685
Fix multiple character mapping errors during documentation generation on Windows. r=firefox-source-docs-reviewers,championshuttler

Comment 11

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/94b82b06b685