Closed Bug 1358541 Opened 9 years ago Closed 7 years ago

tidylib outputs errors and generates full HTML docs

Categories

(developer.mozilla.org Graveyard :: Wiki pages, enhancement)

Product:
developer.mozilla.org Graveyard
Component:
Wiki pages
Platform:
All
Other
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WORKSFORME

People

(Reporter: jwhitlock, Unassigned)

References

Details

(Keywords: in-triage, Whiteboard: [specification][type:bug])

What did you do? ================ tidylib is run against revisions to standardize the HTML for useful diffs between document versions, and stores this in the 'tidied_content' field in the database. What happened? ============== tidylib often generates a UnicodeDecodeError with non-ASCII content, such as: UnicodeDecodeError: 'utf8' codec can't decode byte 0xe5 in position 0: unexpected end of data This is reported to Sentry [1], and is the most common error with 873,000 events. The content we are tidying is partial HTML, which tidylib converts to full HTML docs. The input: <p>Hi.</p> Is tidied to: <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN"> <html> <head> <title></title> </head> <body> <p> Hi </p> </body> </html> This "wrapper" HTML sometimes is visible in diffs. Finally, we do not appear to be specifying an input/output encoding. The default input encoding is ASCII, and the default output is latin-1. [1] https://sentry.prod.mozaws.net/operations/mdn-prod/issues/222527/ What should have happened? ========================== * Only legitimate encoding errors should raise exceptions * The input and output should be UTF-8 encoded * The wrapped HTML should be removed from the tidied output. * Existing tidied content should be incrementally updated to the new standard. * New revisions should be tidied before the diff is sent to Akismet, so that 'no content change' edits can be reliably detected and omitted (bug 1265495) Is there anything else we should know? ====================================== We're using pytidylib 0.2.4. The latest is 0.3.2 [2]. An upgrade may be needed to get the latest, documented code. The full tidylib options [3] include options such as char-encoding [4]. It should be reviewed for the best options for MDN content. [2] https://github.com/countergram/pytidylib [3] http://tidy.sourceforge.net/docs/quickref.html [4] http://tidy.sourceforge.net/docs/quickref.html#char-encoding
See Also: → 1265495
Related to bug 1277311 at all?
Blocks: 1277311
Keywords: in-triage
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → WORKSFORME
No longer blocks: 1277311
Product: developer.mozilla.org → developer.mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.