1134373 - [Compat Data][Importer] Parse KumaScript in Specification comments

Status: RESOLVED FIXED

(developer.mozilla.org Graveyard :: General, enhancement)

Description

[John Whitlock [:jwhitlock]]

What problems would this solve?
===============================
KumaScript is added verbatim to the Section notes field, without warnings.

Who would use this?
===================
MDN staff during the import process.

What would users see?
=====================
MDN staff would be notified of KumaScript that is not understood by the parser. Compat data users would see HTML notes without links to other MDN documents, instead of KumaScript markup.

What would users do? What would happen as a result?
===================================================
MDN staff would not import data until the KumaScript is removed or the parser grows an understanding of the data.

Is there anything else we should know?
======================================
One page with in-comment KumaScript is Web/CSS/vertical-align:

https://developer.mozilla.org/en-US/docs/Web/CSS/vertical-align
https://browsercompat.herokuapp.com/importer/857

Comment 1

[Luke Crouch [:groovecoder]]

Is un-rendered KumaScript the problem? We can fetch the rendered KumaScript by appending &macros to the endpoint. E.g., https://developer.mozilla.org/en-US/docs/Web/CSS/vertical-align?raw&macros

Comment 2

[John Whitlock [:jwhitlock]]

The parse works with the raw version, because it is easier to extract data from the non-rendered versions of KumaScript, for example, 

{{ SpecName('CSS3 Transitions', '#animatable-css', 'vertical-align') }}

vs

<a class="external" href="http://dev.w3.org/csswg/css-transitions/#animatable-css" hreflang="en" lang="en">CSS Transitions<br><small lang="en-US">The definition of 'vertical-align' in that specification.</small></a>

Most of these macros are used to add links to other MDN pages, or apply styling.  The parser drops the MDN links, MDN-specific CSS classes, and adds <code> and &lt; &gt; as needed for general-purpose documentation.

Comment 3

[John Whitlock [:jwhitlock]]

I've submitted PR #33 [1], shipped the updated code to https://browsercompat.herokuapp.com/importer, and re-parsed the MDN pages.

[1] https://github.com/mozilla/web-platform-compat/pull/33

Comment 4

[MDN Team (:mdn-dev)]

Commits pushed to master at https://github.com/mozilla/web-platform-compat

https://github.com/mozilla/web-platform-compat/commit/77018a23fa629533e8915cf86f73ac3d03792f5e
bug 1134373 - Requirements bumps

* django-allauth 0.19.1 -> 0.20.0 - Bump email field size
* django-sortedm2m 0.9.5 -> 0.10.0 - Better customization hooks
* flake8 2.4.0 -> 2.4.1 - pip blacklist
* pylibmc 1.4.2 -> 1.4.3 - Threading fixes
* static3 branch -> 0.6.1 - Explicit UTF-8 reads
* virtualenv 12.1.1 -> 13.0.1 - pip, setuptools upgrades

https://github.com/mozilla/web-platform-compat/commit/ba10fc86df76b1cd8d0086b9387de2d410220518
bug 1134373 - Refactor issue tracking

Store issues in the FeaturePage JSON rather than load when deserializing
the data.  Speeds up page views, importing, and gathering import issues.

https://github.com/mozilla/web-platform-compat/commit/481c8e11b12f650d09077a9741332799fcd7850d
bug 1134373 - Switch spec description to tokenizer

Switch the specification description (third column of the table) from a
simple parser to a tokenizer pattern (like compatibility cells) so that
inline KumaScript can be transformed into plain text.

https://github.com/mozilla/web-platform-compat/commit/3e7959540c1d3d9606e135551daaed1bee404cf8
fix bug 1134373 - Handle spec desc. KumaScript

Translate KumaScript found on MDN in specification descriptions to plain
HTML. Warn on use of {{Spec2}}, which is probably a typo for
{{SpecName}}.

https://github.com/mozilla/web-platform-compat/commit/e113c8f7225edd61e995d8edeb035c688ad62e9b
Merge pull request #33 from jwhitlock/1134373_ks_spec_comments

Fix bug 1134373 - Parse KumaScript in Specification comments