templates: record template usage stats during migration



7 years ago
7 years ago


(Reporter: groovecoder, Assigned: lorchard)


Dependency tree / graph


(Whiteboard: u=developer c=wiki p=1)


(4 attachments, 1 obsolete attachment)



7 years ago
To help with bug 710728.

Comment 1

7 years ago
Quick & dirty template call parsing & extraction in the migration script:

Comment 2

7 years ago
Built a quick page parsing and template call extraction option in the migration script. It runs through all default namespace pages and looks for DekiScript calls, where default namespace are pages without colon-prefixes (eg. Talk:, User:, Special:, etc)

I ran the template extraction on the MindTouch import in my VM:

$ ./manage.py migrate_to_kuma_wiki --all --template-metrics > templates.txt
$ sort templates.txt | uniq -c | sort -rn | head -25
  11906 wiki.languages
  10515 wiki.template
   4975 template.Source
   4703 template.XULElem
   4409 template.PrefAnch
   4318 mediawiki.external
   4257 HTMLElement
   4039 template.XULAttr
   3423 template.Cssxref
   3256 domxref
   2808 Interface
   2680 interface
   2652 CompatUnknown
   1970 template.XULAttrInc
   1861 Cssxref
   1835 gecko_minversion_inline
   1684 template.XULPropInc
   1668 template.Bug
   1633 CompatNo
   1574 SVGElement
   1541 cssxref
   1323 template.DomRef
   1239 template.PreviousNext
   1200 template.Interface
   1177 template.XULRefAttr

Attached is a full dump, without the `head -25`

Comment 3

7 years ago
Hopefully this list gives a good priority list of which template scripts need a looking at and support in Kuma.

The list could definitely use some eyeballs and a sanity check, in case it's missing any templates that someone more familiar with MDN content knows for sure should be there.


7 years ago
Whiteboard: u=developer c=wiki p= → u=developer c=wiki p=1

Comment 4

7 years ago
sheppy, jms, teoli: can you review this list to help with bug 715253 - let us know which data resources are the most prevalent in these scripts?

Comment 5

7 years ago
Self-assigning. Could probably call this closed, but want to get some eyes on the list before that
Assignee: nobody → lorchard


7 years ago
Blocks: 715253

Comment 6

7 years ago
Posted file Second count of template use (obsolete) —
Getting a little more understanding after beginning to poke into these results. Looks like the wiki.template() calls need to be broken down further, since they represent an alternative calling style to invoke templates.

Comment 7

7 years ago
Attachment #588107 - Attachment is obsolete: true

Comment 8

7 years ago
Did some more normalization and collation, trying to convert in-page calls to Template: page names.

Comment 9

7 years ago
Commit pushed to https://github.com/mozilla/kuma

deki-migration: bug 714804, more normalization for better collation of template metrics

Comment 10

7 years ago
Going to call this closed, since a count of templates was produced. Next steps are to continue to refine those counts, and to spend more time reading the templates to get a sense for capabilities needed by a new template system
Last Resolved: 7 years ago
Resolution: --- → FIXED

Comment 11

7 years ago
For good measure, here's another dump from migration. Should be useful for things like this report on the pages with the most templates used:

$ gzip -dc tmpl-use-raw.txt.gz | cut -f1 | sort | uniq -c | sort -rn | head -25
   2110 ja/reftest_opportunities_files
   2073 en/reftest_opportunities_files
   1648 en/Interfaces
   1580 trevorh/Interface_documentation_status
    518 en/Gecko_DOM_Reference
    473 ja/CSS/CSS_Reference/Mozilla_Extensions
    393 en/SVG/Attribute
    353 en/Interfaces_moved_in_Firefox_3.6
    339 en/CSS/CSS_Reference/Mozilla_Extensions
    284 en/HTML/Element/Input
    268 ja/Firefox_4_for_developers
    265 en/DOM/element
    253 pt/Firefox_4_para_desenvolvedores
    251 en/HTML/Attributes
    251 en/Firefox_4_for_developers
    238 en/CSS/CSS_Reference
    230 en/HTML/Content_categories
    227 es/Firefox_4_para_desarrolladores
    222 fr/Référence_CSS
    221 ja/CSS/CSS_Reference
    216 en/HTML/Element
    215 es/Referencia_CSS
    214 zh_tw/Firefox_2_佈景主題之更動
    214 ja/Theme_changes_in_Firefox_2
    213 pl/Zmiany_w_motywie_graficznym_w_Firefoksie_2

Something like top templates will need some more munging (eg. parse the template name from the params, etc)
Version: Kuma → unspecified
Component: Website → Landing pages
Product: Mozilla Developer Network → Mozilla Developer Network
You need to log in before you can comment on or make changes to this bug.