769129 - Templates with non English letters in the name lost all content in the copy

Status: RESOLVED FIXED

(developer.mozilla.org Graveyard :: Wiki pages, defect)

Description

[Trevor]

When the Suivant template was transferred it came across fine (all English letters), however Précédent and it seems others with non English letters did not, such as PrécédentSuivant and ParaPáginaDoAssunto
The only exception seems to be 翻訳中

Comment 1

[Les Orchard [:lorchard]]

Hmm, content is here now: https://developer-new.mozilla.org/en-US/docs/Template:Pr%C3%A9c%C3%A9dent

Did someone fix this already?

Comment 3

[Les Orchard [:lorchard]]

Not fixed. Another page with templates affected by this: https://developer-new.mozilla.org/ja/docs/JavaScript/Guide/JavaScript_Overview

Comment 4

[Les Orchard [:lorchard]]

I have a hypothesis: Progress log messages in the migration script display the document slug, some of which contain non-English characters. However, it looks like the construction of those messages uses non-Unicode strings. That can cause an exception in Python. But, the migration script is built to trap and report those exceptions, yet carry on with the rest if it can.

So, I think maybe progress log messages for documents with URL paths containing non-English characters caused a migration failure for those documents, part way through. Unfortunately, I don't think we captured or kept the log output from the migration script when we did it a few months ago. So, the evidence is gone.

It seems like we can identify the broken documents: The current revision will have empty content, reviewed, and reviewed_by fields. If this works, then we can add an option to the migration script that queries for just those documents without damaging any other work.

Now, here's the weird thing: I've run full migrations on my laptop and on an EC2 server, and have not seen this problem arise. All the documents I found on developer-new that had empty content were fine on my own servers. That makes me think maybe my hypothesis is wrong, because I can't reproduce it at all on my own installations.

Still have some poking to do on this

Comment 5

[[github robot]]

Commits pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/6f34606c32f4a59b33f09a1e1fe22472ab06f9df
bug 769129: Migration tool tweaks to re-migrate empty content

* --withemptycontent option to re-migrate Kuma documents whose current
  revision has empty content. There should be very few documents with
  genuinely empty content.

* --withunreviewed option to re-migrate documents whose current revision
  is unreviewed. Symptom of a failed migration, because migration tries
  to set all current revisions as reviewed

* No longer attempt to update the current revision in place during
  migration. Always create a new revision. This solves a DB error.

* Try to use unicode strings and formats in all log output statements.

* Add a "current_revision_reviewed" column to document admin list view.
  Sorting on this should show documents with empty values.

https://github.com/mozilla/kuma/commit/49f7bcad7caa90c08da3a7b3056be29587575890
bug 769129: Further migration command tweaks

* We will output using UTF-8, ASCII be damned.

* Quick bugfix for already-migrated revisions

https://github.com/mozilla/kuma/commit/839cfc49204f66e42cc80a9258a8013e9a2899b8
Merge pull request #378 from lmorchard/migration-tweaks-769129

bug 769129: Migration tool tweaks to re-migrate empty content