747137 - wiki: Transfer documents between kuma instances via web UI

Status: VERIFIED FIXED

(developer.mozilla.org Graveyard :: Editing, defect, P2)

Description

[Les Orchard [:lorchard]]

We need to be able to:

* Export documents to a local file from a Kuma install via the web.

* Import documents from a local file to a Kuma install via the web.

* For bonus points: Check an export file into git, let the migration script auto-import it. (eg. for Templates) This could be a follow up bug.

---

Brain dump follows:

For a full solution to this bug, an admin-level user of Kuma must be able to complete the cycle of exporting from one instance to another and preserve at least the most recent revision of a document. And, all without filing IT bugs or having shell access to either Kuma instance.

We had bug 734187 to export templates, but the import side doesn't really work. And, we need to be able to transport more than just templates around.

django-smuggler almost gets us there, but there are issues with primary keys on the import side. It might be possible to build a custom importer based on smuggler that adjusts the import on the fly to match up document slugs and associated revisions.

Maybe Django's "natural keys" support [1] can help, at least for some of the foreign key relations (eg. revision->document, {document,revision}->user)?

[1] https://docs.djangoproject.com/en/dev/topics/serialization/#natural-keys

Comment 1

[Les Orchard [:lorchard]]

Oh, and for more context: 

By "primary key issues", I mean that the IDs for Documents, Revisions, and Users are never guaranteed to match up between Kuma instances. So, a straight export and import of a JSON fixture stumbles on that. We might also need to add Users to the export.

The good news is that Documents can be uniquely referred to by locale + slug. And Users can be uniquely referred to by username. I don't think we can quite switch to using those as primary keys, but Django's natural keys support might let us swap those unique fields for IDs.

But, where natural key support doesn't help us is in the IDs of the imported objects themselves. There, I think we need to have logic that looks for an existing record to replace with the unique attributes, or create a new record.

Comment 2

[[github robot]]

Commits pushed to master at https://github.com/mozilla/kuma

https://github.com/mozilla/kuma/commit/a58f93223362a459b0e028ee3a7b729751d4f42a
Bug 747137: Transfer documents between instances

* Support (locale, slug) as natural key for Documents

* Replace smuggler data dump for documents
    * dump_json method in DocumentManager
    * exclude primary keys
    * ensure the current or at least most recent revision is included

* Replace smuggler data load for Documents
    * load_json method in DocumentManager
    * Ignore all incoming primary keys
    * Use Document natural key to find existing document for replacement
    * Replace "creator" on Revision with user uploading data
    * Replace "created" on Revision with current time

* New view for Document dump in admin

* New templates for data dump and load in admin

* Include current revision tags in list of fields searched by keyword
  queries in Document admin.

* Tweak to /etc/hosts in Vagrant VMs to ensure developer-dev.mozilla.org
  can be resolved.

https://github.com/mozilla/kuma/commit/f980a3f9c6b889676c3836f4d6c726f1d1ffc839
Merge pull request #210 from lmorchard/bug-747137-transfer-documents

Bug 747137 transfer documents