721485 - developer-stage9: add migration cron job

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Luke Crouch [:groovecoder]]

See bug 721072 for specifics.

Ping groovecoder and/or lorchard with questions.

Comment 1

[Luke Crouch [:groovecoder]]

Priority bump. Want to turn this on next week if possible so MDN doc team can start playing with it.

Comment 2

[Luke Crouch [:groovecoder]]

Priority bumping again. Can we do this week?

Comment 3

[Jake Maul [:jakem]]

I need more details on this...

Presumably this should be run something like this:

cd <kuma dir>; python26 manage.py cron migrate

Is that right?

Also, how frequently should this be run?

Comment 4

[Luke Crouch [:groovecoder]]

Sorry, details were in bug 721072:

manage.py migrate_to_kuma_wiki --recent

Every 5 minutes would be good, but we might have to adjust if the docs team wants it to be more responsive.

Comment 5

[Jake Maul [:jakem]]

[root@mradm02 cron.d]# cd /data/dekiwiki/www/django/developer.mozilla.org/kuma; python26 manage.py migrate_to_kuma_wiki --recent
Usage: manage.py migrate_to_kuma_wiki [options] 

Migrate content from MindTouch to Kuma

manage.py: error: --recent option requires an argument

Comment 6

[Luke Crouch [:groovecoder]]

Oops, need --recent=20. But again we might need to adjust depending on feedback from docs team.

Comment 7

[Jake Maul [:jakem]]

[root@mradm02 kuma]# cd /data/dekiwiki/www/django/developer.mozilla.org/kuma; python26 manage.py migrate_to_kuma_wiki --recent=20
Traceback (most recent call last):
<snip>
_mysql_exceptions.OperationalError: (2002, "Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2)")


Looks like it's trying to connect to a local MySQL instance... is this not obeying the settings files somehow?

Comment 8

[Luke Crouch [:groovecoder]]

Sorry, most of this is covered in bug 721072.

Need to add the MIGRATION_DATABASES setting value to point to the wikidb on stage9. https://github.com/mozilla/kuma/blob/mdn/puppet/files/vagrant/settings_local.py#L63 for example.

And we can use the scripts/migrate_recent.sh script so that we can change the number of documents in code in the future.

Comment 9

[Les Orchard [:lorchard]]

Looks like a cronjob for this is running now, and we're getting error mails every 5 minutes. Shouldn't break anything, but should probably be disabled until this is worked out

Comment 10

[Luke Crouch [:groovecoder]]

Bump again. Can we change the cron job to scripts/migrate_recent.sh ? That should fix the errors.

Comment 11

[Jake Maul [:jakem]]

[root@mradm02 kuma]# scripts/migrate_recent.sh 
Traceback (most recent call last):
  File "./manage.py", line 28, in ?
    import settings_local as settings
  File "/data/dekiwiki-stage/src/developer-stage9.mozilla.org/kuma/settings_local.py", line 1, in ?
    from settings import *
  File "/data/dekiwiki-stage/src/developer-stage9.mozilla.org/kuma/settings.py", line 377
    'finalize': lambda x: x if x is not None else ''}
                             ^
SyntaxError: invalid syntax


I believe you need to call 'python26' in your scripts/migrate_recent.sh.


I think the MIGRATION_DATABASES block is done now too. It looks like this:

MIGRATION_DATABASES = {
    'wikidb': {
        'NAME': 'dekiwiki9_devmostage',
        'ENGINE': 'django.db.backends.mysql',
        'HOST': '<ip>',
        'USER': '<user>',
        'PASSWORD': '<password>',
    },
}