805861 - create a script to save an index

Status: RESOLVED FIXED

(Marketplace Graveyard :: Search, defect)

Description

[Tarek Ziadé (:tarek)]

ES Indexes take a long time to build, we should start to use the snapshot feature in case something goes wrong on a reindexation

Comment 1

[Wil Clouser [:clouserw]]

Is this a dev bug or an ops bug?

Comment 2

[Tarek Ziadé (:tarek)]

it's a devops bug ? :)

I guess we need to :

1/ try out the snapshot feature on -dev [ops]
2/ build a script that calls this feature [dev]
3/ hook it in the reindex command  [dev]

Comment 3

[Wil Clouser [:clouserw]]

Jason, would you organize #1 with Tarek?

Comment 4

[Tarek Ziadé (:tarek)]

From IRC:

< oremj> tarek: I think we are using the default gateway
< oremj> and the snapshot docs say they don't work with the default gw
< oremj> sounds like a snapshot with the default gateway just means copying the storage

I guess we could provide a system script to run this backup, we can invoke from Django's command machinery.

Comment 5

[Jason Thomas [:jason]]

I created a script that tars the index data on each node for a specified index [1].

It takes around 5 minutes to backup addons_dev index and 45 minutes to backup addons_dev_stats index.

To perform a restore we will need restore the tar on each node and perform a full cluster restart. I will be testing the restore tomorrow from 6AM-7AM PST. 

[1] https://gist.github.com/4035724

Comment 6

[Jason Thomas [:jason]]

During today's testing I performed the following on the addons dev/stage elasticsearch cluster:

1. Created a backup of addons_dev-20121024062906 index using script.
2. Deleted addons_dev-20121024062906 from elasticsearch. This broke addons-dev.allizom.org search.
3. Stopped elasticsearch on all nodes.
4. Restored backup of addons_dev-20121024062906 index to elasticsearch.
5. Started elasticsearch on all nodes. Verified addons_dev alias and addons_dev-20121024062906 index is available in elasticsearch.
6. Tested addons-dev.allizom.org search, looks okay.

Comment 7

[Tarek Ziadé (:tarek)]

cool!

out of curiosity, how big is the backup ? can we do this every day ?

Comment 8

[Tarek Ziadé (:tarek)]

(this == 1.)

Comment 9

[Jason Thomas [:jason]]

(In reply to Tarek Ziadé (:tarek) from comment #7)
> cool!
> 
> out of curiosity, how big is the backup ? can we do this every day ?

Compressed addons_dev index is ~400MB, addons_dev_stats is ~4GB.

We can configure this via cron to run every day and remove backups older than 7 days. How does that sound?

Comment 10

[Wil Clouser [:clouserw]]

(In reply to Jason Thomas [:jason] from comment #9)
> (In reply to Tarek Ziadé (:tarek) from comment #7)
> > cool!
> > 
> > out of curiosity, how big is the backup ? can we do this every day ?
> 
> Compressed addons_dev index is ~400MB, addons_dev_stats is ~4GB.
> 
> We can configure this via cron to run every day and remove backups older
> than 7 days. How does that sound?

Sounds good to me

Comment 11

[Jason Thomas [:jason]]

Added backup cron to dev/stage and prod for addons indices.