559648 - Revert database index from backup

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Eric Shepherd [:sheppy]]

We need to revert the lucene index for MDC to the last backup before the attempted 9.12.2 upgrade. The index is for a newer version of lucene and is not working correctly because of it, and we can't successfully rebuild it either.

MindTouch recommends this procedure:

1. Restore the index file (/var/www/dekiwiki/bin/cache/luceneindex/default) from backup.
2. Wait until a relatively slow time of day, usage wise, for MDC.
3. Disable checkdeki.
4. Have me start an index rebuild.
5. When the index rebuild is complete, re-enable checkdeki.

Let me know when we can do this; needs to be done as soon as feasible so the site is searchable again.

Comment 1

[Shyam Mani [:fox2mike]]

Need to co-ordinate with a few folks on this.

Aravind, can you please get the file from backup? 

Since step 4 is involved, someone from the US will have to schedule this so they're up and around when you're up and around. I'll guess Jeremy does our MDC stuff, so I'll punt to him and let him co-ordinate with you on what's best for the two of you.

Comment 4

[Eric Shepherd [:sheppy]]

Steps 2-5 can be done later; do step 1 (the index backup restore) right away. The rebuild is only needed to update the index to pick up changes made since the morning of the attempted software upgrade.

Comment 5

[Jeremy Orem [:oremj]]

 Can we just rebuild from scratch?

Comment 6

[Eric Shepherd [:sheppy]]

I did that. The resulting file was even worse than the original; that's when we started getting "The text 'foo' uses an encoding that requires escaping" errors.

Comment 7

[Jeremy Orem [:oremj]]

Hmm, does mindtouch have instructions for fully clearing out and rebuilding this cache?  Typically backups take a few days to restore if they are available.

Comment 8

[Eric Shepherd [:sheppy]]

I've asked MindTouch. I'll follow up here when I get a response.

Why does that file take so long to restore, if I may ask? Curious about how this stuff works.

Comment 9

[Dave Miller [:justdave]]

There are no backups.  After poking around a bit, we figured out that the data storage for this is actually within the docroot on each webhead, and not symlinked onto the shared storage.  The shared storage gets backed up, and the cluster master gets backed up, but said directory on the cluster master is completely empty because each webhead is generating the data itself.

Comment 10

[Dave Miller [:justdave]]

OK, they told us the default location instead of telling us it was configurable and where the path is.  With that added detail, it turns out it is stored on the netapp, so we may have a backup after all.

What we need restored:

netapp-b:/vol/mpt_app_cluster/developer.mozilla.org-dekiwiki/lucene

For future reference, you can find this path by running:

grep path.store /etc/dekiwiki/mindtouch.deki.startup.xml

Comment 11

[Christopher Blizzard (:blizzard)]

I just want to get some clarity here.  We're talking about the index for search, not the actual data that you use to build the index, right?

And that data _is_ backed up, right?

Comment 12

[Andrei Hajdukewycz [:sancus]]

I would be happy to help if I had access to do so, though I'm not entirely sure what I can do :)

Comment 13

[Andrei Hajdukewycz [:sancus]]

To be more specific, it seems like it might be possible to get Lucene fixed to reindex from scratch properly faster than a backup can be restored, or I can at least try.

Comment 14

[Eric Shepherd [:sheppy]]

This is just the index for the search, yes. The actual data is all there now. Search is just broken.

justdave and I just spent a couple hours working with MindTouch on this (well, mostly justdave), and the index is currently rebuilding. In theory once it's done all will be well. We're keeping an eye on it while it goes.

Comment 15

[Eric Shepherd [:sheppy]]

This is what I just added to our ticket with MindTouch:

This is very confusing. After working with Brian tonight, we got the index rebuilding.

At this time, it claims the indexing is complete, but one of our hosts is reporting "Your search query ctypes contains characters which need to be escaped. See this FAQ for more information."

The other host is returning incomplete results, as if from an old, out of date index, as if the rebuild isn't finished yet, even though the control panel says it's done.

Comment 16

[Eric Shepherd [:sheppy]]

OK, the reindex wasn't actually done; I was just hitting the other box. But 02, the box doing the reindex, crashed around the time it finished. justdave restarted it just now.

The index seems to be working, for now.

Ironically, 9.12 can resume an index build after a crash, which would have made a lot of this trouble moot -- except of course, well, you know.

Not 100% sure the index fully rebuilt, but hopefully it is. At least all my searches are working, including searches for stuff written just today, so let's go with that for now.

Comment 17

[Jeremy Orem [:oremj]]

(In reply to comment #11)
> I just want to get some clarity here.  We're talking about the index for
> search, not the actual data that you use to build the index, right?
> 
> And that data _is_ backed up, right?

The actual data to build the index is backed up.  This bug was just about the index for search, which can be rebuilt.