686005 - Set up dedicated MindTouch lucene server for devmo

Status: RESOLVED WONTFIX

(mozilla.org Graveyard :: Server Operations, task)

Description

[Eric Shepherd [:sheppy]]

For performance and stability reasons, MindTouch recommends a dedicated machine running MindTouch that handles all API requests related to the lucene index, as well as the actual indexing, with the data files local on that machine. This would not be accessible over the web, but would simply handle API requests for searching.

This is their recommended configuration for multi-node installs of the software.

We should make this change to improve reliability; they can provide guidance as to how to configure it; [e-mail] has that info.

Comment 1

[Jake Maul [:jakem]]

Downgrading priority to avoid paging on-call.

Mindtouch also has the ability to use memcached (with an appropriate license). This would be a suitable place to put that daemon as well, since memcached requires virtually no CPU, and in this case would probably require very minimal RAM usage as well.

I would prefer to try and do this with a nice vanilla Lucene installation, rather than loading it up with all of the devmo / Mindtouch stuff. I don't know how feasible that is, and we might want to talk to Mindtouch about their preferred deployment method for a separate Lucene indexer server. We *could* simply set it up as "pm-dekiwiki04" and just not add it to the LB, but that would not be my preference.

I'm thinking the index data files should be on local storage, rather than NetApp NFS. They're only on NFS so that the servers can share the index, which is bad practice anyway. I don't expect this requirement to significantly alter the server requirements... anything with RAID-1 or better should suffice. Actual storage space is not a concern.

CPU and memory requirements are also fairly small. 2+ cores (4+ would be nice- reindexing is very CPU-intensive) and at least 4GB RAM should be sufficient for starters.

Assigning to cshields to pick hardware. This would be in MPT. As a reminder, we were looking at pm-app-amo29 for this since the bulk of AMO is moved to PHX and they should have plenty of excess capacity in MPT. This server would be more than enough for the job.

Comment 2

[Eric Shepherd [:sheppy]]

From what I understand, you'd need the MindTouch stuff on there because the idea is to have MindTouch API requests directed to it to handle search stuff.

Comment 3

[Jake Maul [:jakem]]

pm-app-amo29 has been drained for this purpose.

RHEL5 might be quickest. That way we can immediately re-use the existing puppet classes, and just tweak them for the new lucene host. I have no idea how it will Mindtouch would interact with RHEL6, and it'd probably be a bigger hassle to make puppet happy.

Comment 4

[Corey Shields [:cshields]]

that node is kickstarted..  kicking this over to you, Jake

Comment 5

[Jake Maul [:jakem]]

At this point, we're not going to bother finishing this off. We're close to eliminating Mindtouch entirely. :)