958059 - elasticsearch storage location for elmo

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Axel Hecht [:Pike]]

For bug 857107, we need a storage location on an elasticsearch cluster.

Filing in elmo for now, Laura, can you move that to the right product/component?

We'll only need a single index.

Are there any particular metrics we should provide?

I expect to only need a few reads every other minute, the busy data is all in the elmo database.

For writes, we'll start off with conversions being triggered by a script in a cron job, so we can stagger that a bit. The end-goal is to have the automation write directly, which is going to be less than a document per second in peak write load, given what the automation jobs need to do between writes.

Comment 1

[Laura Thomson :laura]

Total number and size of documents would be a good one to add. I'll follow up with IT

Comment 2

[Axel Hecht [:Pike]]

We have about 300k documents right now. The size of the documents is a bit hard for me to estimate, for one, because I don't know how to get from my data to the document size, but also because the size of the documents will vary between a few bytes, and possibly also around 200k. That's at least the size of the text log of one of the bigger inputs.

Comment 3

[Brandon Burton [:solarce]]

Is this bug referring to https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=1082396 ?

Comment 4

[Axel Hecht [:Pike]]

Yes, but that page needs an update, the stuff that's "new" there was new in 2011.

Comment 5

[Axel Hecht [:Pike]]

Updated the page, there is a bit more truth now.

Comment 6

[Brandon Burton [:solarce]]

You can use the existing ES cluster in SCL3 ( https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=5112369&src=search#Elasticsearch%28webapp%29-SCL3Production )for this. The frontend address to use is elasticsearch-zlb.webapp.scl3.mozilla.com

Let me know if you have any issues

Comment 7

[Axel Hecht [:Pike]]

CC cyliang, #webdev says she might know.

Do we have a post-1.0 ES cluster coming up in SCL3? There are quite some incompatibilities between pre and post 1.0, and given that we've not picked a side in deployment yet, I'd rather do that now, and possibly on the post-1.0 side.