1131124 - fix Elasticsearch running issues in vagrant vm

Status: RESOLVED FIXED

(Input :: General, defect)

Description

[Will Kahn-Greene [:willkg] ET needinfo? me]

The vagrant_provision.sh script installs Elasticsearch from the Elasticsearch ubuntu repositories. It's currently installing 0.90.10, but at some point in the future our Elasticsearch cluster will get upgraded to something more recent. After Elasticsearch is installed from the package, the provision script runs update-rc.d and sets it up to start on startup.

Thus when a contributor starts their vm, Elasticsearch should be up and running and everything should be groovy.

I've heard several anecdotes about how this isn't always the case. Also, I've heard several anecdotes about Elasticsearch dying after a few test runs.

The current working theory is that Elasticsearch is consuming too much memory for the vagrant vm and that it dies on startup or at some point with out of memory errors.

This bug covers looking into this further and doing something to alleviate the problems.

Comment 1

[Will Kahn-Greene [:willkg] ET needinfo? me]

For now, until there's evidence otherwise, let's assume the theory is correct. Therefore, we should do the following:

1. add a note to the docs about how to increase the amount of memory that vagrant uses for the vm in case the user needs/wants to

2. fix the vagrant_provision.sh script to stomp on the default elasticsearch configuration with a configuration that's better suited for a one-node/shard virtual machine environment--theoretically this will use less memory

3. (if all else fails) increase the default memory requirements to 1024--we should avoid this if we can since it's likely to reduce the pool of contributors who can run this in the first place

Comment 2

[Will Kahn-Greene [:willkg] ET needinfo? me]

Kitsune and DXR both use vagrant and Elasticsearch. It's possible one of them has this working better.

Comment 3

[Will Kahn-Greene [:willkg] ET needinfo? me]

I've been doing other Vagrant-y things lately. I think we should do the following here:

1. Raise the vm memory to 1024mb.

2. Change the vagrant_provision.sh script to stomp on the Elasticsearch config with one that's better suited for one-node/shard Elasticsearch environment might alleviate some memory tensions


I tweak my local Elasticsearch configuration with these lines:

index.number_of_shards: 1
index.number_of_replicas: 0
index.refresh_interval: 10

The way DXR does it is that they tell you to increase the memory of the vm to 6gb. I don't think we want to require that since it prevents many people from contributing to Fjord.

Kitsune doesn't have a working vagrant environment.

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

Erik suggested we make sure we're using at least Java 7, too. OpenJDK is fine.

Comment 5

[Will Kahn-Greene [:willkg] ET needinfo? me]

Anecdote: mkelly had problems with running the tests. He got an OSError("Out of Memory") kind of thing from Python (not Elasticsearch, but I think it's still a memory scarcity problem). He increased the vm memory to 1024mb and didn't have it happen again.

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

LGuruprasad is working on this--assigning it to him.

Comment 7

[Will Kahn-Greene [:willkg] ET needinfo? me]

Bumping this to 2015q2.

Comment 8

[Will Kahn-Greene [:willkg] ET needinfo? me]

In a PR: https://github.com/mozilla/fjord/pull/551

Comment 9

[Will Kahn-Greene [:willkg] ET needinfo? me]

Landed in https://github.com/mozilla/fjord/commit/fb8dce2916d5b4237ea9e3e4d520ea624800f364

This doesn't affect production environments, so I'm marking it FIXED.

Thank you!