1399576 - [ops infra socorro] elasticsearch in stage

Status: RESOLVED FIXED

(Socorro :: Infra, task)

Description

[Miles Crabill [:miles]]

We need to build a working ES 1.4.5 cluster, snapshot the data from the existing cluster in webeng stage to S3, and restore it to the new cluster.

Comment 1

[Miles Crabill [:miles]]

There's some sadness associated with running ES 1.4.5 in Docker as I am doing. I'm butting up against an issue where we are unable to create the snapshots repository on the new ops stage cluster:

$ curl -XPUT 'http://localhost:9200/_snapshot/webeng_stage_s3_snapshots' -d '{ "type":"s3", "settings": { "bucket": "miles-es-snapshots-test", "region": "us-west-2" } }'
> {"error":"RemoteTransportException[[Ultimus][inet[/172.31.17.100:9300]][cluster:admin/repository/put]]; nested: RepositoryVerificationException[[webeng_stage_s3_snapshots] path  is not accessible on master node]; nested: IOException[Unable to upload object tests-Yw6roItkTaWbResbLvuUNQ-master]; nested: AmazonS3Exception[AWS authentication requires a valid Date or x-amz-date header (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 2E8AE690A48EFCB6)]; ","status":500}

So, the crux of the issue is this "AWS authentication requires a valid Date or x-amz-date header" business. Poking around yielded [0] and [1] , which suggests a couple plans of action:

- upgrade the Joda-Time library
- upgrade the AWS SDK for Java
- downgrade Java to a version less than 1.8u60 or to 1.7uX

Of these, downgrading the version of Java in the container is probably the easiest and what I will be attempting first.

[0] https://github.com/elastic/elasticsearch-cloud-aws/issues/233
[1] https://stackoverflow.com/questions/32058431/aws-java-sdk-aws-authentication-requires-a-valid-date-or-x-amz-date-header

Comment 2

[Miles Crabill [:miles]]

So, it's been a while. Here's what's happened:

- We have a working 1.4.5 elasticsearch cluster in ops stage. I opted for downgrading the version of Java in the container (rolling my own container in the process: https://hub.docker.com/r/mozilla/socorro_elasticsearch/tags/)
- The webapp in ops stage can successfully talk to elasticsearch and use it as a datasource
- I have snapshotted and restored webeng stage elasticsearch into the ops stage elasticsearch cluster and verified that that data mostly works in the webapp
- I am working on an automated snapshotting system for the the new elasticsearch cluster

So, this bug is mostly complete.

The deploy code for the ops ES cluster is a little rough around the edges for a few reasons:

- I didn't fully delete the NAT code that we were using for the previous ES 5.3 cluster (commented out)
- I didn't fully delete some of the configuration we were using for the previous ES 5.3 cluster (commented out)

I will wait on our team meeting next week to decide the fate of the 5.3 relevant code.

Todo before closing this bug:
  - Finish automated snapshotting system

Comment 3

[Miles Crabill [:miles]]

We no longer have any need to migrate data to -new-stage. Adjusting the title to fit.

Comment 4

[Will Kahn-Greene [:willkg] ET needinfo? me]

Miles: What's outstanding in this bug?

Comment 5

[Miles Crabill [:miles]]

The snapshotting system is outstanding, but that's a nice-to-have. Overall, the cluster is working just fine and should be reliable for production use.

Comment 6

[Will Kahn-Greene [:willkg] ET needinfo? me]

Miles: Is the snapshotting thing done? Is there a bug for whatever that entails?

Comment 7

[Miles Crabill [:miles]]

The snapshotter isn't done, but that's OK. I've created a bug 1438360 describing it. That doesn't need to block this.

We've merged the infra code for elasticsearch and have snapshot and restore scripts and procedures in place. Contrary to my Comment 3, we will be migrating data, but only for prod.

Marking this fixed, as it only covers stage.