1016690 - A few Kibana dashboards (stored in es) get 404 errors

Status: RESOLVED WONTFIX

(Cloud Services :: Operations: Metrics/Monitoring, task)

Description

[Katie Parlante]

Edwin noticed these dashboards didn't load from the file menu:

https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/elasticsearch/PROD%20-%20QA%20Request%20Time

https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/elasticsearch/PROD%20-%20QA%20Dashboard

I noticed the same for these two:
https://kibana.fxa.us-west-2.prod.mozaws.net/#/dashboard/elasticsearch/FraudDetection

https://kibana.fxa.us-west-2.prod.mozaws.net/#/dashboard/elasticsearch/More%20Endpoint%20Counts

whd noticed they are in the index and you can see them with /_search

Edwin observed that he can't delete them.

fwiw, Edwin has backups of the two on the shared stack, the last one is actually saved in puppet-config. As whd points out, would be nice to know what went wrong.

Comment 1

[James Bonacci [:jbonacci]]

Also here: https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/elasticsearch/Dash%20Test

The few links we have for Stage, seem ok...
So, just on the Prod side.

Comment 2

[Wesley Dawson [:whd]]

When I go to add the dashboards back to ES from the blobs I grabbed via _search two entries show up in the Load Menu. This is somewhat confounding behavior. The dashboard works fine, but if I delete one of the entries from the Load Menu the other one doesn't work (presumably because the document was actually deleted). It strikes me that this could be related to a collision with Edwin but since it happened multiple times I'm reasonably sure it's not.

So as a workaround for now I will reload all these "missing" documents from the index, and the load menu will have extraneous entries. :jbonacci I'm going to leave that Dash Test in the "missing" state so I have a dashboard with the odd behavior to reference while getting to the bottom of this.

Comment 3

[Wesley Dawson [:whd]]

I've added back all the missing dashboards. I'm still investigating the cause and how to get rid of the duplicate entries in the load menu. My guesses are (a) some kind of corruption from the rolling upgrade over the weekend (b) something more insidious related to elasticsearch document/type versioning. If after investigation I find nothing I will assume (a) and blow away/repopulate kibana-int.

Comment 4

[Wesley Dawson [:whd]]

http://www.elasticsearch.org/blog/elasticsearch-1-2-1-released/ is probably the cause. I'll be doing rolling updates to 1.2.1.