All users were logged out of Bugzilla on October 13th, 2018
Edwin noticed these dashboards didn't load from the file menu: https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/elasticsearch/PROD%20-%20QA%20Request%20Time https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/elasticsearch/PROD%20-%20QA%20Dashboard I noticed the same for these two: https://kibana.fxa.us-west-2.prod.mozaws.net/#/dashboard/elasticsearch/FraudDetection https://kibana.fxa.us-west-2.prod.mozaws.net/#/dashboard/elasticsearch/More%20Endpoint%20Counts whd noticed they are in the index and you can see them with /_search Edwin observed that he can't delete them. fwiw, Edwin has backups of the two on the shared stack, the last one is actually saved in puppet-config. As whd points out, would be nice to know what went wrong.
Also here: https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/elasticsearch/Dash%20Test The few links we have for Stage, seem ok... So, just on the Prod side.
When I go to add the dashboards back to ES from the blobs I grabbed via _search two entries show up in the Load Menu. This is somewhat confounding behavior. The dashboard works fine, but if I delete one of the entries from the Load Menu the other one doesn't work (presumably because the document was actually deleted). It strikes me that this could be related to a collision with Edwin but since it happened multiple times I'm reasonably sure it's not. So as a workaround for now I will reload all these "missing" documents from the index, and the load menu will have extraneous entries. :jbonacci I'm going to leave that Dash Test in the "missing" state so I have a dashboard with the odd behavior to reference while getting to the bottom of this.
I've added back all the missing dashboards. I'm still investigating the cause and how to get rid of the duplicate entries in the load menu. My guesses are (a) some kind of corruption from the rolling upgrade over the weekend (b) something more insidious related to elasticsearch document/type versioning. If after investigation I find nothing I will assume (a) and blow away/repopulate kibana-int.
http://www.elasticsearch.org/blog/elasticsearch-1-2-1-released/ is probably the cause. I'll be doing rolling updates to 1.2.1.
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.