All users were logged out of Bugzilla on October 13th, 2018

A few Kibana dashboards (stored in es) get 404 errors

RESOLVED WONTFIX

Status

RESOLVED WONTFIX
4 years ago
3 years ago

People

(Reporter: kparlante, Assigned: whd)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

4 years ago
Edwin noticed these dashboards didn't load from the file menu:

https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/elasticsearch/PROD%20-%20QA%20Request%20Time

https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/elasticsearch/PROD%20-%20QA%20Dashboard

I noticed the same for these two:
https://kibana.fxa.us-west-2.prod.mozaws.net/#/dashboard/elasticsearch/FraudDetection

https://kibana.fxa.us-west-2.prod.mozaws.net/#/dashboard/elasticsearch/More%20Endpoint%20Counts

whd noticed they are in the index and you can see them with /_search

Edwin observed that he can't delete them.

fwiw, Edwin has backups of the two on the shared stack, the last one is actually saved in puppet-config. As whd points out, would be nice to know what went wrong.
Also here: https://kibana.shared.us-west-2.prod.mozaws.net/#/dashboard/elasticsearch/Dash%20Test

The few links we have for Stage, seem ok...
So, just on the Prod side.
(Assignee)

Comment 2

4 years ago
When I go to add the dashboards back to ES from the blobs I grabbed via _search two entries show up in the Load Menu. This is somewhat confounding behavior. The dashboard works fine, but if I delete one of the entries from the Load Menu the other one doesn't work (presumably because the document was actually deleted). It strikes me that this could be related to a collision with Edwin but since it happened multiple times I'm reasonably sure it's not.

So as a workaround for now I will reload all these "missing" documents from the index, and the load menu will have extraneous entries. :jbonacci I'm going to leave that Dash Test in the "missing" state so I have a dashboard with the odd behavior to reference while getting to the bottom of this.
(Assignee)

Comment 3

4 years ago
I've added back all the missing dashboards. I'm still investigating the cause and how to get rid of the duplicate entries in the load menu. My guesses are (a) some kind of corruption from the rolling upgrade over the weekend (b) something more insidious related to elasticsearch document/type versioning. If after investigation I find nothing I will assume (a) and blow away/repopulate kibana-int.
(Assignee)

Comment 4

4 years ago
http://www.elasticsearch.org/blog/elasticsearch-1-2-1-released/ is probably the cause. I'll be doing rolling updates to 1.2.1.
(Assignee)

Updated

3 years ago
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.