1463478 - Logging 2.0 - Kibana in devsvcprod issues

Status: RESOLVED WONTFIX

(Data Platform and Tools Graveyard :: Operations, enhancement, P2)

Description

[Harold Woo]

- Devsvc accounts have elasticsearch indexes created incorrectly (lambda es loader creates bugzilla-prod.* named indexes

- Iam role doesnt have permissions by default to create new indexes

https://console.aws.amazon.com/support/v1#/case/?displayId=5068754321&language=en

{ "Action": "es:ESHttp*", "Resource": [ "arn:aws:es:us-west-2:132922822692:domain/bugzilla/", "arn:aws:es:us-west-2:132922822692:domain/bugzilla/.kibana", "arn:aws:es:us-west-2:132922822692:domain/bugzilla/.kibana/*", "arn:aws:es:us-west-2:132922822692:domain/bugzilla/_cluster/health/.kibana", "arn:aws:es:us-west-2:132922822692:domain/bugzilla/.kibana-4", "arn:aws:es:us-west-2:132922822692:domain/bugzilla/.kibana-4/*", "arn:aws:es:us-west-2:132922822692:domain/bugzilla/_cluster/health/.kibana-4", "arn:aws:es:us-west-2:132922822692:domain/bugzilla/_plugin/kibana/*", "arn:aws:es:us-west-2:132922822692:domain/bugzilla/_nodes", "arn:aws:es:us-west-2:132922822692:domain/bugzilla/_aliases", "arn:aws:es:us-west-2:132922822692:domain/bugzilla/_mapping", "arn:aws:es:us-west-2:132922822692:domain/bugzilla/_mget", "arn:aws:es:us-west-2:132922822692:domain/bugzilla/_msearch", "arn:aws:es:us-west-2:132922822692:domain/bugzilla/_search", "arn:aws:es:us-west-2:132922822692:domain/bugzilla/*/_aliases", "arn:aws:es:us-west-2:132922822692:domain/bugzilla/*/_mapping", "arn:aws:es:us-west-2:132922822692:domain/bugzilla/*/_mapping/field/*", "arn:aws:es:us-west-2:132922822692:domain/bugzilla/*/_mget", "arn:aws:es:us-west-2:132922822692:domain/bugzilla/*/_msearch", "arn:aws:es:us-west-2:132922822692:domain/bugzilla/*/_search", "arn:aws:es:us-west-2:132922822692:domain/bugzilla/*/_field_stats", "arn:aws:es:us-west-2:132922822692:domain/bugzilla/_plugin/kibana/api/index_patterns/*" ], "Effect": "Allow" }

--> workaround was to add 
"arn:aws:es:us-west-2:132922822692:domain/bugzilla/*",
and then manually add the correct indexes for data to show up in kibana

Comment 1

[Harold Woo]

use devsvcdev to test

Comment 2

[Harold Woo]

https://github.com/mozilla-services/cloudops-deployment/blob/master/projects/logging/ansible/playbooks/kibana_index_patterns.yml#L48

Comment 3

[Harold Woo]

IAM permissions can include the following if we want to manually create the indexes

.kibana-4/index-pattern/* - for creating kibana indexes
.kibana-4/config/* - for setting default kibana index

Comment 4

[Harold Woo]

https://github.com/mozilla-services/cloudops-deployment/blob/master/projects/bugzilla/puppet/yaml/app/bugzilla.yaml#L17

https://github.com/mozilla-services/cloudops-deployment/blob/master/projects/logging/ansible/playbooks/kibana_index_patterns.yml#L48

https://github.com/mozilla-services/cloudops-deployment/blob/master/projects/logging/ansible/envs/bugzilla.yml

Comment 5

[Harold Woo]

So it looks like bugzilla in devsvcdev had it's elasticsearch indices setup correctly, either that or a new run of logging-env with latest master has fixed something (last run was Apr 10th).  

bugzilla.app.docker.bugzilla-2018-05-22
bugzilla.app.docker.bugzilla-2018-05-23
bugzilla.app.docker.bugzilla-2018-05-20
bugzilla.app.docker.bugzilla-2018-05-21
bugzilla.app.file.nginx_access-2018-05-21
bugzilla.admin.docker.phabbugz-2018-05-20
bugzilla.app.file.nginx_access-2018-05-22
bugzilla.admin.docker.phabbugz-2018-05-21
bugzilla.admin.docker.phabbugz-2018-05-22
bugzilla.app.file.nginx_access-2018-05-20
bugzilla.admin.docker.phabbugz-2018-05-23
bugzilla.app.file.nginx_access-2018-05-23
bugzilla.app.file.nginx_access-2018-05-18
bugzilla.admin.docker.push-2018-05-22
bugzilla.app.file.nginx_access-2018-05-19
bugzilla.admin.docker.phabbugz-2018-05-18
bugzilla.admin.docker.phabbugz-2018-05-19
bugzilla.app.docker.bugzilla-2018-05-19
bugzilla.app.docker.bugzilla-2018-05-18

The problem is that the kibana proxy (which runs on the rawlogs host)'s Iam role: 
arn:aws:iam::554708791447:role/bugzilla/bugzilla-rawlogs-devsvcdev-Role-KZTBRZ0FFALX

needs another elasticsearch permission (not sure which) but adding * works
"arn:aws:es:us-west-2:554708791447:domain/bugzilla/*"

Then I need to go into Kibana, and add the default index, and other indices manually.  Not sure if this was the case before, or if they appeared automatically.  E.g. adding bugzilla.* and selecting Timestamp as the timestamp field works.

Comment 6

[Harold Woo]

Ok the kibana index patterns should be created, and a default should be set by the logging-env pipeline job (https://github.com/mozilla-services/cloudops-deployment/blob/master/projects/logging/ansible/playbooks/kibana_index_patterns.yml#L48)

Comment 7

[Harold Woo]

So this has not been working since at least April 2017. (See commit where the kibana indices creation steps are skipped https://github.com/mozilla-services/cloudops-deployment/commit/d255f59e83043975ab558a324bb5dc5528e5262c)

I've spent some time unsuccessfully trying to fix the playbook (https://github.com/mozilla-services/cloudops-deployment/pull/2062) but it has issues with importing nested dictionaries from puppet config and parsing them with jinja, and also the kibana rest api from version 4 -> 5 has breaking changes.  Also the query to gain information on all of the indices must add pagination logic.  

I am closing this for now since it has been broken for over a year and there is an easy workaround to manually create kibana indices via the UI which is now documented here (https://mana.mozilla.org/wiki/pages/viewpage.action?pageId=77895822)