1925614 - Create/update panels in Grafana to monitor Elasticsearch 8 cluster health

Status: RESOLVED FIXED

(Socorro :: General, task, P2)

Description

[Bianca Danforth [:bdanforth]]

This bug is part of the story ticket, OBS-308, to deploy and validate Elasticsearch 8 in stage.

Panels should be created/updated in stage and prod environments.

One major change as a result of upgrading ES is that we're moving from a self-hosted to a hosted cluster in Elastic Cloud (see Bug 1925594). While we will likely get a health dashboard out of the box from Elastic Cloud (and it will serve as a good reference), we ultimately want all our service health panels in our Grafana dashboard.

This may require adding a new Grafana data source, though we can check to see if other teams at Mozilla that host an ES cluster in Elastic Cloud have already done this (e.g. Fakespot).

Comment 1

[Bianca Danforth [:bdanforth]]

Some things we can track:

• Things we're currently tracking in our existing, self-hosted Elasticsearch cluster in Grafana, which may or may not include:
• Disk usage, search latency, and indexing speed
  • This is helpful feedback for adjusting the size, zone_count and other attributes of the hot data tier of the cluster.

Comment 2

[Bianca Danforth [:bdanforth]]

Removing dependency on Bug 1925594, since an initial deployment has already been created which can be used to update the dashboard/panels.

Comment 3

[Bianca Danforth [:bdanforth]]

After some investigation and discussion, this ticket is blocked on one part of https://mozilla-hub.atlassian.net/browse/SRELEAD-447 . I added a comment in that ticket mentioning this and requesting that the Socorro dashboard migration be prioritized among the others if possible.

Edit: relud suggested a new fourth option we hadn't considered, and we're going to give that a shot since it decouples this effort from the Grafana/Prometheus migration effort. See https://mozilla-hub.atlassian.net/browse/SRELEAD-447?focusedCommentId=990389 for more information.

Comment 4

[Bianca Danforth [:bdanforth]]

Attachment: [mozilla-it/webservices-infra] feat(socorro): add nginx proxy between telegraf and es 8 cluster (bug-1925614) (#3757)

Comment 5

[Bianca Danforth [:bdanforth]]

biancadanforth merged PR [mozilla-it/webservices-infra]: feat(socorro): add nginx proxy between telegraf and es 8 cluster (bug-1925614) (#3757) in 0425331.

Comment 6

[Bianca Danforth [:bdanforth]]

Attachment: [mozilla-it/webservices-infra] chore(socorro): increase number of nodes in stage es 8 cluster (bug-1925614) (#3804)

Comment 7

[Bianca Danforth [:bdanforth]]

biancadanforth merged PR [mozilla-it/webservices-infra]: chore(socorro): increase number of nodes in stage es 8 cluster (bug-1925614) (#3804) in b14b039.

Comment 8

[Bianca Danforth [:bdanforth]]

I wanted to explain what we did here (relud and I) for posterity before closing the ticket out:

First PR

In order to get metrics from the new ES 8 cluster in Elastic Cloud into Grafana, we added a second server to the Telegraf config to collect the same metrics from ES 8 as from ES 1.4. Unlike ES 1.4, ES 8 requires authentication for API requests, but adding a secret to the existing Telegraf secrets object was non-trivial, so part of the work here was to add an nginx proxy in between Telegraf and the ES 8 cluster to add an Authorization header with the ES username/password encoded into a bearer token.

This configuration will need to be enabled for production as part of setting up the ES 8 cluster, but that is captured in a separate ticket (see OBS-309 > [TBF] Configure Telegraf to collect EC metrics in prod).

Work inside Grafana

Adding the new ES 8 cluster as a server for Telegraf meant any Elasticsearch panels in the Socorro GCP dashboard basically updated automatically using the same query for each panel (since series were grouped by tags like cluster name or node name). The only exception here is the Node CPU utilization panel, which was originally pulling from Kubernetes' container metrics for the ES 1.4 containers. This metric can be pulled from the ES /_node/stats/process API, so the panel was updated to pull data from there instead.

Second PR

After comparing the data between the ES 1.4 and ES 8 cluster, I noticed that the new ES 8 cluster had a cluster health status of yellow, a bunch of unassigned shared and about twice the node heap utilization % as the ES 1.4 cluster. This was because the ES 8 cluster only had one node, while the ES 1.4 cluster had 5 nodes. Updating the EC deployment to increase the number of nodes, number of availability zones and RAM per availability zone brought the ES 8 cluster metrics in line with the ES 1.4 metrics.

The combination of all of this work means that this ticket is now complete, so I'm marking it as resolved.