1543989 - Send hg.mo InfluxDB monitoring data to production instance

Status: RESOLVED FIXED

(Developer Services :: Mercurial: hg.mozilla.org, task)

Description

[Connor Sheehan [:sheehan]]

Currently the data is being sent to my test instance in AWS. This bug tracks sending data to the production instance managed by CloudOps, as well as the tear down of the test instance.

Comment 1

[Kim Moir [:kmoir] ET]

If we enable a hook to record the username of the person and the reason logged for the direct push as described in bug 1515119, would that data be sent to InfluxDB where we could generate alerts or a report?

Comment 2

[Connor Sheehan [:sheehan]]

It's certainly possible, but I'm not sure InfluxDB is the best place for this. Influx is intended for ingesting time-series data that is gathered at regular intervals. For example, monitoring how the load on a server changes over time. Storing this push information in a times series DB seems out-of-place to me.

Comment 3

[Connor Sheehan [:sheehan]]

Attachment: ansible/monitoring-agent: add encrypted InfluxDB credentials (Bug 1543989) r?lars

This commit adds a set of default Ansible variables in the
monitoring-agent role, for sending data to Mozilla's
production InfluxDB instance.

Since sops doesn't yet know how to parse TOML, we store
the secrets as YAML Ansible defaults.

Comment 4

[Connor Sheehan [:sheehan]]

Attachment: ansible/monitoring-agent: send monitoring data to production InfluxDB instance (Bug 1543989) r?lars

This commit tells Telegraf to send data to the production InfluxDB
instance. At the moment we will send data to both the test instance
and the production instance, to verify all the data is correctly
sent.

Comment 5

[K Lars Lohn [:lars] [:klohn]]

Does this warrant some documentation? For example, are there manual actions that need to be taken to periodically change keys or is a set once and forever leave it alone?

Comment 6

[Connor Sheehan [:sheehan]]

Yea, we will probably want some documentation for our InfluxDB monitoring. I'll file a bug.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/baa4bb1271e8
ansible/monitoring-agent: add encrypted InfluxDB credentials r=lars
https://hg.mozilla.org/hgcustom/version-control-tools/rev/32df72677efb
ansible/monitoring-agent: send monitoring data to production InfluxDB instance r=lars

Comment 8

[Connor Sheehan [:sheehan]]

Syntax error in the config file - reopening.

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/d7b0532a4e6a
ansible/monitoring-agent: fix production InfluxDB credentials

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/f13310e10021
ansible/monitoring-agent: remove quotes around boolean variable

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/2a93b060dd7e
ansible/monitoring-agent: remove retention policy from database name