Send vcs sync logs to CloudWatch

RESOLVED FIXED

Status

Developer Services
Servo VCS Sync
RESOLVED FIXED
7 months ago
7 months ago

People

(Reporter: gps, Assigned: gps)

Tracking

Details

MozReview Requests

Submitter Diff Changes Open Issues Last Updated
Loading...
Error loading review requests:

Attachments

(2 attachments)

(Assignee)

Description

7 months ago
We currently don't have off-machine backup of logs from vcs sync. Let's enable that by sending logs to CloudWatch.
Comment hidden (mozreview-request)
Comment hidden (mozreview-request)
(Assignee)

Comment 3

7 months ago
I also have a GitHub PR at https://github.com/mozilla-platform-ops/devservices-aws/pull/43 to create the CloudWatch log group and update the IAM policy to allow writing to it from the vcssync EC2 instance.

There are definitely some things we can tweak about the implementation. The naming convention is a big one. Right now the log stream name is i-07... because that is the EC2 instance. We may want something more static. Alternatively (and this depends heavily on how we'll use CloudWatch), we may want to send all systemd logs to somewhere like a "systemd" log group and use subscriptions/filters to siphon relevant events to other destinations.

At this point, I just want log events going off host in near real time so we have a trusted backup and can easily derive metrics, from which we can create CloudWatch alarms so we are notified when things occur.
I think this is a great idea!  And for right now, I don't think the naming convention is all that important.  I'd like to see how this works out before we get into the weeds on naming and filtering for events.

Comment 5

7 months ago
mozreview-review
Comment on attachment 8867007 [details]
ansible/common: adjust permissions on /var/log/journal;

https://reviewboard.mozilla.org/r/138604/#review142088
Attachment #8867007 - Flags: review?(jwatkins) → review+

Comment 6

7 months ago
mozreview-review
Comment on attachment 8867008 [details]
ansible: add role to send journald to CloudWatch (bug 1364231);

https://reviewboard.mozilla.org/r/138606/#review142090
(Assignee)

Comment 7

7 months ago
For the record, the reason I waded into this this week is because of bug 1363635. I realized we didn't have persisted logs for vcs sync nor adequate monitoring of vcs sync and autoland. This bug solves at least the log persistence problem. It is also possible to perform metrics and alerting from CloudWatch. If the logs are going to CloudWatch in near real time, that opens up some interesting possibilities. I want to also explore the monitoring/alerting aspect. Although I understand if we don't commit to that if there is something better.

Comment 8

7 months ago
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/7cb50653f005
ansible: add role to send journald to CloudWatch ; r=dividehex
Status: ASSIGNED → RESOLVED
Last Resolved: 7 months ago
Resolution: --- → FIXED
(Assignee)

Comment 9

7 months ago
Comment on attachment 8867008 [details]
ansible: add role to send journald to CloudWatch (bug 1364231);

dividehex granted r+ via IRC.
Attachment #8867008 - Flags: review?(jwatkins) → review+
You need to log in before you can comment on or make changes to this bug.