1376537 - Can Balrog push release event notifications?

Status: RESOLVED MOVED

(Release Engineering Graveyard :: Applications: Balrog (backend), enhancement, P3)

Description

[Bob Micheletto [:bobm]]

The services operations team would like to track Fx release events on our various dashboards to see how client releases influence server traffic.  

Would it be possible to push the release events, or subscribe to them from RSS or some other mechanism?

Comment 1

[Bob Micheletto [:bobm]]

Assigning to bhearsum per :mostlygeek

Comment 2

[[e-mail] (:bhearsum)]

I'd like to understand the granularity of data that you're looking for a bit more. Balrog itself doesn't decide things like "we are shipping 54.0 now", we have outside systems (Ship It & Release Promotion) that decide that, and make that change in Balrog. These systems are best able to answer this basic question. There's also https://product-details.mozilla.org/1.0/, which can be polled to discover when we ship something new.

However, releases are not always in a simple "shipped" or "not shipped" state. We typically ship at a throttle rate of 25% to begin with, then back off to 0%, and finally raise it up to 100% if all is well. These changes are usually made directly by a human in Balrog, and the aforementioned systems are not aware of them. If you need this level of detail, Balrog is currently the only source for it.

I think it might be good if we had a chat on Vidyo to make sure we're both on the same page.

Comment 3

[Bob Micheletto [:bobm]]

(In reply to Ben Hearsum (:bhearsum) from comment #2)
> I'd like to understand the granularity of data that you're looking for a bit
> more. Balrog itself doesn't decide things like "we are shipping 54.0 now",
> we have outside systems (Ship It & Release Promotion) that decide that, and
> make that change in Balrog. These systems are best able to answer this basic
> question. There's also https://product-details.mozilla.org/1.0/, which can
> be polled to discover when we ship something new.

When new versions are shipping we would like to know.  The timestamps in the product-details json are date only.  Date and time would be most useful.  We can probably get those particular events when items are added to the product delivery S3 bucket.

> However, releases are not always in a simple "shipped" or "not shipped"
> state. We typically ship at a throttle rate of 25% to begin with, then back
> off to 0%, and finally raise it up to 100% if all is well. These changes are
> usually made directly by a human in Balrog, and the aforementioned systems
> are not aware of them. If you need this level of detail, Balrog is currently
> the only source for it.

We are also definitely interested in these throttle events.  

> I think it might be good if we had a chat on Vidyo to make sure we're both
> on the same page.

We can try a Vidyo chat.  I'm at the all hands so the conf WIFI might not be 100% up to the task.

Here are the specific things we are looking for:
* Events that describe when a new version of a product has been released, regardless of throttle.
* Events that describe when the throttle level has changed for a given product.

Comment 4

[[e-mail] (:bhearsum)]

Bob, Benson and I talked about this a bit more today. Bob's specific use case is that he wants to be able to overlay release shipping and throttling events on traffic graphs for other services in Datadog. The relevant information to him is the Background Rate, Firefox version being shipped, and the timestamp of the change. We already store all of this in Balrog's history tables, so the main thing here is getting it pushed to a useful place.

Datadog supports many different data sources, but right now we're leaning towards sending SNS events because it should be pretty easyto do on the Balrog side (eg: https://stackoverflow.com/questions/40802200/example-script-to-send-sms-via-aws-sns-using-boto), and trivial to set-up as a data source in Datadog (http://docs.datadoghq.com/integrations/awssns/).

One thing I'm slightly concerned about is putting SNS notifications in the critical path of writing changes to the DB. Benson suggested setting a very low timeout on them (maybe as low as one second), and simply ignoring errors if the message fails to send. This should be fine, and is definitely no worse than what we do for e-mail notification.

We've already got the concept of "change monitors" for e-mail notification (https://github.com/mozilla/balrog/blob/741a56cc72d3e9e2f316f099c5de36900ad0d2ce/auslib/db.py#L2543) so we shouldn't much architecture work or scaffolding.

Still to do: decide on exactly what needs to go in the message, and to format it

Comment 5

[[e-mail] (:bhearsum)]

Dug into this a bit more today. After some help and pointers from Bob, it's pretty clear that we want to create a topic, and create a webhook subscription for it that points at Datadog (as documented on https://docs.datadoghq.com/integrations/awssns/#receiving-sns-messages-in-the-event-stream). This should allow us to overlay events onto Datadog graphs. The events _should_ contain the SNS message if I've understood things correctly.

My next step here will be to give this all a try by hand to make sure these assumptions are correct.

Comment 6

[[e-mail] (:bhearsum)]

I think it's unlikely I'll be looking at this before Q4, possibly not until next year.

Comment 7

[[e-mail] (:bhearsum)]

I don't think this is going to make the cut for Q4, unless someone tells me it's more important than other things on the docket. Going to unassign this for now.

Comment 8

[[e-mail] (:bhearsum)]

Bob, I realize this is a Very Old bug at this point, but is it still relevant?

Comment 9

[Bob Micheletto [:bobm]]

(In reply to [e-mail] (:bhearsum) from comment #8)

Bob, I realize this is a Very Old bug at this point, but is it still relevant?

We still want it. Though, I'd prefer if the annotations came by way of the Grafana Annotations API.

Comment 10

[[e-mail] (:bhearsum)]

(In reply to Bob Micheletto [:bobm] from comment #9)

(In reply to [e-mail] (:bhearsum) from comment #8)

Bob, I realize this is a Very Old bug at this point, but is it still relevant?

We still want it. Though, I'd prefer if the annotations came by way of the Grafana Annotations API.

Alright, we're moving Balrog bugs to Github Issues, so I've opened https://github.com/mozilla-releng/balrog/issues/1127 for this.