1388932 - QE needs a notification that updates are available on beta/release channel

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, enhancement, P2)

Description

[Nick Thomas [:nthomas] (UTC+12)]

We used to have an email with subject
  firefox beta 55.0 updates are available on the beta channel now <EOM>
and then signoffs came along, and a fix was made [1]. 

There's no notification when the scheduled change is actioned in Balrog, at least not one that reaches the r-d list that QE are monitoring. They use that to start verifying the production channel.

[1] https://github.com/mozilla-releng/releasetasks/commit/1aefa5913f6468cb789ba1429aada60fbffdab8e

Comment 1

[Nick Thomas [:nthomas] (UTC+12)]

Balrog might be the best place to fix this, and bug 1337892 is relevant.

Comment 2

[Mihai Tabara [:mtabara]⌚️GMT]

To begin with, indeed this would be a very useful piece!

I could be wrong but this is not something trivial to add in the current notifications system from releasetasks. Shouldn't be too complex either but with Ship-it v2 future still-to-be-decided, this is definitely something we should be looking into fixing short-term.

Context:
Before we had signoffs, we could rely on the "publish to balrog" builders to enact to be able to say everything is fine and submit notification. But ever since then signoffs were turned on we've basically lost this ability from current workflow. Current notifications system is fully contained within the releasetasks graph but this intended notification is chained to an external entity (Balrog). So unless we make Balrog send some sort of Pulse/API call/whatever message that we can read from the graph or something like that, there's pretty much nothing we can do. At least not with the current arch. 

Solutions:
1. Wait for bug 1337892 to be implemented. That sounds like a very clean and good solution, Balrog-side.

2. Ideally, if Ship-it v2 will fix this coordination in the glorious future as it'll have all the actors involved on the same page.

3. Until 1) or 2) get solved we can continue to send  "This is live now" short emails at the RelMan push-to-request emails. It's not great but it should cover our backs until we can automate.

Comment 3

[[e-mail] (:bhearsum)]

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #2)
> To begin with, indeed this would be a very useful piece!
> 
> I could be wrong but this is not something trivial to add in the current
> notifications system from releasetasks. Shouldn't be too complex either but
> with Ship-it v2 future still-to-be-decided, this is definitely something we
> should be looking into fixing short-term.
> 
> Context:
> Before we had signoffs, we could rely on the "publish to balrog" builders to
> enact to be able to say everything is fine and submit notification. But ever
> since then signoffs were turned on we've basically lost this ability from
> current workflow. Current notifications system is fully contained within the
> releasetasks graph but this intended notification is chained to an external
> entity (Balrog). So unless we make Balrog send some sort of Pulse/API
> call/whatever message that we can read from the graph or something like
> that, there's pretty much nothing we can do. At least not with the current
> arch. 
> 
> Solutions:
> 1. Wait for bug 1337892 to be implemented. That sounds like a very clean and
> good solution, Balrog-side.

I agree that this would be clean. This is not something that's on the radar right now though. If this is needed in the near-term, I don't think this is viable.

It's likely that I think https://bugzilla.mozilla.org/show_bug.cgi?id=1376537 will happen sooner, which would make it relatively easy to write something that subscribes to those events and sends e-mail in response. There might even be some fancy way to do it in SNS or Lambda and avoid the need to deploy anything ourselves.

> 2. Ideally, if Ship-it v2 will fix this coordination in the glorious future
> as it'll have all the actors involved on the same page.
> 
> 3. Until 1) or 2) get solved we can continue to send  "This is live now"
> short emails at the RelMan push-to-request emails. It's not great but it
> should cover our backs until we can automate.

As another option for the short term, something could be written that polls Balrog and watches for a mapping change on the correct Rule. It could be run as a Task that is downstream of the Schedule Publishing Task.

Comment 4

[Jordan Lund (:jlund)]

From the description, I believe this was taken care of elsewhere.

We have emails like firefox 66.0b13 build1/mozilla-beta updates are ready for signoff in Balrog! being sent to release-signoff@m.c

We also have #releaseduty notifications.

Since we are likely integrating balrog signoffs through shipit signoffs, I think we can close this. Please re-open if I misunderstood.

Comment 5

[Jordan Lund (:jlund)]

FTR esr60 merge bug: Bug 1453993 - Tracking bug for 2018-04-30 migration work