772962 - add "finished with automation" email to release process

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, enhancement, P3)

Description

[hwine]

To avoid manual monitoring for when all update verifies have completed (which is when slaves can be released), it would be great to have an email to that effect.

Audience would be release team only.

Comment 1

[[e-mail] (:bhearsum)]

(In reply to Hal Wine [:hwine] from comment #0)
> To avoid manual monitoring for when all update verifies have completed
> (which is when slaves can be released), it would be great to have an email
> to that effect.

I think that bug 763944 covers this specific use case (which actually isn't at the end of the automation, merely the end of the deliverables). However, "finished with automation" is actually a little vague. Technically, we're not done with automation until we get the "ready for release" mail. For betas, this is shortly after the "push to mirrors" builder runs. For final releases it is multiple days after that builder is run.

Is there another use case other than resetting reserved slaves? If not, I'm not sure this serves a purpose since "end up update verify" is a bit of arbitrary point.

Comment 2

[hwine]

(In reply to Ben Hearsum [:bhearsum] from comment #1)
> I think that bug 763944 covers this specific use case 

Agreed, that would be better (best?), this might be "sooner"

> Is there another use case other than resetting reserved slaves? If not, I'm
> not sure this serves a purpose since "end up update verify" is a bit of
> arbitrary point.

atm, the steps which depend on uptake monitoring are somewhat brittle.

Maybe the reality is we (humans) need to start polling when the "start_uptake_monitoring" email comes through, and that's the key email, and we already have it!

If you agree, let me know and I'll update the docs to highlight that step, then close this bug. (tb 14.0b5 let me learn a _lot_ about this since I filed the bug)

Comment 3

[[e-mail] (:bhearsum)]

(In reply to Hal Wine [:hwine] from comment #2)
> (In reply to Ben Hearsum [:bhearsum] from comment #1)
> > I think that bug 763944 covers this specific use case 
> 
> Agreed, that would be better (best?), this might be "sooner"

I'm not opposed to this if it's a lot easier, but I don't think bug 763944 is very hard.

> > Is there another use case other than resetting reserved slaves? If not, I'm
> > not sure this serves a purpose since "end up update verify" is a bit of
> > arbitrary point.
> 
> atm, the steps which depend on uptake monitoring are somewhat brittle.

What makes you think they are brittle? Having an issue with the release you just did doesn't mean they're brittle in general ;).

> Maybe the reality is we (humans) need to start polling when the
> "start_uptake_monitoring" email comes through, and that's the key email, and
> we already have it!

I explicitly think we should not do this. If we're _really_ concerned about uptake monitoring breaking, let's figure out a way to have Nagios or something else poll and bleet if there's an issue. Humans monitoring releases has got to go.

Comment 4

[hwine]

(In reply to Ben Hearsum [:bhearsum] from comment #3)
> What makes you think they are brittle? Having an issue with the release you
> just did doesn't mean they're brittle in general ;).

Well, it's been about 1/2 of my releases - granted, some were corner cases (first in new data center, etc)

And, rail acknowledges that the monitoring does not survive a reconfig. That leaves a hole that can only avoided by multiple people coordinating every time.

> > Maybe the reality is we (humans) need to start polling when the
> > "start_uptake_monitoring" email comes through, and that's the key email, and
> > we already have it!
> 
> I explicitly think we should not do this. If we're _really_ concerned about
> uptake monitoring breaking, let's figure out a way to have Nagios or
> something else poll and bleet if there's an issue. Humans monitoring
> releases has got to go.

Agreed - _if_ there is an alternative. ATM, I see none.

Comment 5

[[e-mail] (:bhearsum)]

I think this is a "nice to have", and nobody has stepped up so I'm P3'ing it.

Comment 6

[[e-mail] (:bhearsum)]

(In reply to Hal Wine [:hwine] from comment #0)
> To avoid manual monitoring for when all update verifies have completed
> (which is when slaves can be released), it would be great to have an email
> to that effect.
> 
> Audience would be release team only.

We don't have to release slaves anymore, which addresses your specific use case. We also run "postrelease" now, which implies "finished with automation" to my mind. I'm calling this FIXED, but feel free to re-open if you disagree.