1042056 - release runner failure e-mail should be sent to relman or release-drivers

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect)

Description

[[e-mail] (:bhearsum)]

We had releases fail to start early this morning because of failures during release runner. They didn't start, and no indication was given, except to [e-mail]. This is confusing, and makes it impossible for anyone else to try and debug on their own.

We should also increase the number of lines of context included. I'd suggest 100 lines for now, which should end up including all of the release sanity output.

Comment 1

[Rail Aliiev [:rail]]

Maybe use last 1000 lines and attach as text? 100 won't be enough I bet.

Comment 2

[[e-mail] (:bhearsum)]

(In reply to Rail Aliiev [:rail] from comment #1)
> Maybe use last 1000 lines and attach as text? 100 won't be enough I bet.

Yeah, maybe. My only concern is that you may end up with context from a previous run of release runner. Maybe we can have max 1000, but walk backwards only until we see "Fetching release requests" to avoid getting a previous run?

Comment 3

[Rail Aliiev [:rail]]

... and replace the shell wrapper with something better :)

Comment 4

[[e-mail] (:bhearsum)]

Attachment: send failure mail to release-drivers

This should get the mail sent to the right place. bug 872316 is a better place to track improving the log, which I plan to pick up soon.

Comment 5

[[e-mail] (:bhearsum)]

Comment on attachment 8462696 [details] [diff] [review]
send failure mail to release-drivers

Hmm, Rail is away for awhile. Probably good to have someone from RelMan sign off on this too.

Comment 6

[Sylvestre Ledru [:Sylvestre]]

Comment on attachment 8462696 [details] [diff] [review]
send failure mail to release-drivers

LGTM

Comment 7

[Chris AtLee [:catlee]]

Comment on attachment 8462696 [details] [diff] [review]
send failure mail to release-drivers

Review of attachment 8462696 [details] [diff] [review]:
-----------------------------------------------------------------

::: manifests/moco-config.pp
@@ +189,5 @@
>  
>      $vmwaretools_version = "9.4.0-1280544"
>      $vmwaretools_md5 = "4a2d230828919048c0c3ae8420f8edfe"
>      $releaserunner_notify_from = "Release Eng <[e-mail]>"
> +    $releaserunner_notify_to = "<[e-mail]>"

I'm a little worried about having the email address as a bare address enclosed by <>. Is this supported?  Otherwise, LGTM.

Comment 8

[[e-mail] (:bhearsum)]

Comment on attachment 8462696 [details] [diff] [review]
send failure mail to release-drivers

(In reply to Chris AtLee [:catlee] from comment #7)
> Comment on attachment 8462696 [details] [diff] [review]
> send failure mail to release-drivers
> 
> Review of attachment 8462696 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: manifests/moco-config.pp
> @@ +189,5 @@
> >  
> >      $vmwaretools_version = "9.4.0-1280544"
> >      $vmwaretools_md5 = "4a2d230828919048c0c3ae8420f8edfe"
> >      $releaserunner_notify_from = "Release Eng <[e-mail]>"
> > +    $releaserunner_notify_to = "<[e-mail]>"
> 
> I'm a little worried about having the email address as a bare address
> enclosed by <>. Is this supported?  Otherwise, LGTM.

I added a string in front of the <> upon landing.

Comment 9

[[e-mail] (:bhearsum)]

Puppet will deploy this soon, and the next run of release runner will pick it up.