840524 - improve release runner failure e-mail

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect)

Description

[[e-mail] (:bhearsum)]

1) Should include part or all of the log.
2) Should tell somewhere exactly where to find the full log on the machine it ran on.

Comment 1

[hwine]

#2 needs to include either - which credentials to use, or (better, imo), cltbld should have read access to the logs

Comment 2

[hwine]

Oh, or even better - iirc, the supervisord web UI provides access to some logs. That would allow the possibility of an eventual tunnel for non-build-vpn release runners.

Comment 3

[[e-mail] (:bhearsum)]

This caused delay during 19.0b1 - we really need to fix it soon.

Comment 4

[John Hopkins (:jhopkins)]

I received this warning email from shipit:

 'A buildbot master reconfig started at Wed, 20 Feb 2013 10:02:13 PST has been running for
 0:10:00.382857 without completing.'

Can we enhance this email to include the name of the buildbot master whose reconfig is stuck?

Comment 5

[[e-mail] (:bhearsum)]

(In reply to Hal Wine [:hwine] from comment #1)
> cltbld should have read access to the logs

100% agreed. These aren't puppetized yet (bug 836289) so I went ahead and did this by hand:
[cltbld@buildbot-master36 ~]$ tail -F /var/log/supervisor/release-runner.log
2013-02-22 13:45:14,910 - DEBUG - Sleeping for 30 seconds before polling again
2013-02-22 13:45:44,909 - DEBUG - Fetching release requests
2013-02-22 13:45:45,050 - INFO - No new releases: {'releases': []}
2013-02-22 13:45:45,050 - DEBUG - Sleeping for 30 seconds before polling again
2013-02-22 13:46:15,051 - DEBUG - Fetching release requests
2013-02-22 13:46:15,175 - INFO - No new releases: {'releases': []}
2013-02-22 13:46:15,175 - DEBUG - Sleeping for 30 seconds before polling again
2013-02-22 13:46:45,175 - DEBUG - Fetching release requests
2013-02-22 13:46:45,298 - INFO - No new releases: {'releases': []}
2013-02-22 13:46:45,298 - DEBUG - Sleeping for 30 seconds before polling again

Comment 6

[[e-mail] (:bhearsum)]

Attachment: send some output with the log

Completely untested.

Comment 7

[Rail Aliiev [:rail]]

Comment on attachment 717314 [details] [diff] [review]
send some output with the log

I think this is the easiest way to go.

As a possible another way to go would be sending CRITICAL and ERROR log messages via email.

Comment 8

[[e-mail] (:bhearsum)]

Attachment: send logs

(In reply to Rail Aliiev [:rail] from comment #7)
> Comment on attachment 717314 [details] [diff] [review]
> send some output with the log
> 
> I think this is the easiest way to go.
> 
> As a possible another way to go would be sending CRITICAL and ERROR log
> messages via email.

Hmm, I don't think this will catch all errors? Eg, uncaught exceptions.

Here's a tested version of my patch which seems to work.

Comment 9

[[e-mail] (:bhearsum)]

I updated the checkout on bm36 and restarted release runner is supervisord.

(In reply to Hal Wine [:hwine] from comment #2)
> Oh, or even better - iirc, the supervisord web UI provides access to some
> logs. That would allow the possibility of an eventual tunnel for
> non-build-vpn release runners.

I think this is explicitly a separate bug. I don't feel strongly about it. If you do, please file it.

(In reply to Hal Wine [:hwine] from comment #1)
> #2 needs to include either - which credentials to use, or (better, imo),
> cltbld should have read access to the logs

cltbld has access to the logs.

(In reply to John Hopkins (:jhopkins) from comment #4)
> I received this warning email from shipit:
> 
>  'A buildbot master reconfig started at Wed, 20 Feb 2013 10:02:13 PST has
> been running for
>  0:10:00.382857 without completing.'
> 
> Can we enhance this email to include the name of the buildbot master whose
> reconfig is stuck?

This is also a separate bug IMO. I filed it in bug 847380.