856580 - release runner shouldn't send failure mail when dying during polling

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect)

Description

[[e-mail] (:bhearsum)]

That is, it shouldn't send mail after each individual fail to poll. Because of the stort sleep time we added in bug 849783, this can cause mail to pile up and unnecessary panic. However, it should send mail if it doesn't poll successfully after a longer period of time. Maybe 15-20min?

Comment 1

[[e-mail] (:bhearsum)]

Attachment: don't e-mail on polling failures

This doesn't get us sending mail after extended failures polling, but I'm not sure how to implement that in the current architecture where supervisor handles the retrying...it would be easier in some ways if release-runner.py was a long running process. I think this is good enough for now though.

Comment 2

[[e-mail] (:bhearsum)]

Attachment: use exit code 5

11:16 < rail> bhearsum: afaik, when python raises an exception by default it exits 1. this means that everything below will be ignored, won't it?
11:17 < rail> like line 379
11:24 < bhearsum> huh, let's me test that
11:24 < bhearsum> i thought it returned some really high number...
11:25 < bhearsum> you're right
11:25 < bhearsum> ok, we can return 5 for polling failure instead then

Comment 3

[Rail Aliiev [:rail]]

Comment on attachment 734699 [details] [diff] [review]
use exit code 5

Hunk #2 may be optional in this case.

Comment 4

[[e-mail] (:bhearsum)]

(In reply to Rail Aliiev [:rail] from comment #3)
> Comment on attachment 734699 [details] [diff] [review]
> use exit code 5
> 
> Hunk #2 may be optional in this case.

Yeah...may as well keep it though - now we can distinguish between polling failed, sendchange failed, and other failures.

Comment 5

[[e-mail] (:bhearsum)]

Comment on attachment 734699 [details] [diff] [review]
use exit code 5

Landed and update the running instance on bm36.