810395 - get release runner working under supervisord

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect, P2)

Description

[[e-mail] (:bhearsum)]

* create a supervisord config with logging enabled
* create a simple wrapper which:
  * starts release runner
  * checks the return code
  * in case of failure:
    * send an email (and update the status in the DB?) about the failure with last N lines of code
    * sleep 3 days (in case of weekend)


Also need to make sure that everything works w/out a tty available.

Comment 1

[Rail Aliiev [:rail]]

Just to update the WIP status. Everything listed above is done. We have some issues with logging though, see https://bugzilla.mozilla.org/show_bug.cgi?id=810393#c2. Maybe it's related to the fact that there no tty attached in this scenario.

Comment 2

[Rail Aliiev [:rail]]

It worked finally. We hit 2 issues:

1) the config file used ~/ssh/path_to_key. It turns out that HOME is not set under supervisor. changing the path to /home/cltbld/.ssh/path_to_key fixed the problem.

2) ~/.ssh/authorized_keys on bm25 and bm34 were out of sync (puppet issues). Restarting puppet on bm25 and adding the key to bm34 fixed the problem.