909083 - buildapi needs something to restart it

Status: RESOLVED INVALID

(Release Engineering :: General, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

During today's maintenance a db connection used by buildapi01 went away during work on the network. buildapi crashed out at this point, and we don't have anything to bring it back up again (no active puppet, no supervisord).

Comment 1

[Hal Wine [:hwine] use NI!]

Grabbing to apply some bandaids

Comment 2

[Hal Wine [:hwine] use NI!]

Attachment: hourly_check -- cronjob to ensure buildapi is running

BANDAID - until something better is done. Will email release@ if it finds buildapi down, and page hwine if it doesn't come up.

Comment 3

[Hal Wine [:hwine] use NI!]

Attachment: crontab.buildapi01 -- additions to existing one

BANDAID - run the hourly check and email release@ if any issues

Comment 4

[Hal Wine [:hwine] use NI!]

Attachment: hourly_check -- cronjob to ensure selfserve agent is running

BANDAID - script to restart selfserve-agent if it is not running -- this runs on buildbot-master36, this bug seemed closest to mark that fact

Comment 5

[Hal Wine [:hwine] use NI!]

Attachment: crontab.bm36 - lines added to existing crontab

BANDAID - restart selfserve agent if not running

Comment 6

[Hal Wine [:hwine] use NI!]

bandaids applied -- please remove when proper solution is applied

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

moved to releng cluster

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

So now we have Apache + WSGI, so making a request relaunches buildapi if required ?

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

If it "crashes" that's caught either by mod_wsgi (Python exception) or by the Apache parent process (segfault), and restarted immediately.

If it gets wedged somehow, I believe the request would eventually time out, again either at the mod_wsgi or Apache levels.  But I haven't seen this happen so I'm not sure.