894124 - self-serve agent needs to recycle the message connection

Status: RESOLVED WORKSFORME

(Release Engineering :: General, defect)

Description

[Nick Thomas [:nthomas] (UTC+12)]

This is spun off from bug 893518. The summary is
* the self-serve agent runs on bm36 and talks to rabbit2.build.scl1
* it keeps a connection open to the message queue to receive new requests (which devs submit to buildapi through tbpl)
* firewall changes in bug 891375 meant that sessions from bm36 were being dropped after 30 minutes of idleness, the app didn't notice and we didn't pick up new changes

The timeout has now been lengthened on the firewall to work around, but the app level fix is to recycle the connection every few minutes like we do elsewhere.

catlee also checked the sysctl settings and we should have appropriate keepalive settings on bm36 already, but something may be overriding them. Also, the agent will probably move to bm81 at some point.