806337 - BMM: Lock relay boards and add retries

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Dustin J. Mitchell [:dustin] (he/him)]

I need to test that relays support concurrent access.  If not, they need to be locked.  In either case, they need to be retried and status verified to ensure that a "reboot" actually reboots the board.

Comment 1

[Jake Watkins [:dividehex]]

As noted at https://bugzilla.mozilla.org/show_bug.cgi?id=806152#c9 , we should assume the relay boards do not handle multiple connections so locks and retries should be implemented in bmm as suggested.

BMM/lifeguard/mozpool should also be the only place to properly request a panda board reboot, whether via human or code, so states are maintained properly.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

I'm working on this right this very instant.  The relay code already checks the status after power-off and again after power-on, so with a bit of locking this should be good to go.  I'm also going to add some short socket timeouts so we're not stuck waiting to talk to a relay that's not there.

Comment 3

[Jake Watkins [:dividehex]]

We might also want to add a tcp connect timeout to the server side.

Comment 4

[Dustin J. Mitchell [:dustin] (he/him)]

Timeouts are done:
  http://hg.mozilla.org/build/bmm/file/b032b833c3cf/mozpool/bmm/relay.py

I used asyncore so that all socket operations are async, and we can timeout waiting for them.  So connection delays, socket delays, and so on are all handled appropriately, and we can pretty much guarantee that as long as the local CPU isn't tied up, the relay functions will return within their timeout.

Locking is in there, too, but much simpler:
  http://hg.mozilla.org/build/bmm/file/b032b833c3cf/mozpool/bmm/relay.py#l65