789213 - Delay SUTAgent "ready" until the SDCard is ready

Status: RESOLVED FIXED

(Testing Graveyard :: SUTAgent, defect)

Description

[Justin Wood (:Callek)]

So it is my theory that the relatively frequent "SDCard not mounted" issues stem from us, with SUTAgent trying to access the SDCard too early, and causing Android to barf.

I suspect that watching for ACTION_MEDIA_MOUNTED first will save us here.

The short-term, easy fix is to never open/listen on the two sockets the SUTAgent uses if the sdcard doesn't mount for some reason, this will prevent us from doing any triage at all via SUT when its not mounted, but does make sense from our automation perspective, we need the SDCard mounted at almost every step of the way.

The longer term fix is to still open the ports and accept commands that *can not* touch the SDCard on the command port, and all commands on the dialback port [so that we can diagnose deeper if need be, with commands that *have the potential* to touch the sdcard). In this scenario on command port any commands that *could* touch the SDCard would throw an agent error.

See- http://stackoverflow.com/questions/3417161/android-intent-for-sdcard-ready

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: start sutagent after MEDIA_MOUNTED instead of BOOT_COMPLETED (1.0)

the only thing I can think of as a problem with this is on a device that doesn't have a media card to mount.  Otherwise I have seen consistently that we usually launch SUTAgent before MEDIA_MOUNTED and in the few times I tested we now will start SUTAgent about a minute after receiving the MEDIA_MOUNTED intent.

Comment 2

[William Lachance (:wlach)]

Comment on attachment 659002 [details] [diff] [review]
start sutagent after MEDIA_MOUNTED instead of BOOT_COMPLETED (1.0)

This looks like the right thing: http://stackoverflow.com/questions/8248617/run-a-service-when-device-starts-after-sd-finishes-loading

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/809b8b4c90e6

Comment 4

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/809b8b4c90e6

Comment 5

[Justin Wood (:Callek)]

Joel, Will.

Without reopening/new bug just yet, are we *sure* this code does as we expect. I just witnessed, first-hand a device which I PDU rebooted, after it had new SUTAgent Version, however it came up without a ready SDCard.

I vagualy remember being told that SUTAgent does not automatically start with the device, since its not installed as a service, and instead, watcher starts it. If that is correct, do we need to modify how we applied this change in SUTAgent itself to - itself - await the MOUNTED intent broadcast, or do we need to develop a way to update watcher properly, or both?

Comment 6

[William Lachance (:wlach)]

(In reply to Justin Wood (:Callek) from comment #5)
> Joel, Will.
> 
> Without reopening/new bug just yet, are we *sure* this code does as we
> expect. I just witnessed, first-hand a device which I PDU rebooted, after it
> had new SUTAgent Version, however it came up without a ready SDCard.
> 
> I vagualy remember being told that SUTAgent does not automatically start
> with the device, since its not installed as a service, and instead, watcher
> starts it. If that is correct, do we need to modify how we applied this
> change in SUTAgent itself to - itself - await the MOUNTED intent broadcast,
> or do we need to develop a way to update watcher properly, or both?

It is true that the watcher service will, by default, start sutagent if it's not running, but only on a timer (the same one we were using for the ping checks). Looks like we do this about 60 seconds after startup... so yeah, maybe there's a race between watcher startup and the sdcard mounted intent being broadcast.

Maybe we should have the watcher check if the sdcard is mounted before trying to (re)start the agent.

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

Very likely it is the watcher starting it up.  The SUTAgent is registered to a broadcast receiver.  When we receive the MEDIA_MOUNTED intent via broadcast, the system (android) will take action on what is registered to listen for that.  

I would like to know how frequent this is now vs the frequency before the 1.13 sutagent deployment.  I know we don't have perfect stats for this, but looking at a few days of infrastructure related failures might give us a rough estimate.