1159577 - Autophone - device issue 2015-04-28 nexus-5-kot49h-{1,2,3,4}

Status: RESOLVED FIXED

(Testing Graveyard :: Autophone, defect)

Description

[Bob Clary [:bc] (inactive)]

At approximately 7PM PT, nexus-5-kot49h-3 in the autophone rack in the mountain view qa lab stopped responding. Rebooting the host did not recover the device over usb. adb devices does show it but adb shell hangs when attempting to connect to it.

It may need to have its usb connection disconnected then reconnected or perhaps it just needs rebooting.

mschifer, stephend, or geo: can you please check on it tomorrow morning?

Comment 1

[Bob Clary [:bc] (inactive)]

Out of the nexus 5 devices, 2 and 4 are off line and 1 and 3 are acting up. Lets expand this to try to recover as many as we can.

Lets do a factory reset by rebooting into recovery mode and doing a factory reset for nexus-5-kot49h-{1,2,3,4}. The devices will need 

wifi:              enable wifi. If the signal is sufficient, I will
                   configure wpa_supplicant.
                   make sure the device isn't in airplane mode.
sound:             silent
display:           turn off auto rotate
                   turn down the brightness to the lowest setting
screen timeout:    1 minute
location:          turn off gps
Bluetooth:         turn off
security:          set screen unlock to none
                   turn off Scan device for security threats
                   turn off Verify at app installation/over USB
applications &     turn on unknown sources
development:       turn on android debugging
                   stay awake
                   turn off protect sdcard if it is enabled.
accounts&sync:     turn off background data and auto-sync

You may need to go to settings->about phone, then tap the build number 7 times to activate the developer menu.

Also, once the device is connected back to the mac mini host, we will need to respond to the dialog asking if we want to allow usb debugging from the host. Check the box that says "Always allow from this computer" and hit Ok. I think there is a setting that turns this prompt off but I can't find it at the moment.

The factory reset shouldn't remove the superuser app. You need to launch it, press the settings button , then set superuser to automatically grant permission, turn off logs and turn off notifications.

If we can't recover these devices, lets take them to ServiceDesk and have them recycled and removed from Inventory.

Comment 2

[Geo Mealer [:geo] -- This account is inactive after 2015-07-07]

mschifer and I brought back [1..4] to the point where bc was able to take over. Think this can be closed out.

Comment 3

[Bob Clary [:bc] (inactive)]

snorp, nchen: The nexus-5 devices are working pretty well so far after the factory resets with the exception of Second run, local-twitter Throbber start which is extremely noisy. The other tests/measurements appear fine.

http://phonedash.mozilla.org/#/org.mozilla.fennec/throbberstart/local-twitter/rejected/2015-04-30/2015-04-30/cached/errorbars/standarderror/notry

I had thought that the issue was with the failing nexus-5-kot49h-1 but I'm beginning to wonder if this isn't a product change. It started around April 17 with

https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=84a83069abb5&tochange=594d0fd874d9

and was full in effect for 

https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=84a83069abb5&tochange=e3c19212d8d0

If you expand the phonedash range you can see the Second run, local-twitter Throbber start was much smoother prior to April 17.

With the current rejection level of 15% stderr, this noise on this single measurement causes the test run to be rejected and retried which doubles the time it takes for a measurement to complete. I think I'm going to increase the rejection level to 40% or higher to prevent the rejection/retry but want to make sure you are aware of the change in behavior on April 17 and make sure I'm not ignoring a real product issue and mistaking it for a device issue.

Comment 4

[(inactive) Jim Chen [:jchen] [:darchons]]

It's strange that changes related to shutdown could affect Nexus 5 and only Nexus 5 startup metrics (both this and the earlier improvements in Nexus 5 startup IIRC). I'm assuming the Nexus 5 runs the same scripts as the other devices? What exactly is second-run? How are we shutting down Fennec after the first-run? Do we tell Fennec to shutdown or just kill it?

Comment 5

[Bob Clary [:bc] (inactive)]

Yes, the Nexus 5s run the same tests as the other devices.

The profile is created with specified preferences and the quitter xpi and a page containing a call to quitter is called which stops the browser after the page loads.

Then the test page is loaded and it is also cleanly shut down via a call to quitter after the load event first. The Throbber start and stop messages are measured relative to the time of the Start proc message in the logcat. This is the First Run.

Then the test page is loaded again and cleanly shutdown via quitter. The Throbber start and stop messages are measured relative to the time of the Start proc message in the logcat. This is the Second Run and essentially tests the cached performance.

This may not be related to the code changes at all. It may just be the deterioration of the storage on these devices after being run for so long.  I just wanted to make sure before putting into the 'flaky device behavior bin'.