1219172 - 28 jobs have been stuck since Oct 21, 2015 11:41:27 AM PDT while trying to get a remote adb connection

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Bitbar, defect)

Description

[Johan Lorenzo [:jlorenzo]]

Some of our Jenkins jobs have been stuck for 6 days. At the beginning, I thought it was related to the VPN being restarted (in bug 1217053). But we keep seeing jobs that remain frozen at the same state. Here's a recent example:

> # The script directly found an available device
> 05:43:42 [Testdroid] - Started session 187932
> 05:43:42 [Testdroid] - ADB port: 1106
> 05:43:42 [Testdroid] - ADB host: fxos.testdroid.com
> 05:43:42 [Testdroid] - Android serial: f061400d
> 05:43:42 [Testdroid] - Marionette port: 1107
> 05:43:42 [Testdroid] - Marionette host: fxos.testdroid.com
> 05:43:42 [Testdroid] - Marionette forwarding host: 172.27.240.3
> 05:43:42 [Testdroid] - Marionette forwarding port: 30006
> 05:43:42 executing script 'Create virtual environment'
> # The python package installation goes well.
> 05:43:58 Successfully installed plivo-0.9.6
> 05:43:58 + adb -H 54.67.13.230 -P 1106 logcat -c
> 05:45:01 ** Cannot start server on remote host
> 05:45:01 - waiting for device -
> 05:46:05 ** Cannot start server on remote host
> 05:46:05 error: cannot connect to daemon
> 05:47:08 ** Cannot start server on remote host
> 05:47:08 - waiting for device -
> 05:48:12 ** Cannot start server on remote host
> 05:48:12 error: cannot connect to daemon
> 05:49:15 ** Cannot start server on remote host
> 05:49:15 - waiting for device -
> # an so on

Some jobs that were started 6 days ago, have never failed, they've kept asking for the same session.

Comment 1

[Johan Lorenzo [:jlorenzo]]

Could this be related to the changes done in the devices sessions?

Comment 2

[Johan Lorenzo [:jlorenzo]]

In the meantime, I cancelled all the stuck jobs.

Comment 3

[Martijn Wargers (dead)]

Thanks, No-Jun noticed this earlier too, that some jobs would just get stuck and we would have to cancel them manually.

Comment 4

[Sakari Rautiainen]

Should the adb command define the android serial id(-s) as well: adb -H 54.67.13.230 -P 1106 logcat -c
There is multiple devices in the same host so I don't think that logcat -c command would work if serial id is not defined.
 Does the jenkins job do "waiting for device:" for ever, or is there timeout for that?
Is there any code which does adb kill-server? It will affect all the devices in the same host if done.

Comment 5

[Johan Lorenzo [:jlorenzo]]

(In reply to Sakari Rautiainen from comment #4)
> There is multiple devices in the same host so I don't think that logcat -c
> command would work if serial id is not defined.
-c is to flush the logs already present on the device. I think -s is to specify the serial, though. I thought the port forwarding enabled us to target one specific device. Isn't that the case/ 

> Does the jenkins job do "waiting for device:" for ever, or is there timeout
> for that?
I believed Jenkins was configured to cancel a job after a given time, but some jobs remained running for more than a week.

> Is there any code which does adb kill-server? It will affect all the devices
> in the same host if done.
That sounds about right, I'll try that tomorrow, when less builds are queue.

Comment 6

[Sakari Rautiainen]

"adb -H 54.67.13.230 -P 1106 logcat -c" command sends message to adb server, and as there is multiple devices connected to same server adb server doesn't know which specific device adb client is trying to connect. Therefore -s flag is mandatory. Even though each device has it own port which adb client then uses, it is still used to connect to the same adb server.

Comment 7

[Johan Lorenzo [:jlorenzo]]

I checked some green jobs we've had and the I always see these type of commands:
> adb -H 54.67.13.230 -P 1072 logcat -c
> adb -H 54.67.13.230 -P 1088 logcat -c
> adb -H 54.67.13.230 -P 1086 logcat -c

Do you think these builds passed thanks to a race condition? Like only 1 device is available at that time on that port?

Comment 8

[Sakari Rautiainen]

Ok, in that case I think that you are setting environment variable ANDROID_SERIAL as otherwise the adb should fail with error: error: more than one device and emulator

Comment 9

[Johan Lorenzo [:jlorenzo]]

I checked the job configuration and we don't set the ANDROID_SERIAL environment there. However, the Jenkins plugin does[1]. This part doesn't change between a job and another. The logs in comment 0 show a correct serial id found. Moreover,the error message is not the one in comment 8.

So, I'm not sure if the problem is on the command-line side. What do you think Sakari?

[1] https://github.com/mozilla/testdroid-marionette-plugin/blob/6d4ff2893827981bdea6a310a6352000eba30c23/src/main/java/jenkins/plugins/testdroid/DeviceSessionWrapper.java#L221

Comment 10

[Johan Lorenzo [:jlorenzo]]

Attachment: Job stuck - Job started on Oct 30 2015, 5:12am PDT

Here's another example of a job stuck. The logs show we got the device with the ID f0364197. Sakari, would it be possible to see what happened to that device around Oct 30st 05:18:27am PDT?

Comment 11

[Sakari Rautiainen]

I checked server side logs, and I didn't see anything unusual there. At the same time there was other tests running on the same cluster where the device f0364197 is, so ADB server was fine. The device was released after timeout in 4 hours later, and there seemed to be online all the time. I need to add a bit more debugging output into tcp proxies to be able to understand error better. 
I would suggest that you add timeout value for adb command(s). - using command like timelimit allows you to specify maximum time that command should be allowed to run.

Comment 12

[Johan Lorenzo [:jlorenzo]]

I can add the timeout, however it'll be harder to spot which job has been stuck afterwards. Jenkins will return a red build, along all the other genuine red builds. I'd prefer we diagnose the issue, before putting the timeout.

13 new jobs have been stuck for the last 2 days. Maybe we'll be more lucky with these ones:

---|--------------------------|------------------|---------------|-------------------
 # | Job start (PST timezone) | Adb logcat start | Device serial | ADB host:port
---|--------------------------|------------------|---------------|-------------------
 1 | Nov 9th, 20:42:42        | 20:49:33         | f30ecc97      | 54.67.13.230:1106
 2 | Nov 10th, 03:32:49       | 03:38:11         | 356cd07c      | 54.67.13.230:1102
 3 | Nov 10th, 04:32:40       | 04:38:48         | 94fdcece      | 54.67.13.230:1108
 4 | Nov 10th, 05:04:27       | 05:10:43         | 94f8ce15      | 54.67.13.230:1114
 5 | Nov 10th, 05:04:27       | 05:10:44         | 356cd0b6      | 54.67.13.230:1116
 6 | Nov 10th, 08:27:42       | 08:33:39         | 94f7ce17      | 54.67.13.230:1101
 7 | Nov 10th, 09:52:46       | 09:58:43         | f30ecc97      | 54.67.13.230:1123
 8 | Nov 10th, 10:32:45       | 10:48:43         | 3562d023      | 54.67.13.230:1108
 9 | Nov 11th, 05:44:23       | 05:51:20         | 356cd0b6      | 54.67.13.230:1101
10 | Nov 11th, 10:17:41       | 10:23:00         | f309ccd8      | 54.67.13.230:1113
11 | Nov 11th, 19:54:26       | 20:00:51         | 356cd0b6      | 54.67.13.230:1107
12 | Nov 11th, 19:54:26       | 20:01:15         | 94f8ce15      | 54.67.13.230:1111
13 | Nov 11th, 20:17:46       | 20:23:14         | 1b41c231      | 54.67.13.230:1117

Comment 13

[Sakari Rautiainen]

Thanks, I was able to found the issue.. It was actually the firewall which blocked the connections, so currently the open port range was 1000 to 1100 but ports were given from 1024 upto 1200. So any port above 1100 was blocked by firewall. I changed the port range and now it should work again.

Comment 14

[Johan Lorenzo [:jlorenzo]]

Thanks Sakari! 

Marking this bug as resolved, based on comment 13.