Closed Bug 946908 Opened 11 years ago Closed 11 years ago

Check connectivity to android emulators before and after tests

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

Product:
Infrastructure & Operations Graveyard
Component:
CIDuty
Platform:
x86
macOS
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: gbrown, Assigned: gbrown)

References

Details

Attachments

(1 file)

check sut and adb connectivity before and after test set
4.08 KB, patch
armenzg
: review+
Details | Diff | Splinter Review
In bug 927602, we seem to lose connectivity to the emulator during a test. The test harness reports a timeout, the test fails, and we can see that the emulator process is still running (because the kill is reported) but we don't know: - is the emulator accepting adb commands? - is the sutagent running on the emulator and accepting commands? I want to update android_emulator_unittest.py to answer these questions.
From a recent run: 13:01:59 INFO - ##### 13:01:59 INFO - ##### Running start-emulators step. 13:01:59 INFO - ##### 13:01:59 INFO - Running pre-action listener: _resource_record_pre_action 13:01:59 INFO - Running main action method: start_emulators 13:01:59 INFO - Let's kill every process called compiz 13:01:59 INFO - Created temp file /tmp/tmpYzkPpd. 13:01:59 INFO - Trying to start the emulator with this command: emulator -avd test-1 -debug init,console,gles,memcheck,adbserver,adbclient,adb,avd_config -port 5554 -qemu -m 1024 -enable-kvm 13:01:59 INFO - Sleeping 10 seconds 13:02:09 INFO - Attempt #1 to redirect ports: (5554, 20701, 20700) 13:02:09 INFO - test-1: 5554; sut port: 20701/20700 13:02:09 INFO - Created temp file /tmp/tmpkvhEYT. 13:02:09 INFO - Trying to start the emulator with this command: emulator -avd test-2 -debug init,console,gles,memcheck,adbserver,adbclient,adb,avd_config -port 5556 -qemu -m 1024 -enable-kvm 13:02:09 INFO - Sleeping 10 seconds 13:02:19 INFO - Attempt #1 to redirect ports: (5556, 20703, 20702) 13:02:19 INFO - test-2: 5556; sut port: 20703/20702 13:02:19 INFO - Checking emulator test-1 13:02:19 INFO - Attempt #1 to connect to SUT on port 20701 13:02:19 INFO - Connected to port 20701 13:02:19 INFO - Trying again after EOF 13:02:19 INFO - Sleeping 30 seconds 13:02:49 INFO - Attempt #2 to connect to SUT on port 20701 13:02:49 INFO - Connected to port 20701 13:02:49 INFO - Trying again after EOF 13:02:49 INFO - Sleeping 30 seconds 13:03:19 INFO - Attempt #3 to connect to SUT on port 20701 13:03:19 INFO - Connected to port 20701 13:03:19 INFO - SUT response: $> 13:03:19 INFO - ['adb', '-s', 'emulator-5554', 'shell', 'ps']: 13:03:19 INFO - USER PID PPID VSIZE RSS WCHAN PC NAME 13:03:19 INFO - root 1 0 400 272 c0269783 08056ac6 S /init 13:03:19 INFO - root 2 0 0 0 c0225fd0 00000000 S kthreadd 13:03:20 INFO - root 3 2 0 0 c021ac82 00000000 S ksoftirqd/0 13:03:20 INFO - root 4 2 0 0 c0223621 00000000 S events/0 13:03:20 INFO - root 5 2 0 0 c0223621 00000000 S khelper 13:03:20 INFO - root 11 2 0 0 c0223621 00000000 S suspend 13:03:20 INFO - root 163 2 0 0 c0223621 00000000 S kblockd/0 ... 13:03:20 INFO - u0_a30 1633 807 227816 21592 ffffffff b7f89837 S com.android.providers.calendar 13:03:20 INFO - u0_a36 1650 807 225500 18396 ffffffff b7f89837 S com.android.voicedialer 13:03:20 INFO - u0_a42 1663 807 233636 20080 ffffffff b7f89837 S com.android.exchange 13:03:20 INFO - u0_a44 1678 807 226412 18912 ffffffff b7f89837 S com.mozilla.watcher 13:03:20 INFO - u0_a8 1699 807 234248 21804 ffffffff b7f89837 S com.android.calendar 13:03:20 INFO - u0_a1 1732 807 225488 18436 ffffffff b7f89837 S com.android.musicfx 13:03:20 INFO - u0_a45 1745 807 239520 27712 ffffffff b7f89837 S com.mozilla.SUTAgentAndroid 13:03:20 INFO - root 1771 854 1328 616 c020222d b7f4cfda S /system/bin/sh 13:03:20 INFO - root 1772 1771 1648 548 00000000 b7f94cd6 R ps 13:03:20 INFO - 13:03:20 INFO - None 13:03:20 INFO - Checking emulator test-2 13:03:20 INFO - Attempt #1 to connect to SUT on port 20703 13:03:20 INFO - Connected to port 20703 13:03:20 INFO - Trying again after EOF 13:03:20 INFO - Sleeping 30 seconds 13:03:50 INFO - Attempt #2 to connect to SUT on port 20703 13:03:50 INFO - Connected to port 20703 13:03:50 INFO - SUT response: $> 13:03:50 INFO - ['adb', '-s', 'emulator-5556', 'shell', 'ps']: 13:03:50 INFO - USER PID PPID VSIZE RSS WCHAN PC NAME 13:03:50 INFO - root 1 0 400 280 c0269783 08056ac6 S /init 13:03:50 INFO - root 2 0 0 0 c0225fd0 00000000 S kthreadd 13:03:50 INFO - root 3 2 0 0 c021ac82 00000000 S ksoftirqd/0 13:03:50 INFO - root 4 2 0 0 c0223621 00000000 S events/0 ... 13:32:10 INFO - ##### 13:32:10 INFO - ##### Running stop-emulators step. 13:32:10 INFO - ##### 13:32:10 INFO - Running pre-action listener: _resource_record_pre_action 13:32:10 INFO - Running main action method: stop_emulators 13:32:10 INFO - Checking emulator test-1 13:32:10 INFO - Attempt #1 to connect to SUT on port 20701 13:32:10 INFO - Connected to port 20701 13:32:10 INFO - SUT response: $> 13:32:10 INFO - ['adb', '-s', 'emulator-5554', 'shell', 'ps']: 13:32:10 INFO - USER PID PPID VSIZE RSS WCHAN PC NAME 13:32:10 INFO - root 1 0 400 272 c0269783 08056ac6 S /init 13:32:10 INFO - root 2 0 0 0 c0225fd0 00000000 S kthreadd 13:32:10 INFO - root 3 2 0 0 c021ac82 00000000 S ksoftirqd/0 13:32:10 INFO - root 4 2 0 0 c0223621 00000000 S events/0 13:32:10 INFO - root 5 2 0 0 c0223621 00000000 S khelper 13:32:10 INFO - root 11 2 0 0 c0223621 00000000 S suspend 13:32:10 INFO - root 163 2 0 0 c0223621 00000000 S kblockd/0 ...
Attachment #8343277 - Flags: review?(armenzg)
Attachment #8343277 - Flags: review?(armenzg) → review+
Blocks: 936226
That failed on Cedar because adb is not in the path. 14:34:14 INFO - Connected to port 20701 14:34:14 INFO - SUT response: $> 14:34:14 INFO - Running post-action listener: _resource_record_post_action 14:34:14 FATAL - Uncaught exception: Traceback (most recent call last): 14:34:14 FATAL - File "/builds/slave/talos-slave/test/scripts/mozharness/base/script.py", line 1132, in run 14:34:14 FATAL - self.run_action(action) 14:34:14 FATAL - File "/builds/slave/talos-slave/test/scripts/mozharness/base/script.py", line 1074, in run_action 14:34:14 FATAL - self._possibly_run_method(method_name, error_if_missing=True) 14:34:14 FATAL - File "/builds/slave/talos-slave/test/scripts/mozharness/base/script.py", line 1015, in _possibly_run_method 14:34:14 FATAL - return getattr(self, method_name)() 14:34:14 FATAL - File "scripts/scripts/android_emulator_unittest.py", line 425, in start_emulators 14:34:14 FATAL - self._check_emulator(emulator) 14:34:14 FATAL - File "scripts/scripts/android_emulator_unittest.py", line 253, in _check_emulator 14:34:14 FATAL - p = subprocess.Popen(ps_cmd, stdout=subprocess.PIPE) 14:34:14 FATAL - File "/usr/lib/python2.7/subprocess.py", line 679, in __init__ 14:34:14 FATAL - errread, errwrite) 14:34:14 FATAL - File "/usr/lib/python2.7/subprocess.py", line 1249, in _execute_child 14:34:14 FATAL - raise child_exception 14:34:14 FATAL - OSError: [Errno 2] No such file or directory 14:34:14 FATAL - Running post_fatal callback... Updated to use self.adbpath: https://hg.mozilla.org/build/mozharness/rev/2b787e4d30f6
That works now. An example where we cannot use adb or sut to connect to the emulator at the end of the test, even though the emulator is still running: https://tbpl.mozilla.org/php/getParsedLog.php?id=31528341&tree=Cedar&full=1
In production.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Component: Platform Support → Buildduty
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: