1118291 - [raptor] Orng tap intermittent failures when running multiple iterations of launch_test on b2g-emulator

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Gaia::PerformanceTest, defect)

Description

[Robert Wood [:rwood]]

Start the b2g-emulator, create the raptor gaia profile (gaia/make raptor). Then run multiple iterations of the raptor launch_test (use the following command line):

~/gaia$ DEBUG=* RUNS=10 RAPTOR_EMULATOR=1 APPS='communications/dialer' RAPTOR_DATA_GAIA_REV=$(git rev-parse HEAD) node tests/raptor/launch_test

Almost every time, out of 10 runs at least one will fail, waiting on the homescreen for the app icon to be tapped:

[Test] Starting run 9
  raptor:coldlaunch [Launching] communications/dialer application +6s
  raptor:coldlaunch [Launching] Translate to device pixels (X: 120 => 120, Y: 233 => 233) +1ms
  raptor:coldlaunch [Launching] Triggering device tap on application icon +0ms
  adb:command Send '001Chost:transport:emulator-5554' +15s
  adb:command Send '0037shell:echo "tap 120 233 1 1" > /data/local/tmp/orng-cmd' +0ms
  adb:command Send '001Chost:transport:emulator-5554' +5ms
  adb:command Send '0041shell:/data/local/orng /dev/input/event0 /data/local/tmp/orng-cmd' +0ms
[Test] Run 9 timed out, retry attempt 1
  raptor:device [Logging] Clearing Log +2m
  raptor:command [Executing Command] adb logcat -c +0ms
  raptor:device [Closing App] PID 2340 +83ms
  raptor:command [Executing Command] adb shell kill 2340 +0ms
  adb:command Send '001Chost:transport:emulator-5554' +2.4m
  adb:command Send '0043shell:echo "drag 250 250 250 350 10 150" > /data/local/tmp/orng-cmd' +0ms
  adb:command Send '001Chost:transport:emulator-5554' +3ms
  adb:command Send '0041shell:/data/local/orng /dev/input/event0 /data/local/tmp/orng-cmd' +1ms
  raptor:coldlaunch [Launching] communications/dialer application +6s
  raptor:coldlaunch [Launching] Translate to device pixels (X: 120 => 120, Y: 233 => 233) +0ms
  raptor:coldlaunch [Launching] Triggering device tap on application icon +0ms
  adb:command Send '001Chost:transport:emulator-5554' +6s
  adb:command Send '0037shell:echo "tap 120 233 1 1" > /data/local/tmp/orng-cmd' +0ms
  adb:command Send '001Chost:transport:emulator-5554' +3ms
  adb:command Send '0041shell:/data/local/orng /dev/input/event0 /data/local/tmp/orng-cmd' +0ms
Error: Test timeout exceeded 150000ms
    at Phase.timeoutError (/home/rwood/raptor/lib/suite/phase.js:50:22)
    at Phase.fail (/home/rwood/raptor/lib/suite/phase.js:225:12)
    at /home/rwood/raptor/lib/suite/phase.js:222:18
    at /home/rwood/raptor/node_modules/promise/lib/core.js:33:15
    at flush (/home/rwood/raptor/node_modules/promise/node_modules/asap/asap.js:27:13)
    at process._tickCallback (node.js:415:13)

Single iterations always seem to work. Note that the swipehack is running on the b2g-emulator at the start and during any retries, but doesn't seem to fix it (not the same issue). Figure out if this is an emulator, orangutan, adb, or timing issue.

Comment 1

[Robert Wood [:rwood]]

Attachment: https://github.com/eliperelman/node-orng/pull/2

Since the emulator is so slow, writing the orng command to the emulator script sometimes wasn't finished before actually calling orng and attempting to execute the script. Fixed by having a slight delay between writing the command/script and actually executing it. Optimum delay seems to be 250ms (100 successful runs of the launch_test) and that should leave some leeway in case the emulator is even slower on the cloud.

Comment 2

[:Eli Perelman]

Available on npm: orng@0.3.1

Comment 3

[Robert Wood [:rwood]]

Merged (thanks Eli) https://github.com/eliperelman/node-orng/commit/e044eae62a21aca340a6d7adcbed5f00043ae060