685360 - Android reftest can't get return value from adb shell, incorrectly passes test

Status: RESOLVED WORKSFORME

(Testing :: General, defect)

Description

[Benoit Girard (:BenWa)]

remote-reftest for android relies on the value from adb shell to know if the process ran successfully however a known issue of adb shell is that it always return 0.
See: http://code.google.com/p/android/issues/detail?id=3254

I confirmed this by adding an "abort()" in the fennec start up code. adb shell returns 0 and reftest looks for TEST-UNEXPECTED and since no test have ran reftest thinks that all test have passed.

It looks like we will need a hacky solution until this is fixed. Mozilla's Taramin project has wrapped the command in a script that prints the error code: https://bugzilla.mozilla.org/show_bug.cgi?id=544884

Also if we searched the log to make sure that all test have executed we could drastically reduce the number of false positive, however that should probably be a separate bug.

Comment 1

[Benoit Girard (:BenWa)]

Attachment: remote-reftest-false-positive.log

Comment 2

[Benoit Girard (:BenWa)]

We also get a notice that the application crashed in the adb logcat. We should inline the logcat in the reftest log result. Perhaps dump it inline after the adb shell command returns? This is useful for debugging but not as reliable as the error code to determine success.

Should we spin this off in another bug?

Comment 3

[Geoff Brown [:gbrown]]

Today we no longer rely on the adb return code and fail when no tests are run.