682129 - Add logcat in parallel with tegra debug unittests

Status: RESOLVED WONTFIX

(Release Engineering :: General, defect, P4)

Description

[Aki Sasaki (not active)]

We don't get STDOUT from the tegra debug unittests (these go to device with no listeners).

We run a logcat at the end of the job, but this is often the very tail end of the log rather than the whole thing.

Running a log on the device isn't a good idea because disk == memory, and writing a significant log to disk will cause us to oom.

We should, while running run_tests.py, have a parallel process running logcat from SUT that writes to the Foopy's disk.

Comment 1

[Aki Sasaki (not active)]

We may be able to solve this by using adb instead of sut.

Comment 2

[Phil Ringnalda (:philor)]

Just a note here, since I wasn't paying enough attention in bug 649690 where I should have noted it: the debug tests will not be made visible until they have the assertions in the log, since otherwise we'd be saying "I'm sorry, you have to back out your patch, because it asserts on Android; no, I'm sorry, I can't tell you what assertion it hits, only that it hits one."

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

so for debug only tests we should turn on logcat scraping.  

this would be:
 - before test starts turn on adb via tcp
 - spawn a thread for adb logcat and collect data in a log on the host
   - this needs to be for specific providers (gecko, ???)
 - when test is done, terminate logcat and thread, clean up adb connection list
 - cat the host_logcat.log to the log we report to the tests


This is a good first step to ensure we have full logcat info from a test.  

Future steps would be to interleave the logcat output with the test output so we have a unified front. We could experiment with redirecting all stdout to the logcat and seeing if that could be consolidated in a single log from the device.

Comment 4

[Brandon Sterne (:bsterne)]

The fuzzing team is interested in having this feature as well since a number of our fuzzers, which we're now running on mobile, need to get data back from STDOUT.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

this would also require adjusting the android setting: 'adb shell setprop log.redirect-stdio true' and restarting the adb service.  I have never tested this to see if stdout from fennec makes it to the logcat output, I assume it does.

Comment 6

[Aki Sasaki (not active)]

The BaseHelper+LocalLogger objects (bug dependency added) will possibly help here.
Also, that may potentially allow us to run, say, all talos suites for a given build at the same time on a single machine (i.e., one linux buildbot slave controlling 7 tegras in the same script, in parallel).

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

after using adb/tcp for a while, I have found it to be highly unreliable when dealing with >1 device/host and somewhat unreliable with a 1:1 mapping of device:host.

The only viable solution I see is to redirect logcat output to a local file on the filesystem.  This wouldn't be ideal for Talos, but could work for mochitest/reftest.

Comment 9

[Phil Ringnalda (:philor)]

[capacity] because without this unblocking debug tests on Android, I VERY STRONGLY IN ALL CAPS recommend that people never push to try with anything other than -b do, because running opt tests doesn't show your debug-only failures, and running debug tests doesn't run any Android or b2g tests at all.

Comment 11

[Geoff Brown [:gbrown]]

Rather than the approach in comment 3, I was thinking of using sut to exec logcat at the start of the test. I am pretty sure that sutagent already execs commands in a separate thread, so we would just need to do something at the devicemanager or harness level to avoid waiting for logcat to finish, and then of course collect the logcat output at the end of the test and add that to the test log.

A related problem is collecting logcat when there is a loss of connectivity ("xxx seconds without output", "timeout in command xxx", etc). We might learn more about those failures if the test was collecting logcat continuously, so that the test log could report the logcat up until the loss of connectivity.

I have become befuddled when trying to solve both problems at the same time. Maybe I should try it in two stages: 1. Collect all the logcat at the end of the test. 2. Collect logcat continuously. 


Will the potentially huge increase in the size of the test log cause any new problems? Is there a point at which we should say, oh, that logcat is too big to put in the test log?

Comment 12

[Kartikaya Gupta (email:[e-mail])]

If there is such a point, I would like to have a trychooser option to enable the full logcat even if it's not enabled by default. There are definitely times when I just want the full log regardless of how big it is.

Comment 13

[Joel Maher ( :jmaher ) (UTC -8)]

I doubt the logcat will be too big, but I believe we have a 100M limit on our log files (imposed by twisted or buildbot).

In the early days I recall doing this and sutagent would timeout because it expected the command to finish.  Maybe this is more of a reality now with the minor changes into the sutagent.

Comment 14

[Reuben Morais [:reuben]]

We want to collect logcat continuously because logcat itself uses a ring buffer that has a hardcoded size, so it's not us trimming the log after we get it, but rather the logcat does not go all the way to the beginning of the test run.

Comment 15

[Geoff Brown [:gbrown]]

We now have complete logcats, collected continuously, for Android emulator jobs (bug 961207) and for Android 4.0 Opt/Debug (bug 962743). Those solutions do not apply to the Android 2.2 / Tegra jobs, since they were made in mozharness. Since we are running debug tests only on Pandas now and the Tegras are near the end of their lives, I don't think it's worth our time to pursue this bug.

Comment 16

[Chris Cooper [:coop] (he/him)]

WONTFIXing based on comment #15.