717018 - merge robotium logs into a single logfile

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Shriram (irc: Mavericks)]

robotium is run one test at a time and reports results for a given test; we need to make robotium run all tests at once or find a way to merge the log files into a single unified log to work with tbpl

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

This is an interesting scenario.  Right now we have to wait for the test to complete (bug 717725), which means there is no harness level wait to check for a timeout.  If we could timeout and prove that we could run 10 test files in a single 'am instrument' command, it would be nice to not run each test one at a time.

We have done a lot of fixes to make robotium more reliable, so it might be possible to do this.  

If we choose to run one test file at a time, then we need to parse all the logs.  I am not sure the best method for this.  There is some code in maemkit (http://hg.mozilla.org/qa/maemkit) to do this, and we could use that.  I would say the log merging should be a temporary fix.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

We will want to do this in the short term once we get robocop running on tbpl.  This way we can report all the results.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

this will be a python harness based log parser that will be able to return the proper return values so we can be orange/red/green.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: stitch logs together and return proper error code (1.0)

basic patch to create a single unified mochitest style log at the end of a test run while handling the return value/exit codes from mochitest.

Comment 5

[Jeff Hammel]

Comment on attachment 595289 [details] [diff] [review]
stitch logs together and return proper error code (1.0)

+        with open(self.localLog) as currentLog:

This is a 2.5ism.  Do we need to code to 2.4 here?

probably better doing lines = currentLog.readlines() vs the data.split()

+            if reend.match(line):
+                end_found = True
Should just return at this point

It would be nice to use logparser.   I guess this is not possible?  Or worth considering?

Comment 6

[Geoff Brown [:gbrown] (pto Mar 24)]

Comment on attachment 595289 [details] [diff] [review]
stitch logs together and return proper error code (1.0)

Review of attachment 595289 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/mochitest/runtestsremote.py
@@ -349,0 +351,26 @@
> > +    def addLogData(self):
> > +        with open(self.localLog) as currentLog:
> > +            data = currentLog.read()
> > +
NaN more ...

I'll just mention that for me personally, I would like to see a big bold visual separation before the summary:
logFile.append("==================== TEST SUMMARY ====================")

@@ -425,3 +482,4 @@
> >  
> >              try:
> >                  retVal = mochitest.runTests(options)
> > +                logData = mochitest.addLogData()

addLogData() does not return anything. Did you want to accumulate the log data in a return value and then pass that to printLog? That might be neater...but the member variable works fine.

Comment 7

[Geoff Brown [:gbrown] (pto Mar 24)]

(In reply to Geoff Brown [:gbrown] from comment #6)
> NaN more ...

Oh bugzilla, why do you do that?

The intended context there was:

+    def printLog(self):
+        passed = 0
+        failed = 0
+        todo = 0
+        incr = 1
+        logFile = [] 
+        logFile.append("0 INFO SimpleTest START")

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

added a few minor comments and edits before push.  I have to make this match the log file parsers that we have so our systems can properly annotate and turn green/orange/red.

inbound:
https://hg.mozilla.org/integration/mozilla-inbound/rev/7be822540243

Comment 9

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/7be822540243