Closed Bug 717018 Opened 9 years ago Closed 9 years ago

merge robotium logs into a single logfile

Categories

(Testing :: General, defect)

ARM
Android
defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED
mozilla13

People

(Reporter: kshriram18, Assigned: jmaher)

References

Details

Attachments

(1 file)

robotium is run one test at a time and reports results for a given test; we need to make robotium run all tests at once or find a way to merge the log files into a single unified log to work with tbpl
Assignee: nobody → jmaher
Status: UNCONFIRMED → NEW
Ever confirmed: true
OS: Windows Vista → Android
Hardware: x86 → ARM
Version: unspecified → Trunk
This is an interesting scenario.  Right now we have to wait for the test to complete (bug 717725), which means there is no harness level wait to check for a timeout.  If we could timeout and prove that we could run 10 test files in a single 'am instrument' command, it would be nice to not run each test one at a time.

We have done a lot of fixes to make robotium more reliable, so it might be possible to do this.  

If we choose to run one test file at a time, then we need to parse all the logs.  I am not sure the best method for this.  There is some code in maemkit (http://hg.mozilla.org/qa/maemkit) to do this, and we could use that.  I would say the log merging should be a temporary fix.
We will want to do this in the short term once we get robocop running on tbpl.  This way we can report all the results.
this will be a python harness based log parser that will be able to return the proper return values so we can be orange/red/green.
basic patch to create a single unified mochitest style log at the end of a test run while handling the return value/exit codes from mochitest.
Attachment #595289 - Flags: review?(jhammel)
Attachment #595289 - Flags: review?(gbrown)
Comment on attachment 595289 [details] [diff] [review]
stitch logs together and return proper error code (1.0)

+        with open(self.localLog) as currentLog:

This is a 2.5ism.  Do we need to code to 2.4 here?

probably better doing lines = currentLog.readlines() vs the data.split()

+            if reend.match(line):
+                end_found = True
Should just return at this point

It would be nice to use logparser.   I guess this is not possible?  Or worth considering?
Attachment #595289 - Flags: review?(jhammel) → review+
Comment on attachment 595289 [details] [diff] [review]
stitch logs together and return proper error code (1.0)

Review of attachment 595289 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/mochitest/runtestsremote.py
@@ -349,0 +351,26 @@
> > +    def addLogData(self):
> > +        with open(self.localLog) as currentLog:
> > +            data = currentLog.read()
> > +
NaN more ...

I'll just mention that for me personally, I would like to see a big bold visual separation before the summary:
logFile.append("==================== TEST SUMMARY ====================")

@@ -425,3 +482,4 @@
> >  
> >              try:
> >                  retVal = mochitest.runTests(options)
> > +                logData = mochitest.addLogData()

addLogData() does not return anything. Did you want to accumulate the log data in a return value and then pass that to printLog? That might be neater...but the member variable works fine.
Attachment #595289 - Flags: review?(gbrown) → review+
(In reply to Geoff Brown [:gbrown] from comment #6)
> NaN more ...

Oh bugzilla, why do you do that?

The intended context there was:

+    def printLog(self):
+        passed = 0
+        failed = 0
+        todo = 0
+        incr = 1
+        logFile = [] 
+        logFile.append("0 INFO SimpleTest START")
added a few minor comments and edits before push.  I have to make this match the log file parsers that we have so our systems can properly annotate and turn green/orange/red.

inbound:
https://hg.mozilla.org/integration/mozilla-inbound/rev/7be822540243
https://hg.mozilla.org/mozilla-central/rev/7be822540243
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla13
Depends on: 736920
You need to log in before you can comment on or make changes to this bug.