Last Comment Bug 808417 - remotereftest.py should print the actual exception if one occurred during dm.recordLogcat() or reftest.runTests()
: remotereftest.py should print the actual exception if one occurred during dm....
Status: RESOLVED FIXED
:
Product: Testing
Classification: Components
Component: Reftest (show other bugs)
: Trunk
: ARM Android
: -- normal (vote)
: mozilla19
Assigned To: Ed Morley [:emorley]
:
:
Mentors:
Depends on:
Blocks: 732325
  Show dependency treegraph
 
Reported: 2012-11-04 09:40 PST by Ed Morley [:emorley]
Modified: 2012-11-12 01:54 PST (History)
3 users (show)
See Also:
Crash Signature:
(edit)
QA Whiteboard:
Iteration: ---
Points: ---
fixed
fixed


Attachments
Part 1: remotereftest.py should print the traceback (1.63 KB, patch)
2012-11-04 09:50 PST, Ed Morley [:emorley]
jmaher: review+
Details | Diff | Splinter Review
Part 2: Make runTests() exception reporting consistent across test harnesses (5.11 KB, patch)
2012-11-04 09:53 PST, Ed Morley [:emorley]
jmaher: review+
Details | Diff | Splinter Review

Description Ed Morley [:emorley] 2012-11-04 09:40:12 PST
http://hg.mozilla.org/mozilla-central/file/ab099c9e1a09/layout/tools/reftest/remotereftest.py#l446

   439     try:
   440         cmdlineArgs = ["-reftest", manifest]
   441         if options.bootstrap:
   442             cmdlineArgs = []
   443         dm.recordLogcat()
   444         reftest.runTests(manifest, options, cmdlineArgs)
   445     except:
   446         print "TEST-UNEXPECTED-FAIL | | exception while running reftests"
Comment 1 Ed Morley [:emorley] 2012-11-04 09:50:45 PST
Created attachment 678142 [details] [diff] [review]
Part 1: remotereftest.py should print the traceback

...and Automation Error seems more appropriate than TEST-UNEXPECTED-FAIL.

(at least from the try runs I've done so far, most have been device-manager related exceptions, eg: https://tbpl.mozilla.org/php/getParsedLog.php?id=16737220&tree=Try)
Comment 2 Ed Morley [:emorley] 2012-11-04 09:53:04 PST
Created attachment 678143 [details] [diff] [review]
Part 2: Make runTests() exception reporting consistent across test harnesses

No need to output sys.exc_info()[1] when traceback.print_exc() does the same; adds tracebacks where missing etc.
Comment 3 Ed Morley [:emorley] 2012-11-04 14:27:44 PST
https://tbpl.mozilla.org/?tree=Try&rev=de234f1e54fb
Comment 4 Joel Maher ( :jmaher) 2012-11-05 03:05:03 PST
Comment on attachment 678142 [details] [diff] [review]
Part 1: remotereftest.py should print the traceback

Review of attachment 678142 [details] [diff] [review]:
-----------------------------------------------------------------

please test this on try server for one of each OS.  adding a new python module could have odd consequences in our different versions of python.
Comment 5 Joel Maher ( :jmaher) 2012-11-05 03:07:02 PST
Comment on attachment 678143 [details] [diff] [review]
Part 2: Make runTests() exception reporting consistent across test harnesses

Review of attachment 678143 [details] [diff] [review]:
-----------------------------------------------------------------

My biggest concern here is we are removing the info from the 'Automation Error' line which would make all exceptions generic from a parsing standpoint.  I see no technical problem with this approach and as this is coming from the guy who is making tbpl more streamlined, I will assume this is for the better.
Comment 6 Ed Morley [:emorley] 2012-11-05 05:07:27 PST
https://hg.mozilla.org/integration/mozilla-inbound/rev/4222b4c6a3e9
https://hg.mozilla.org/integration/mozilla-inbound/rev/267a4eb4b236

(In reply to Joel Maher (:jmaher) from comment #5)
> My biggest concern here is we are removing the info from the 'Automation
> Error' line which would make all exceptions generic from a parsing
> standpoint.  I see no technical problem with this approach and as this is
> coming from the guy who is making tbpl more streamlined, I will assume this
> is for the better.

Yeah, the exceptions I've seen on Try all have the appropriate prefixes :-)

eg:
{
DMError: Remote Device Error: unable to connect to 10.250.49.44 after 5 attempts
DMError: Automation Error: Timeout in command rmdr /mnt/sdcard/tests/profile
DMError: Remote Device Error: unable to connect to 10.250.51.55 after 5 attempts
Automation Error: error pushing file: Automation Error: Timeout in command push /mnt/sdcard/writetest 10378
Remote Device Error: unable to write to sdcard
}

(and nice that we can now see these :-D)
Comment 8 Ryan VanderMeulen [:RyanVM] 2012-11-10 10:06:51 PST
I was going to try to uplift this to Aurora, but there are some differences between the two and I didn't want to inadvertently break something. Ed, can we please do so?
Comment 9 Ed Morley [:emorley] 2012-11-12 01:54:11 PST
(In reply to Ryan VanderMeulen from comment #8)
> I was going to try to uplift this to Aurora, but there are some differences
> between the two and I didn't want to inadvertently break something. Ed, can
> we please do so?

Sure :-)

https://hg.mozilla.org/releases/mozilla-aurora/rev/0460fcc8e2ef
https://hg.mozilla.org/releases/mozilla-aurora/rev/020af45119f4

https://hg.mozilla.org/releases/mozilla-beta/rev/8b5a77ed01e8
https://hg.mozilla.org/releases/mozilla-beta/rev/f49b7d8ed734

Note You need to log in before you can comment on or make changes to this bug.