495164 - maemkit needs tweaks to work in tinderbox

Status: RESOLVED WONTFIX

(Testing :: General, defect)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: xpcdriver 1.0

Since completing maemkit the unittests have changed as well as missing features that are useful when integrating in with tinderbox.

Since the scope of maemkit is limited right now, we will use this bug to track all the issues found while getting the tests stabilized on tinderbox.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

xpcdriver.py problem:
running the manifest file verbatim like ted does.  Instead, I now build a list of tests and run one file at a time, this time utilizing the timeout mechanism that I have for mochitest to force a timeout after 5 minutes.

Comment 2

[Aki Sasaki (not active)]

Traceback (most recent call last):
  File "maemkit-chunked.py", line 22, in <module>
    main()
  File "maemkit-chunked.py", line 19, in main
    runTest(mk)
  File "maemkit-chunked.py", line 12, in runTest
    testdriver.getTests()
  File "/media/mmc2/maemkit/xpcdriver.py", line 34, in getTests
    testfiles = sorted(glob(os.path.join(testdir, "test_*.js")))
NameError: global name 'glob' is not defined
program finished with exit code 1

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: xpcdriver 1.0.1

had put 'from glob import glob' in maemkit, but didn't have that in this patch.  Moved to xpcdriver.py for clarity.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: xpcdriver 1.0.2

fixed logging to create log files based on test we are testing instead of overall manifest name.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: Mochidriver 1.0

update to mochidriver to fix the split logic.  There was an issue while executing shutil.move where it would throw an exception on a directory and ignore all files after that.  This is fixed and split/stitch works 10 times in a row for the whole mochitest tree on my n810 maemo device.

Lets get this applied to the tinderbox maemo mochitest runs and see if there are any other reliability issues.

Comment 6

[cmtalbert]

Comment on attachment 380252 [details] [diff] [review]
xpcdriver 1.0.2

>diff -r 527dc910d556 xpcdriver.py

>-    buffered = True
>-    utilname = os.path.normpath(os.path.join(self.options["utility-path"], "runxpcshelltests.py"))
>-    command = "python " + utilname
>-    command += " --xre-path=" + os.path.normpath(self.options["xre-path"]) + " " 
>-    command += " --manifest=" + os.path.normpath(self.options["manifest"]) + ".maemkit " 
>-    command += os.path.normpath(os.path.join(self.options["xre-path"], "xpcshell")) + " "
>-    command += " --keep-going"
>+    for test in self.directories:
>+      utilname = os.path.normpath(os.path.join(self.options["utility-path"], "runxpcshelltests.py"))
>+      command = "python " + utilname
>+      command += " --xre-path=" + os.path.normpath(self.options["xre-path"]) + " " 
>+      command += " --manifest=" + os.path.normpath(self.options["manifest"]) + " " 
>+      command += " --test-path=" + test + " " 
>+      command += os.path.normpath(os.path.join(self.options["xre-path"], "xpcshell")) + " "
Why are you dropping the command += "--keep-going" from the new loop?  Is that not needed?  I suspect it might not be since you are adding the timeout stuff below, but I wanted to double check that it wasn't a typo.

Otherwise, this looks good.  r=ctalbert with that question answered (and if it is supposed to be in there, that you put it back in of course :) )

Comment 7

[cmtalbert]

Comment on attachment 381812 [details] [diff] [review]
Mochidriver 1.0

This looks good. r=ctalbert

Comment 8

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: initial ability for maemkit to run single file at a time and using remote web server

updating maemkit with additional tweaks for running as a single test.  Here is a patch that will run mochitests one test file at a time.  This patch depends on the xda tools.

Comment 9

[Aki Sasaki (not active)]

Hm, do we retire maemkit now that we're not testing on n900s?

Comment 10

[cmtalbert]

(In reply to Aki Sasaki [:aki] from comment #9)
> Hm, do we retire maemkit now that we're not testing on n900s?

I certainly hope so.

--> WONTFIX :D