501172 - Test scripts for NSPR on Symbian

Status: RESOLVED WONTFIX

(Firefox for Android Graveyard :: General, enhancement)

Description

[Harry Li]

User-Agent:       Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.1) Gecko/20090624 Firefox/3.5 GTB5
Build Identifier: 

Patches will be submitted later.

Reproducible: Always

Comment 1

[Nelson Bolyard (seldom reads bugmail)]

I confirm this is an RFE.  :)
Is this a Fennec RFE?  or an NSPR RFE?

Comment 2

[cmtalbert]

One question, will these tests be cross platform, or will they be symbian specific?  One thing we wanted was to have a set of nspr tests that we could run against each new mobile platform to get an early idea of potential problems.

Thanks.

Comment 3

[Brad Lassey [:blassey] (use needinfo?)]

I would assume the intent would be for them to be cross platform.  Obviously Harry's primary concern is Symbian though.

Comment 4

[Harry Li]

Attachment: initial draft

This is a python script which could be run on a S60 phone with PyS60 runtime. The script could call nspr test executables and verify if the exe returns 0 indicating a successful run. This is verified on a Nokia N78 phone. However, I am not request for reviewing/patch landing for some reasons.

1 - Are you NSPR/Testing/QA peers going to make a single python script to test NSPR test cases on multi-platforms? If so, this one is obvious incomplete. Actually it only test exes on Symbian phone now.

2 - One note: for the reason Symbian OS put all executable binaries in one single directory, I made all test exes a prefix "nspr_tests_", so that's why you see this in the script.

3 - In the past I logged the output(stdout,stderr) to file system to see what's going on while executing a test case. But here in the script I just verify if the exec returns zero. Then are we going to leave debugging/development logging function out of it, what's the reason?

Thanks for reviewing and commenting the script.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

All of our existing test harness scripts usually have a few common pieces to them:
 - output of TEST-PASS, TEST-UNEXPECTED-FAIL, TEST-KNOWN-FAIL for each test.  This could be done as a result of your python script analyzing the return value and printing the status:
TEST-PASS | zerolen.exe | 
TEST-UNEXPECTED-FAIL | xnotify.exe | exit code 1
...

the stdout of the individual test is captured and stored in a log file.  The pass fail status is printed out to the screen and either redirected to a log file (xpcshell), or written to a log file (mochitest) as specified on the command line.

 - We usually lauch with subprocess.Popen.  I don't know if this is available for symbian python:
http://mxr.mozilla.org/mozilla-central/source/testing/xpcshell/runxpcshelltests.py#224

 - For xpcshell and crashtest, we use a manifest file to list the tests we are going to test.  This is generated for xpcshell at build time:
http://mxr.mozilla.org/mozilla-central/source/config/rules.mk#159

and then read in via command line argument to runxpcshelltests.py:
http://mxr.mozilla.org/mozilla-central/source/testing/xpcshell/runxpcshelltests.py#273



Overall this is a great start.  Making this multi-platform would be a big win as well as keeping it with a similar interface to our other scripts.  Since you are not running xpcshell or fennec from this script there is no need for all the code to setup environment variables and profiles.  That should keep this script nice and simple.

Thanks for making these tests!

Comment 6

[Nelson Bolyard (seldom reads bugmail)]

The NSS module owners do NOT want to require builders and testers of NSPR to
have Python to be able to build and test it.  I guess we're willing to make 
an exception for Symbian, but that is not our desire for all platforms.

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

Good point Nelson.  What would be a good scripting language?  

Should this be something written in C?  I don't know of any other scripting language that is supported on all platforms as well as Python is.

Comment 8

[Nelson Bolyard (seldom reads bugmail)]

This issue is why I asked, in comment 1, whether this is an NSPR RFE or a 
Fennec RFE.  NSPR has a set of test programs that it uses on all platforms. 
They're all in c.  On all existing platforms except WinCE, testing is done
by cd'ing into mozilla/nsprpub/*.OBJ/pr/tests and running "gmake runtests"
which essentially runs a trivial shell script. (I'll explain about WinCE below).

Comment 5 tells me that project Fennec has its own set of test programs, 
and the thing they all have in common is python.  So, it seems to me that 
if this is a Fennec RFE, then it makes sense to do this in python and add the 
python script not into NSPR directories, but into Fennec test directories.  
OTOH, if this is an NSPR RFE, and the tests for Symbian are to become part 
of the NSPR sources, then IMO, it should try to follow one of the two 
existing NSPR approaches, if possible, and only resort to yet-another 
language if neither of NSPR's present two approaches works for Symbian.

On all presently supported NSPR platforms except WinCE, we build a pretty 
large number of separate individual executable programs, and run each of 
them.  On WinCE, Each of those sources files is built as a dynamically 
loadable shared library, rather than as a separate executable.  There is 
one executable program, written in c, that loads those shared libs one at
a time and runs the tests.  See mozilla/nsprpub/*OBJ/pr/tests/Makefile.
Perhaps symbian should try to do something like (or even reuse) the code 
for testing on WinCE.  (Sorry if this is stating the obvious, or suggesting
something you've already tried.)

Comment 9

[Brad Lassey [:blassey] (use needinfo?)]

(In reply to comment #1)
> I confirm this is an RFE.  :)
> Is this a Fennec RFE?  or an NSPR RFE?

In my mind this is a NSPR RFE.  There is no special requirement from Fennec other than being able to test nspr on our target platforms.

(In reply to comment #6)
> The NSS module owners do NOT want to require builders and testers of NSPR to
> have Python to be able to build and test it.  I guess we're willing to make 
> an exception for Symbian, but that is not our desire for all platforms.

I don't think adding a python script option to the tests adds a dependency on python.  It does add options though.  Right now there is a perl script and a shell script available. For two of our three target platforms (wince and symbian) a python script is a viable option, where as perl or shell scripts do not seem to be.


(In reply to comment #8)
> Comment 5 tells me that project Fennec has its own set of test programs, 
> and the thing they all have in common is python.  

Fennec is using (or attempting to use) Firefox's testing harnesses, so nothing Fennec specific. As far as I know, Fennec and Firefox, Python scripts are preferred over native code or perl.

Comment 10

[Nelson Bolyard (seldom reads bugmail)]

If python is the only way to test on any platform, then it becomes a 
requirement for that platform.  I am willing to consider this as an 
exception for Symbian only.  I do not know if the other NSPR owners/peers
will agree to that.

Comment 11

[Harry Li]

Now I will consider there are only TWO methods to implement NSPR test harness for Symbian: native c++ and Python. The first one requires Symbian C++ SDK (of course the NSPR itself needs too), and in my guess the ugly side is it will add some platform specific lines in Makefile. The second one requires Python (PyS60) on a test target (an emulator running on windows or a real phone). It seems the second one gives less NSPR directory and files changes, but I think there might be more changes to be made when tests are deployed.

I hope NSPR owners/peers could give me some feedback on which method do you prefer. Thanks.

Comment 12

[Mark Finkle (:mfinkle) (use needinfo?)]

no longer working on the port