799855 - Run the NSS test suite during make check

Status: NEW

(Firefox Build System :: General, defect)

Description

[Mike Hommey [:glandium]]

Running the NSS test suite involves various things:
- Fix/Adjust the NSS test suite to allow to be run in the mozilla build environment
- Fix the NSS test suite not to copy libxul and other big files over and over again, wasting disk space
- Fix the NSS test suite to run properly on OSX and Windows
- Change the NSS test suite output so that tinderbox logs can find failures.
- Build the required NSS utils
- Hook the NSS test suite to the mozilla build system

Comment 1

[Mike Hommey [:glandium]]

AFAICT, some test suite failures can also leave server processes around, which create even more failures on subsequent runs because the new test suite can't start a server because the port is bound.

Comment 2

[Mike Hommey [:glandium]]

Attachment: PoC

This is very crude, but works for me on linux, except all fips tests fail because we don't sign libraries.
The change to security/nss/tests/fips/fips.sh makes the disk space waste copying all files from DIST/lib less bad, but it's still copying a 700MB libxul.so multiple times...
Pushed to try to see how it works on other platforms.

Comment 3

[Mike Hommey [:glandium]]

https://tbpl.mozilla.org/?tree=Try&rev=48caad7001b7

Got FIPS failures on osx, as expected and a failure to run at all on windows because DOMSUF defaults to a value returned by domainname, but domainname is not in msys on the windows slaves.

Comment 4

[Ludovic Hirlimann [:Usul]]

(In reply to Mike Hommey [:glandium] from comment #3)
> https://tbpl.mozilla.org/?tree=Try&rev=48caad7001b7
> 
> Got FIPS failures on osx, as expected and a failure to run at all on windows
> because DOMSUF defaults to a value returned by domainname, but domainname is
> not in msys on the windows slaves.

Shall we modify the slaves for this ?

Comment 5

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

FWIW, I always build and test on windows. I modified my hosts file to include localhost.local4 and localhost.local6 so I could build with DOMSUF=local4.

Nobody ever runs the tests out of a mozilla-central repository. They are always run out of the a CVS checkout. That's why they just "copy everything."

IIRC, the *.chk files needed for the FIPS tests need to be generated *after* we've Authenticode-signed the DLLs. And, then the tests need to run after that. It is not clear to me how the build automation handles the authenticode signing of the DLLs.

Comment 6

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

(In reply to Mike Hommey [:glandium] from comment #2)
> Created attachment 669889 [details] [diff] [review]
> PoC
> 
> This is very crude, but works for me on linux, except all fips tests fail
> because we don't sign libraries.
> The change to security/nss/tests/fips/fips.sh makes the disk space waste
> copying all files from DIST/lib less bad, but it's still copying a 700MB
> libxul.so multiple times...

FIPS tests can be temporarily disabled by setting NSS_TEST_DISABLE_FIPS=1 in the environment.

Our goal is to make this work as a separate test suite (so we'll have a new letter N next to M (1 2 3 4 5 6) and X in tbpl), which means we'll need to package up the test executables and libraries and ship them off somewhere.

It is worth looking at the work that was done in bug 772144. It seems like for that Android work, Bob made good use of the "make package" target to build a tarball with all the executables and libraries in it. Perhaps doing something similar for all builds would make sense, to minimize the copying overhead, especially considering we're going to soon move these tests off the builders anyway.

Comment 7

[(not currently active) Ted Mielczarek]

We have a "package-tests" target that complements "make package":
http://mxr.mozilla.org/mozilla-central/source/testing/testsuite-targets.mk#305

We package up all the test files and test harnesses into one package that gets uploaded alongside the build, and then that's used to run each test suite against the packaged build.

If you're going to go this route and the ultimate goal is to have it running on tinderbox, and you'd also like to have them run on Android devices, then the best plan is probably to write a Python driver script for the harness. We have lots of Python libraries useful for writing unit test harnesses, and we also have Python modules for running tests on mobile devices, so it'll be a lot easier to make things work by leveraging all that.

An example of a really simple test harness I put together recently is:
http://mxr.mozilla.org/mozilla-central/source/testing/runcppunittests.py

It just runs the compiled C++ unit tests currently, and doesn't have any support for running on-device, but if we were packaging the C++ unit tests we could use this script to run them against a packaged build very easily.

The xpcshell test harness is also informative, because we also have a version of it that runs tests on mobile devices:
http://mxr.mozilla.org/mozilla-central/source/testing/xpcshell/runxpcshelltests.py
http://mxr.mozilla.org/mozilla-central/source/testing/xpcshell/remotexpcshelltests.py

You can see that we normally accomplish this by subclassing a test harness class and then overriding specific methods that need to do things differently.

Comment 8

[Kai Engert (:KaiE:)]

(In reply to Brian Smith (:bsmith) from comment #6)
> 
> It is worth looking at the work that was done in bug 772144. It seems like
> for that Android work, Bob made good use of the "make package" target to
> build a tarball with all the executables and libraries in it. Perhaps doing
> something similar for all builds would make sense, to minimize the copying
> overhead, especially considering we're going to soon move these tests off
> the builders anyway.

I had improved Bob's work in bug 784316, it's checked in.

make package_for_testing

Comment 10

[(not currently active) Ted Mielczarek]

This bug differs slightly from what bug 814773 proposed, I'm not sure if it matters or not.

bug 814773 proposed building NSS as a standalone project and then running the tests.

This bug proposes building the NSS tests as part of a Firefox build and running the tests during "make check".

The former is probably still needed in order to run per-checkin tests for NSS, since we only uplift NSS snapshots/releases into mozilla-central.

The latter provides a useful sanity check because we're testing the exact bits we ship with Firefox (same toolchain, compiler flags, etc).

Comment 11

[Pete Moore [:pmoore][:pete]]

Hey Glandium,

What is the status of this one? Trying to clean up the NSS queue.

Thanks,
Pete

Comment 12

[Mike Hommey [:glandium]]

I don't know. Brian should be able to tell whether this is still something of interest or not.

Comment 13

[Brian Smith (:briansmith, :bsmith, use NEEDINFO?)]

It would absolutely be great if we could run the NSS tests along with the Gecko tests (except for the certificate tests). The NSS tests run on the nss-crypto.org buildbot [1], however those tests are testing a very different configuration of NSS than what we use in Gecko. I do not think it is an extraordinary amount of work to get the NSS tests running in TBPL but it requires somebody that is familiar with the work of adding new test suites to TBPL and adapting existing tests to work with our infrastructure (adapting the test output to be prefixed with TEST-UNEXPECTED-FAIL, etc.).


[1] https://bot.nss-crypto.org:8011/waterfall?reload=180