804057 - Implement XULTest for compiled XUL unit test

Status: RESOLVED DUPLICATE of bug 767231

(Testing :: General, defect)

Description

[Benoit Girard (:BenWa)]

Attachment: Missing ok/todo implementation + mozconfig option

The aim is to introduce compiled C++ test that are linked against XUL that can use anything provided by XUL. There is no XUL initialization that takes place such as thebes/xpcom as this is not meant to test integration. This framework is ideal for unit testing modular function and objects such as TiledLayerBuffer and the sample test provided.

XULTest.h has a description of how to write a unit test:
// XULTest is a framework for unit test that are linked against
// XUL. To write a test create a file TestDESCRIPTION.cpp and
// add the following to the make file:
//
//   XULTTEST_CPPSRCS = TestDESCRIPTION.cpp
//
// In TestDESCRIPTION.cpp create one method per test with the following
// signature:
//
//  static void testCaseDescription();
//  XULTEST_REGISTER(testCaseDescription);
//
// Each test case should perhaps a series of check using the assertion
// methods provided in this header. These are modeled after mochitest's
// SimpleTest.js.

These tests are NOT to ship and would not be compiled/linked to release versions of XUL but would need to be triggered from a mozconfig option. Thus libxul can have no dependency on xultest.

Comment 1

[Benoit Girard (:BenWa)]

Sample run:

~/ssd-mozilla/mozilla-central/builds/obj-ff-64gdb/dist/bin> ./firefox -unittest
TEST-INFO | Starting XULTest
TEST-INFO | Running all tests
TEST-INFO | Running Test '/Volumes/SSD-Mac1/Users/benoitgirard/ssd-mozilla/mozilla-central/tree/gfx/layers/TestLayerBuffer.cpp#TestEmptyUpdate' 1/3
TEST-PASS | /Volumes/SSD-Mac1/Users/benoitgirard/ssd-mozilla/mozilla-central/tree/gfx/layers/TestLayerBuffer.cpp#TestEmptyUpdate
TEST-INFO | Running Test '/Volumes/SSD-Mac1/Users/benoitgirard/ssd-mozilla/mozilla-central/tree/gfx/layers/TestLayerBuffer.cpp#testConstructor' 2/3
TEST-PASS | /Volumes/SSD-Mac1/Users/benoitgirard/ssd-mozilla/mozilla-central/tree/gfx/layers/TestLayerBuffer.cpp#testConstructor
TEST-INFO | Running Test '/Volumes/SSD-Mac1/Users/benoitgirard/ssd-mozilla/mozilla-central/tree/gfx/layers/TestLayerBuffer.cpp#testTileStart' 3/3
TEST-PASS | /Volumes/SSD-Mac1/Users/benoitgirard/ssd-mozilla/mozilla-central/tree/gfx/layers/TestLayerBuffer.cpp#testTileStart

Comment 2

[Benoit Jacob [:bjacob] (mostly away)]

There are two possible approaches to compiled libxul unit tests:
 Approach 1. Build tests directly into libxul
 Approach 2. Link libxul a second time with all symbols visible by default. Link tests to it rather than adding them to it.

IIUC the present patch is Approach 1. FWIW I personally prefer approach 2 because of a somewhat abstract concern that it's preferable to be as unintrusive as possible to libxul. Anyway, it's good to start having tests and fixing the test harness API and we can then switch build approaches later.

Comment 3

[Benoit Girard (:BenWa)]

We've discussed this offline in Toronto and we all agree we should link a second libxul test. I'll continue the patch in this direction. If we change our mind it should be easy to revert this decision.

Comment 4

[(no longer active)]

Comment on attachment 673763 [details] [diff] [review]
Missing ok/todo implementation + mozconfig option

So here's a number of comments that I have about the patch.  This is a summary of our lunch conversation:

1. I think it would be a bit better if we link libxul twice as opposed to having test code inside it which we expect to remove in release versions, as the former does not affect our users in any way, but the latter could in case we hit a weird edge case in PGO etc.

2. I think we should reuse the TestHarness.h API if we can (see https://developer.mozilla.org/en-US/docs/Compiled-code_automated_tests) and also look into improving that harness to make it look more similar to mochitest, etc.

3. About how much of libxul to initialize, I agree that it's better if we can avoid initializing much and let people initialize stuff as they need.  But as we've learned in the case of xpcshell tests, we'll soon end up with having to support alternate initialization methods for tons of stuff, and that will be using code different to the one that we use inside Firefox, which is a maintenance pain.  So I think it's more pragmatic to initialize everything that a normal Firefox process would before running the set of tests.

4. For registering test functions, instead of adding the path to the file name to the namespace, I think we can just use C++ namespaces, so that if you have a test function called TestThisVerySpecificThing, you can say:

XUL_TEST_REGISTER(TestThisVerySpecificThing);

and if you have a function called Test (which can be a generic name), you would be able to put it in a C++ namespace and just pass in the fully qualified name, like this:

XUL_TEST_REGISTER(foo::bar::Test);

Then we can just stringify the macro argument and ensure that the resulting strings are unique at runtime.

5. For running the tests, I think it would be very helpful if we can run xultest once with an argument to print the test manifest, and then run it multiple times once per test, like the following:

$ ./xultest -xultest-manifest
xultest: TestThisVerySpecificThing
xultest: foo::bar::Test
$ ./xultest -xultest TestThisVerySpecificThing
running TestThisVerySpecificThing...

This should make things easy for running individual tests in parallel, or groups of them if we add some fanciness which let you run all of the test in a namespace or something.

Comment 5

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

I outlined a strategy for doing this some time ago.  https://groups.google.com/d/topic/mozilla.dev.platform/Q8AJiU9yJDk/discussion

There has been some effort towards this in bug 767231.  I think I would prefer to reuse gtest rather than invent our own testing API and whatnot.

Comment 6

[(no longer active)]

(In reply to comment #5)
> I outlined a strategy for doing this some time ago. 
> https://groups.google.com/d/topic/mozilla.dev.platform/Q8AJiU9yJDk/discussion

I think that what I said in my feedback should be sort of a reiteration of the things that you said on that thread.  :-)  Do you think there's somewhere that we disagree?

> There has been some effort towards this in bug 767231.  I think I would prefer
> to reuse gtest rather than invent our own testing API and whatnot.

Using gtest is fine by me, I guess.  I think that using the gtest harness is orthogonal to whether or not we can use libxul stuff in our tests, though (but it does affect some of the other points I raised in my feedback.)

Comment 7

[Benoit Girard (:BenWa)]

(In reply to Kyle Huey [:khuey] ([e-mail]) from comment #5)
> I outlined a strategy for doing this some time ago. 
> https://groups.google.com/d/topic/mozilla.dev.platform/Q8AJiU9yJDk/discussion
> 
> There has been some effort towards this in bug 767231.  I think I would
> prefer to reuse gtest rather than invent our own testing API and whatnot.

I'm not looking at introducing a new testing API but rather using something like mochitest or TestHarness.h. The only thing new is the test registration. Are there any benefit to using gtest?

From what I can tell there hasn't been any traction on that front and this approach is very simple.

Comment 8

[Kyle Huey (Exited; not receiving bugmail, old account, do not use)]

(In reply to Ehsan Akhgari [:ehsan] from comment #6)
> (In reply to comment #5)
> > I outlined a strategy for doing this some time ago. 
> > https://groups.google.com/d/topic/mozilla.dev.platform/Q8AJiU9yJDk/discussion
> 
> I think that what I said in my feedback should be sort of a reiteration of
> the things that you said on that thread.  :-)  Do you think there's
> somewhere that we disagree?

No, I'm providing the backstory for this bug, not responding specifically to you.

Comment 9

[(not currently active) Ted Mielczarek]

(In reply to Benoit Girard (:BenWa) from comment #7)
> I'm not looking at introducing a new testing API but rather using something
> like mochitest or TestHarness.h. The only thing new is the test
> registration. Are there any benefit to using gtest?

Your comment 0 introduced the concept of test registration etc. I don't think we should reinvent that wheel, gtest already implements that. The benefit to using gtest is that it's a full-featured well-supported test framework, and people outside of Mozilla are familiar with it.

> From what I can tell there hasn't been any traction on that front and this
> approach is very simple.

If we do something else, then we'll have to rewrite a bunch of tests when we want to do the right thing. We should just do the right thing in the first place.

Not trying to be stop energy here, I think most of the work here is the linkage issues anyway, not the actual test harness.

Comment 10

[Benoit Girard (:BenWa)]

Since the work for GTest is near completion I'm convinced this approach is counter productive.