1061860 - Allow selection of tests to run sequentially + use the same profile

Status: RESOLVED WONTFIX

(Testing :: XPCShell Harness, defect)

Description

[Honza Bambas (:mayhemer)]

The goal is to let the xpcshell test run (actually a gecko instance) gracefully shutdown and persists "a stuff" in the profile.  Then, we want to run another test that use this same profile and checks the persistence of "the stuff" is correct.

The main purpose is checking our HTTP cache properly persists its data.  It cannot be done w/o the graceful shutdown in the middle.

This is actually very important to have, we completely lack automated testing of HTTP cache persistence correctness.

Comment 1

[(not currently active) Ted Mielczarek]

Just throwing an idea out there, we add a tag "persistent-profile = name", like:

[test1.js]
[test2.js]
persistent-profile = foo
[test3.js]
persistent-profile = foo
[test4.js]

In this scenario, tests 2 and 3 would be run sequentially, and the working directory (which includes the profile) would be persisted between them. Other tests would be run with a temporary directory as normal. Tests should be run in the order they're listed in the manifest.

Does that sound reasonable?

Comment 2

[(not currently active) Ted Mielczarek]

This does bend the expectations of our tests a bit, in that they generally expect to be standalone. We'd have to be careful of this during chunking and things like that.

An alternative would be to launch another xpcshell process from your test, and do that more than one time. That's a bit awakward though.

Comment 3

[Honza Bambas (:mayhemer)]

What about to have just a single test file, but have more then just a single run_test() function?  Rather have an array or just run_test2/3/4..().  But that would mean the test harness would need to keep state logic on such tests, doable, but potentially complicated.

Comment 4

[Honza Bambas (:mayhemer)]

And in the ini file one could specify a set of entry points, like:
[test_xxx.js]
entry_points=run_test,run_test2,run_test3

and all would run with the same temp profile.

Comment 5

[Honza Bambas (:mayhemer)]

(In reply to Honza Bambas (:mayhemer) from comment #4)
> And in the ini file one could specify a set of entry points, like:
> [test_xxx.js]
> entry_points=run_test,run_test2,run_test3
> 
> and all would run with the same temp profile.

This way you can also run the test as a single one with |./mach xpcshell-tests network/test/unit/test_my_persistence_check.js|

Comment 6

[Gregory Szorc [:gps]]

This seems a bit hacky for xpcshell tests. Wouldn't tests like this be the domain of Marionette tests? See also bug 940954.

Comment 7

[Honza Bambas (:mayhemer)]

WONTFIX according: no longer needed + comment 6.