443324 - Ability to run tests without rebuilding (--enable-tests)

Status: RESOLVED DUPLICATE of bug 421611

(Release Engineering :: General, defect)

Description

[Mike Schroepfer]

Each of our unittest boxes does a full rebuild from source with --enable-tests.  This is particularly painful on windows which, thanks to PGO, takes many hours to do.  If we could treat unittests like talos and run the tests separately from the builds many hours would be saved.   Options:

a) We can isolate the changes --enable-tests do to the build and post-process through a script
b) We can just always build with -enable-tests (except for the nightlies)
....

Comment 1

[Mike Schroepfer]

Volunteering Ted to help either determine that this is a crazy idea or get it done :-).

Comment 2

[(not currently active) Ted Mielczarek]

Reed is right in that we have some other bugs on file (probably bug 383136 or bug 421611 are the best match for this). The main problem here is that we have a lot of tests that need to be compiled to be run (all of our C++ unit tests), and aren't packaged with the build. In addition, we have a whole class of unit tests that depend on xpcshell, which doesn't get packaged with the build.

Overall, this is a noble goal, but it's not easy, and I'm still not sure it's really worth the effort.

Comment 3

[[e-mail] (:bhearsum)]

I just had a look at the unittest Buildbot waterfall. Here are the times for the source update/compile portion of a run on various machines:
1.9:
qm-centos5-01/02/04: 3 minutes
qm-xserve01/06: 3 minutes
qm-win2k3-01/02: 5 minutes
qm-win2k3-pgo01 (PGO!): 1h33min
1.9.1:
qm-centos5-moz2-01/qm-centos5-03: 2 minutes
qm-moz2mini01 (os x): 2 minutes
qm-win2k3-moz2-01/03: 7 minutes

Some of these builds have changes, some do not. Without trawling the Waterfall I would guess that all of the unit test machines (minus PGO) do update+compile in less than 10 minutes.

Comment 4

[Mike Schroepfer]

(In reply to comment #3)
> I just had a look at the unittest Buildbot waterfall. Here are the times for
> the source update/compile portion of a run on various machines:
> 1.9:
> qm-centos5-01/02/04: 3 minutes
> qm-xserve01/06: 3 minutes
> qm-win2k3-01/02: 5 minutes
> qm-win2k3-pgo01 (PGO!): 1h33min
> 1.9.1:
> qm-centos5-moz2-01/qm-centos5-03: 2 minutes
> qm-moz2mini01 (os x): 2 minutes
> qm-win2k3-moz2-01/03: 7 minutes
> 
> Some of these builds have changes, some do not. Without trawling the Waterfall
> I would guess that all of the unit test machines (minus PGO) do update+compile
> in less than 10 minutes.
> 

Right - the minus-PGO is the big problem here - that puts our average build/compile time at 1.5 hrs...

Comment 5

[Mike Shaver (:shaver -- probably not reading bugmail closely)]

(In reply to comment #2)
> Reed is right in that we have some other bugs on file (probably bug 383136 or
> bug 421611 are the best match for this). The main problem here is that we have
> a lot of tests that need to be compiled to be run (all of our C++ unit tests),
> and aren't packaged with the build. In addition, we have a whole class of unit
> tests that depend on xpcshell, which doesn't get packaged with the build.

When we package according to the manifest, could we also create a zip that is "everything in dist that _isn't_ in the manifest", and just distribute that firefox-flotsam.zip to the test machines as well as the packaged builds?  Seems like a straightforward packaging step to add, and then it's just some unzip and copy to get the right bits in place.  Enthusiastic testers could also grab the flotsam zip to run tests and so forth.

Comment 6

[Ray Kiddy]

It is definitely possible to create a separately installable bit that contains tests and can point to any browser. See:

http://xoatlicue.blogspot.com/2008/01/standalone-test-product.html

http://xoatlicue.blogspot.com/2008/03/really-neat-test-reusability-situation.html

I guess when I posted this, it was still in the "crazy idea" phase of development.