Closed Bug 443324 Opened 12 years ago Closed 11 years ago

Ability to run tests without rebuilding (--enable-tests)

Categories

(Release Engineering :: General, defect, critical)

defect
Not set
critical

Tracking

(Not tracked)

RESOLVED DUPLICATE of bug 421611

People

(Reporter: mtschrep, Assigned: ted)

References

Details

Each of our unittest boxes does a full rebuild from source with --enable-tests.  This is particularly painful on windows which, thanks to PGO, takes many hours to do.  If we could treat unittests like talos and run the tests separately from the builds many hours would be saved.   Options:

a) We can isolate the changes --enable-tests do to the build and post-process through a script
b) We can just always build with -enable-tests (except for the nightlies)
....
Whiteboard: DUPEME
Depends on: 443327
Volunteering Ted to help either determine that this is a crazy idea or get it done :-).
Assignee: nobody → ted.mielczarek
No longer depends on: 443327
Reed is right in that we have some other bugs on file (probably bug 383136 or bug 421611 are the best match for this). The main problem here is that we have a lot of tests that need to be compiled to be run (all of our C++ unit tests), and aren't packaged with the build. In addition, we have a whole class of unit tests that depend on xpcshell, which doesn't get packaged with the build.

Overall, this is a noble goal, but it's not easy, and I'm still not sure it's really worth the effort.
I just had a look at the unittest Buildbot waterfall. Here are the times for the source update/compile portion of a run on various machines:
1.9:
qm-centos5-01/02/04: 3 minutes
qm-xserve01/06: 3 minutes
qm-win2k3-01/02: 5 minutes
qm-win2k3-pgo01 (PGO!): 1h33min
1.9.1:
qm-centos5-moz2-01/qm-centos5-03: 2 minutes
qm-moz2mini01 (os x): 2 minutes
qm-win2k3-moz2-01/03: 7 minutes

Some of these builds have changes, some do not. Without trawling the Waterfall I would guess that all of the unit test machines (minus PGO) do update+compile in less than 10 minutes.
(In reply to comment #3)
> I just had a look at the unittest Buildbot waterfall. Here are the times for
> the source update/compile portion of a run on various machines:
> 1.9:
> qm-centos5-01/02/04: 3 minutes
> qm-xserve01/06: 3 minutes
> qm-win2k3-01/02: 5 minutes
> qm-win2k3-pgo01 (PGO!): 1h33min
> 1.9.1:
> qm-centos5-moz2-01/qm-centos5-03: 2 minutes
> qm-moz2mini01 (os x): 2 minutes
> qm-win2k3-moz2-01/03: 7 minutes
> 
> Some of these builds have changes, some do not. Without trawling the Waterfall
> I would guess that all of the unit test machines (minus PGO) do update+compile
> in less than 10 minutes.
> 

Right - the minus-PGO is the big problem here - that puts our average build/compile time at 1.5 hrs...

(In reply to comment #2)
> Reed is right in that we have some other bugs on file (probably bug 383136 or
> bug 421611 are the best match for this). The main problem here is that we have
> a lot of tests that need to be compiled to be run (all of our C++ unit tests),
> and aren't packaged with the build. In addition, we have a whole class of unit
> tests that depend on xpcshell, which doesn't get packaged with the build.

When we package according to the manifest, could we also create a zip that is "everything in dist that _isn't_ in the manifest", and just distribute that firefox-flotsam.zip to the test machines as well as the packaged builds?  Seems like a straightforward packaging step to add, and then it's just some unzip and copy to get the right bits in place.  Enthusiastic testers could also grab the flotsam zip to run tests and so forth.
It is definitely possible to create a separately installable bit that contains tests and can point to any browser. See:

http://xoatlicue.blogspot.com/2008/01/standalone-test-product.html

http://xoatlicue.blogspot.com/2008/03/really-neat-test-reusability-situation.html

I guess when I posted this, it was still in the "crazy idea" phase of development.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → DUPLICATE
Duplicate of bug: 421611
Product: mozilla.org → Release Engineering
Whiteboard: DUPEME
You need to log in before you can comment on or make changes to this bug.