695916 - make it as simple as possible for Gecko/Firefox developers to run Add-on SDK tests

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Myk Melez [:myk] [@mykmelez]]

It should be as simple as possible for Gecko/Firefox developers to run Add-on SDK tests.

Ideally, this would take the form of a make target like those for other automated test frameworks <https://developer.mozilla.org/en/Mozilla_automated_testing> that downloads the version of the SDK referenced in testing/jetpack/jetpack-location.txt <http://mxr.mozilla.org/mozilla-central/source/testing/jetpack/jetpack-location.txt> (if it isn't already downloaded), expands it, activates it, and triggers its test runner.

Comment 1

[[e-mail] (:bhearsum)]

RelEng doesn't own any test framework code, this is likely an ateam project.

Comment 2

[Jeff Hammel]

It is my understanding that make targets shouldn't require downloading or touching net in general. I could be wrong.  We should make the guidelines clearer in any case.

Comment 3

[cmtalbert]

Myk, what you're proposing makes sense to me.  I like the idea.  However, the convention has always been that the testing make targets do not touch the network.  If we can't break that, then there is little we can do here.  But, I believe this is really hurting quality for no tangible benefit.  I don't want a bunch of post-build make targets that do a bunch of downloading, but I think in this case it would be ok because:
A) the addon sdk won't be in the main codebase for a while (if ever)
B) this is ongoing battle between ffx developers and addonsdk developers is costing us developer productivity and the well-intentioned convention just isn't worth it in this case
C) we aren't downloading very much - it's about 3Mb.
D) we are downloading from our own hg server.  If that's down, we have bigger fish to fry.
E) I think we can actually throw a reasonable warning/error message if it fails.

Ted, would you be willing to look at a patch for this?  I'll volunteer to write it.

Comment 4

[Myk Melez [:myk] [@mykmelez]]

I appreciate the general principle that make targets should not touch the network, but it seems reasonable to make an exception here, given that the SDK's code is in a separate codebase.  If that isn't possible, however, then the next best thing would be to put a script into testing/jetpack/ that does the same thing, so running tests is still a one-step operation, even though the step is to invoke a script rather than a make target.

Perhaps the way to move forward on this bug is to write that script, which the make target, if we decide to implement one, would then invoke.

Comment 5

[Dave Townsend [:mossop]]

Looks like this is about the last remaining issue between us and unhiding the jetpack tests. Clint, do you think you'd still have time to write the patch for this?

Comment 6

[cmtalbert]

(In reply to Dave Townsend (:Mossop) from comment #5)
> Looks like this is about the last remaining issue between us and unhiding
> the jetpack tests. Clint, do you think you'd still have time to write the
> patch for this?

Yeah, I could do it. It will probably take about a week or two for me to get to it - I juggle quite a bit with mgmt stuff these days.  I'll write the script a la comment 4, and we can then use that script in a make target if we so decide we want one.

Comment 7

[Eddy Bruel [:ejpbruel]]

(In reply to Clint Talbert ( :ctalbert ) from comment #6)
> (In reply to Dave Townsend (:Mossop) from comment #5)
> > Looks like this is about the last remaining issue between us and unhiding
> > the jetpack tests. Clint, do you think you'd still have time to write the
> > patch for this?
> 
> Yeah, I could do it. It will probably take about a week or two for me to get
> to it - I juggle quite a bit with mgmt stuff these days.  I'll write the
> script a la comment 4, and we can then use that script in a make target if
> we so decide we want one.

I've been pushing for this, so I'd like to say thanks in advance for your effort! :-)

Comment 8

[Dave Townsend [:mossop]]

Bug 793928 should fix this as once we're in tree it will provide a simple "make jetpack-tests" and probably a mach command to run the tests.

Comment 9

[Dave Townsend [:mossop]]

Bug 793928 introduced a mach command and "make jetpack-tests" to run the jetpack test suite. It could be better (bug 837278) but I think we can call this fixed now.