567042 - 'cfx testall' should run tests in example packages

Status: RESOLVED FIXED

(Add-on SDK Graveyard :: General, defect, P1)

Description

[Atul Varma [:atul]]

Entering 'examples/reading-data' and running 'cfx test' currently results in the following error:

  error: TEST FAILED: test-main.test_id (failure)
  error: fail: "{6724fc1b-3ec4-40e2-8583-8061088b3185}" != "MyUniqueID"

This particular failure isn't so important as is the fact that the example packages aren't having their tests run on 'cfx testall'. Because of this, we froze the tree for 0.4 yesterday without realizing that some of our tests were actually failing.

Comment 1

[Dietrich Ayala (:dietrich)]

Tentatively blocking 0.4 on this. It's ok to ship with known issues - but we should at least know what they are.

Comment 2

[Myk Melez [:myk] [@mykmelez]]

Atul: is this something you can look at?

Comment 3

[Atul Varma [:atul]]

I've just added 'cfx testex', which tests all example code:

  http://hg.mozilla.org/labs/jetpack-sdk/rev/6d1036b00989

I'm thinking that next it might be most useful to make 'cfx testpkgs', which essentially does what 'cfx testall' currently does, and changing 'cfx testall' to effectively run 'cfx testcfx && cfx testpkgs && cfx testex'. All of them unfortunately require different test runners and/or environments, which is why there are currently three different ones. Oye.

Comment 4

[Myk Melez [:myk] [@mykmelez]]

(In reply to comment #3)
> I've just added 'cfx testex', which tests all example code:
> 
>   http://hg.mozilla.org/labs/jetpack-sdk/rev/6d1036b00989

Does that mean we can resolve this bug fixed?  Or do you think we still need to make testall include those tests?


> I'm thinking that next it might be most useful to make 'cfx testpkgs', which
> essentially does what 'cfx testall' currently does, and changing 'cfx testall'
> to effectively run 'cfx testcfx && cfx testpkgs && cfx testex'. All of them
> unfortunately require different test runners and/or environments, which is why
> there are currently three different ones. Oye.

We should probably target the overall cleanup of these testing commands for 0.5 while doing the simplest, lowest-risk fix for 0.4, given where we're at.  Can we just get cfx testall to also run cfx testex in addition to what it currently runs?

Comment 5

[Atul Varma [:atul]]

Well, we should probably also have 'testall' do 'testcfx' in addition to what it currently does too, because currently that isn't being run either. Alas.

Comment 6

[Atul Varma [:atul]]

Ok, it should be fixed now:

  http://hg.mozilla.org/labs/jetpack-sdk/rev/94949f3c59fa

That said, the reading-data example still fails, which we need to fix asap.

Comment 7

[Myk Melez [:myk] [@mykmelez]]

The Add-on SDK is no longer a Mozilla Labs experiment and has become a big enough project to warrant its own Bugzilla product, so the "Add-on SDK" product has been created for it, and I am moving its bugs to that product.

To filter bugmail related to this change, filter on the word "looptid".