598760 - Stop telling people to run (our failing) unit tests as their introduction to the SDK

Status: RESOLVED FIXED

(Add-on SDK Graveyard :: General, defect)

Description

[Drew Willcoxon :adw]

Our unit tests are pretty unstable.  We can't even get them all to pass ourselves.  People's introduction to the SDK via the Getting Started guide is a screen full of failures and warnings that they don't understand and that are probably not relevant to what they're trying to do.  People write to the mailing list wondering what they've done wrong, but they usually haven't done anything wrong.  People have pinged me on IRC wondering why tests fail or why the number of tests that are run is different for Firefox 3.6 than for the betas, but usually that's irrelevant trivia.  It's an obstacle to using Jetpack.

So the Getting Started guide should just mention running `cfx` rather than `cfx testall`.  Probably the testall info should be moved to a troubleshooting or some such page.

Patch coming up soon.

Comment 1

[Atul Varma [:atul]]

I guess that is reasonable. It was added to the docs back in the days when there were very few (or zero) random-fail tests and all the tests took under a second to run.

Another thing I've noticed is that because a lot of the tests do things with the GUI and make assertions about window focus, I can't just type "cfx testall" and then go check my email, because then some kind of test will fail. This means that even if our tests don't random-fail yet also assume that the user doesn't touch their computer for the duration of the run, we could get false bug reports if the user starts multitasking while running tests (or if Norton Antivirus pops up while the tests are running, etc).

Comment 2

[Drew Willcoxon :adw]

Attachment: patch

Adds a Troubleshooting section to the Reference, talks about running only `cfx` instead of `cfx testall` in Getting Started.

Comment 3

[Drew Willcoxon :adw]

Attachment: patch 2

Updates the example cfx testall output and adds "Check Your Firefox or XULRunner" and "Don't Leave Non-Jetpack Files Lying Around" sections to Troubleshooting.

Atul, please feel free to give feedback if you want.

Comment 4

[Atul Varma [:atul]]

Comment on attachment 477741 [details] [diff] [review]
patch 2

nice!

i am a bit wary of displaying the cfx help output on that tutorial page, partly b/c it's a giant wall of text and partly b/c unlike example test output it is actually useful information that users might end up looking at instead of the cfx tool docs--and if we actually have to keep it up-to-date too then that could suck.

what if we had the user run 'cfx testcfx'?  that at least kicks the tires on cfx and should raise any weird python version incompat issues that arise...  hmm.

i really dig the troubleshooting page. only suggestion would be to link to a mibbit page that puts them in the right server+channel instead of using the irc: protocol, as lots of folks don't use irc (and those that do will be able to figure out where to go on their own). in fact it might even be cool to just embed the mibbit widget in in the troubleshooting page itself, to lower the barrier even more.

anyhow, overall i think this is good stuff so i will r+ it; you are welcome to make changes based on my suggestions, no worries if you don't.

Comment 5

[Drew Willcoxon :adw]

(In reply to comment #4)
> cfx tool docs--and if we actually have to keep it up-to-date too then that
> could suck.

Yeah, good point.  Though we already have to keep references to the current version number up to date, so maybe adding this task wouldn't be too burdensome.

> what if we had the user run 'cfx testcfx'?  that at least kicks the tires on
> cfx and should raise any weird python version incompat issues that arise... 

Hmm.  I do like the fact that running `cfx` alone introduces people to it.  They get to see all the commands and options.  testcfx on the other hand doesn't provide any learning to a new user.  And IIRC from the mailing list, Python incompat issues start arising when people run the activate script.

> to figure out where to go on their own). in fact it might even be cool to just
> embed the mibbit widget in in the troubleshooting page itself, to lower the
> barrier even more.

A great idea.  I looked into creating a widget, but they require you to create mibbit widget manager account to do so, which I'm not comfortable doing.  I'll link to them instead.

Comment 6

[Drew Willcoxon :adw]

Attachment: patch 3

* Keeps the `cfx` step in Getting Started but lists only the first line
  of its output.
* Adds a Python section to Troubleshooting.
* Links to #jetpack on mibbit.

Comment 7

[Drew Willcoxon :adw]

http://hg.mozilla.org/labs/jetpack-sdk/rev/2f74f9bf0e2d

Comment 8

[Myk Melez [:myk] [@mykmelez]]

The Add-on SDK is no longer a Mozilla Labs experiment and has become a big enough project to warrant its own Bugzilla product, so the "Add-on SDK" product has been created for it, and I am moving its bugs to that product.

To filter bugmail related to this change, filter on the word "looptid".