617536 - Difficult to figure out why browser-chrome tests are not being found

Status: RESOLVED WONTFIX

(Testing :: Mochitest, defect)

Description

[Steve Fink [:sfink] [:s:]]

Attachment: Add a "List Tests" option with info about where tests are searched for

I recently wrote my first browser-chrome mochitest, and one of the hardest parts was simply getting it to see my test. Admittedly, I'm not very good at following directions, but it didn't help that the instructions at https://developer.mozilla.org/en/Browser_chrome_tests were a little off (I just fixed them; they had the wrong path for plain mochitests.)

But it's a tricky process. You have to get your makefile to put the tests in the right place, name your test correctly, and pass in the correct --test-path. The path _tests/testing/mochitest/tests (or /browser, /chrome, etc.) mentions "test" 4 times, and it's very unclear what each level is for. But worst of all, if you get anything wrong it just says "whoops, no test files were found. You screwed up your --test-path" which may or may not be the problem. It makes sense for people who want to run existing tests, but is misleading for newbie test developers like me. (Worst of all, if you don't give it a --test-path, it runs all the other tests and you have to wait, then dig through all the output to discover your test was never run.)

Unfortunately, the structure of the code makes it difficult to produce a more helpful error message, but I'm attaching a patch I wrote after getting everything straightened out that would have made things go much faster for me. I don't claim its beautiful code, but the interesting information is sort of hidden by the abstraction layers right now, so I just tried to come up with a way to dig out the data that wasn't horribly invasive.

This patch adds a "List Tests" button in addition to the "Run Tests" button, and clicking on it provides the necessary information about where on the filesystem it is scanning for your test files. It has some major flaws, and only works for browser-chrome tests (because browser-chrome doesn't share the plain mochitest test-finding logic, unfortunately). I believe it improves the current state, though, and can at least serve as a POC feature request.

Comment 1

[:Gavin Sharp [email: [e-mail]]]

I agree that the setup is confusing. Bug 370750 should help, by making it so that you don't need to really care about (as much) Makefile goop.

I don't really want to take your patch, because I think its impact on harness code/UI complexity outweighs the potential benefit to someone getting tests running for the first time.

I am definitely open to other ideas that make the initial setup process easier, though. The "browser_" filename restriction should be called out in the "Adding a new test" section of those docs rather than earlier - I've made that change. We could perhaps also tweak the "no tests" error to mention that the test-path should be your test file's location relative to the root of the source tree.

Comment 2

[Steve Fink [:sfink] [:s:]]

What I really wanted, and originally set out to do, was to add in a couple bits of information to the "no tests" error. Minimally, I wanted the path that was searched, and preferably the pattern that it was looking for too. But given the way things are set up, that seemed harder than the List Tests button.

Hm... unless it could unconditionally print that stuff out? Then at least I could see it in the logs when I got the "no test" error. I don't know how verbose that would end up being, though.

I didn't expect this patch to be used, but I'd really like *something* in addition to the doc improvements.

Comment 3

[Steve Fink [:sfink] [:s:]]

mach seems to have made this problem go away, or at least is the mechanism by which this would be fixed, so this bug is not useful.