Open Bug 673001 Opened 9 years ago Updated 6 years ago

Show commands to run failing unittests, after the TEST-UNEXPECTED-FAIL

Categories

(Testing :: General, defect)

defect
Not set

Tracking

(Not tracked)

People

(Reporter: cmtalbert, Unassigned)

Details

On TBPL, when things fail, it's not always apparent to most developers how to go run that section of tests on their own.  This would be a nice addition.

So, I'm in no position to dictate UI, so, let's say that when a test fails on TBPL, the person reviewing the test has some means to find out what command they need to use for re-running that suite of tests locally. (However, some means should not include digging through the build log). They should be re-run as close to what the tinderbox runs, so here's the information we'd need.

* URL to download the binary (url1)
* URL to download the test package (url2)
* URL to download the symbols (url3)
* The command line to run the tests

This could very nearly be a bash script that is filled in by a process that scrapes the information from the build log.  The bash script could be downloaded from TBPL and the developers could run the script.  Something like:
wget <url1>
wget <url2>
wget <url3>
if windows
  unzip <url1>
elif linux
  tar -xjf <url1>
else
  hdiutil whatever <url1>
unzip -d tests <url2>
unzip -d symbols <url3>
python runtests.py blah blah blah blah blah


For Android based tests, we could do the same thing, but that has the extra issue of requiring the developer to have an android environment to debug in, so let's skip android for this for now, as I have a different solution for that in mind.
Shouldn't this stuff live in MDN?  I don't think that TBPL is the best place for our developer documentation to live.  :-)
(In reply to comment #0)
> python runtests.py blah blah blah blah blah

FTR, that should be `make mochitest-{plain,chrome,browser-chrome}`. We shouldn't recommend the usage of `python runtests.py`.
Guys, I'm not talking about documenting anything on TBPL.  I'm talking about having a way to click on something and get a script that will run *exactly what tinderbox runs* for a failing test.

I don't know how many times I've personally pulled that information out of the build logs for developers.  It seems like it'd be quite the win to give people the ability to get to this information easily without parsing buildlogs, knowing what to download, what to unzip etc.

And you can't run things the way the automation does with make "<testtarget>"  The make targets are simply shortcuts to "python runtests.py <arguments>" We provide the make targets so that developers with build trees handy can quickly run simple tests that they are likely to want.  Also, we have the implicit rule that those make targets do not hit the network.  In order to truly run the way the automation does, with the builds the automation produced, you need to download symbols and test package from a specific changeset and we won't be adding that functionality to the make target.

If this wouldn't be useful, then we can wontfix this bug and go on with life.  But, given the number of times I've pulled this information together for developers, I think it deserves a some thought.
(In reply to comment #3)
> I don't know how many times I've personally pulled that information out of the
> build logs for developers.  It seems like it'd be quite the win to give people
> the ability to get to this information easily without parsing buildlogs,
> knowing what to download, what to unzip etc.

Yes, I totally understand (me being one of those people who has tried to dig this information in the past).

But I still think that my suggestion about this being documented on MDN is valid.  Once we have that, we can _definitely_ link to that documentation from TBPL (in other words, I'm not advocating this bug to be WONTFIXed).
I'm all for this. (Full disclosure: I think this bug was filed as a result of a comment I made during the initial Go Fast meeting.)

Optimally, I'd like it to allow a couple of things:
1. The "run the exact test suite that failed as tinderbox did" that ctalbert's describing above
2. An option to run an individual failing test (presumably by generating a manifest for it or something)
3. An option to do both of the above using your own local compile
4. For all of the above, an option (or instructions, when relevant) to run under a debugger.

I've read through the MDN docs, repeatedly, and it's still a major pain to figure out how to run tests. Especially since when I finally manage to run them, some problems won't reproduce on my machine even though they're 100% consistent on tinderbox/buildbot. Then I switch to another test suite, and I have to start all over again.

I'd prefer if it did the setup (pull down the files etc.), then spat out the commands to run rather than doing everything automagically.

As an additional nice to have, I'd like a way to run the focus-hogging tests under Xnest or Xvfb or whatever (and the moral equivalent on Windows.) And progress indicators. No pony, though -- I want a friggin' unicorn to go along with this.

I wouldn't expect tbpl code to be doing most of the work for this, though -- can it hit a CGI that just takes a test path or something?
Sorry, I had that queued up since before Ehsan's comment.

It's great to have it documented on MDN, but you still have to scan through twenty pages of log output to figure out what was run. Linking directly from a tbpl-visible type of test to the correct portion of MDN would help, but I suspect it's not enough to get people over the usability hurdle.
Whiteboard: [buildfaster:p1]
I believe this would be more suited to being output in the logs, along with the TEST-UNEXPECTED-FAIL, otherwise we're going to pollute every TBPL annotated summary, with a bunch more lines of generic copy (which sheriffs have to skim through every time before starring intermittent orange, to ensure there are no additional failures further down in the log).
Component: Tinderboxpushlog → General
Product: Webtools → Testing
Summary: Show commands to run failing unittests on TBPL → Show commands to run failing unittests, after the TEST-UNEXPECTED-FAIL
(In reply to Ed Morley [:edmorley UTC+1] from comment #7)
> I believe this would be more suited to being output in the logs, along with
> the TEST-UNEXPECTED-FAIL, otherwise we're going to pollute every TBPL
> annotated summary, with a bunch more lines of generic copy (which sheriffs
> have to skim through every time before starring intermittent orange, to
> ensure there are no additional failures further down in the log).

Perhaps, once we switch to mozharness for unit tests, we can send people to a page with information on how to use mozharness locally and run such unit tests (bug 650880).
actually now that we are on mozharness, it has become more difficult to run these by hand.  I would think we need a tool in tree that we could point to a log file and it could help us work the magic?
You need to log in before you can comment on or make changes to this bug.