1404472 - Add a mach command to generate a report of (# tests, # skipped, # failed) per manifest/directory

Status: RESOLVED FIXED

(Testing :: General, enhancement, P3)

Description

[Geoff Brown [:gbrown]]

I have a prototype based on TestResolver.

Comment 1

[Geoff Brown [:gbrown]]

Attachment: wip / prototype

$ ./mach test-report netwerk
Test manifest: /home/gbrown/src/netwerk/test/reftest/reftest.list
  4 tests, 0 skipped on one or more platforms, 0 skipped on all platforms, 0 failed on one or more platforms
Test manifest: /home/gbrown/src/netwerk/test/browser/browser.ini
  6 tests, 2 skipped on one or more platforms, 0 skipped on all platforms, 0 failed on one or more platforms
Test manifest: /home/gbrown/src/netwerk/test/unit/xpcshell.ini
  308 tests, 19 skipped on one or more platforms, 2 skipped on all platforms, 3 failed on one or more platforms
Test manifest: /home/gbrown/src/netwerk/test/mochitests/mochitest.ini
  16 tests, 0 skipped on one or more platforms, 0 skipped on all platforms, 0 failed on one or more platforms
Test manifest: /home/gbrown/src/netwerk/test/httpserver/test/xpcshell.ini
  29 tests, 1 skipped on one or more platforms, 0 skipped on all platforms, 0 failed on one or more platforms
Test manifest: /home/gbrown/src/netwerk/test/crashtests/crashtests.list
  4 tests, 0 skipped on one or more platforms, 0 skipped on all platforms, 0 failed on one or more platforms
Test manifest: /home/gbrown/src/netwerk/cookie/test/unit/xpcshell.ini
  7 tests, 0 skipped on one or more platforms, 0 skipped on all platforms, 0 failed on one or more platforms
Test manifest: /home/gbrown/src/netwerk/test/unit_ipc/xpcshell.ini
  43 tests, 43 skipped on one or more platforms, 0 skipped on all platforms, 0 failed on one or more platforms
Test manifest: /home/gbrown/src/netwerk/cookie/test/browser/browser.ini
  1 tests, 0 skipped on one or more platforms, 0 skipped on all platforms, 0 failed on one or more platforms
Total: 418 tests, 65 skipped on one or more platforms, 2 skipped on all platforms, 3 failed on one or more platforms


Known limitations:
 - doesn't understand reftest annotations
 - doesn't count reftests "properly"

Also, it would be nice to generate a report for a bugzilla component (list of components for an owner); I don't know how to do that.

Comment 2

[Joel Maher ( :jmaher ) (UTC -8)]

ideally I would like to have a dashboard that does this- in fact davehunt has a dashboard that queries activedata and generates some related information:
http://fxtestr.us-west-2.elasticbeanstalk.com/unittest?branch=autoland&path=netwerk&since=today-1week

It might have different limitations based on the data/availability.

Overall this would be useful- especially if you could query further:
./mach test-report netwerk --verbose
<lists the tests and related skipped configs>

./mach test-report netwerk --verbose --suite=xpcshell
<only lists xpcshell related tests>


the 4 different data points here are interesting:
Total: 418 tests, 65 skipped on one or more platforms, 2 skipped on all platforms, 3 failed on one or more platforms

I am curious what the failed status is?  is that a different item in a manifest?

Comment 3

[Geoff Brown [:gbrown]]

(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #2)

Thanks for the feedback -- much appreciated.

> I am curious what the failed status is?  is that a different item in a
> manifest?

This is the difference between skip-if and fail-if. With proper support for reftests, it would get more complicated: fuzzy-if, random-if, etc...not sure how important all these distinctions are.

Comment 4

[Geoff Brown [:gbrown]]

Attachment: wip / prototype

This version support --component:

$ ./mach test-report --component Core::Layout
Test manifest: /home/gbrown/src/layout/reftests/backgrounds/reftest.list
  216 tests, 0 skipped on one or more platforms, 0 skipped on all platforms, 0 failed on one or more platforms
Test manifest: /home/gbrown/src/layout/reftests/css-blending/reftest.list
  120 tests, 0 skipped on one or more platforms, 0 skipped on all platforms, 0 failed on one or more platforms
Test manifest: /home/gbrown/src/layout/reftests/border-dotted/reftest.list
  12 tests, 0 skipped on one or more platforms, 0 skipped on all platforms, 0 failed on one or more platforms
...

The implementation is naive / brute-force-ish and performance is not great, but not too bad either: a few seconds for component lookup on the top src dir, in my experience.

Comment 5

[Joel Maher ( :jmaher ) (UTC -8)]

this seems useful to maybe include in |./mach test-info| ?  I would like to get these stats easier, and applying a patch to get them is not ideal :)

Comment 6

[Geoff Brown [:gbrown]]

Attachment: wip / prototype

Now provided as a subcommand of test-info:

  mach test-info report [--component product::component] [--verbose]

I'm not comfortable checking this in until I add proper reftest manifest support. Sorry.

Comment 7

[Geoff Brown [:gbrown]]

Attachment: Bug 1404472 - Add mach subcommand for per-component test-info report; r=jmaher

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d76bbec5d855
Add mach subcommand for per-component test-info report; r=jmaher

Comment 9

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/d76bbec5d855