Closed Bug 1404472 Opened 2 years ago Closed 3 months ago

Add a mach command to generate a report of (# tests, # skipped, # failed) per manifest/directory

Categories

(Testing :: General, enhancement, P3)

enhancement

Tracking

(firefox70 fixed)

RESOLVED FIXED
mozilla70
Tracking Status
firefox70 --- fixed

People

(Reporter: gbrown, Assigned: gbrown)

References

(Blocks 1 open bug)

Details

Attachments

(1 file, 3 obsolete files)

I have a prototype based on TestResolver.
Attached patch wip / prototype (obsolete) — Splinter Review
$ ./mach test-report netwerk
Test manifest: /home/gbrown/src/netwerk/test/reftest/reftest.list
  4 tests, 0 skipped on one or more platforms, 0 skipped on all platforms, 0 failed on one or more platforms
Test manifest: /home/gbrown/src/netwerk/test/browser/browser.ini
  6 tests, 2 skipped on one or more platforms, 0 skipped on all platforms, 0 failed on one or more platforms
Test manifest: /home/gbrown/src/netwerk/test/unit/xpcshell.ini
  308 tests, 19 skipped on one or more platforms, 2 skipped on all platforms, 3 failed on one or more platforms
Test manifest: /home/gbrown/src/netwerk/test/mochitests/mochitest.ini
  16 tests, 0 skipped on one or more platforms, 0 skipped on all platforms, 0 failed on one or more platforms
Test manifest: /home/gbrown/src/netwerk/test/httpserver/test/xpcshell.ini
  29 tests, 1 skipped on one or more platforms, 0 skipped on all platforms, 0 failed on one or more platforms
Test manifest: /home/gbrown/src/netwerk/test/crashtests/crashtests.list
  4 tests, 0 skipped on one or more platforms, 0 skipped on all platforms, 0 failed on one or more platforms
Test manifest: /home/gbrown/src/netwerk/cookie/test/unit/xpcshell.ini
  7 tests, 0 skipped on one or more platforms, 0 skipped on all platforms, 0 failed on one or more platforms
Test manifest: /home/gbrown/src/netwerk/test/unit_ipc/xpcshell.ini
  43 tests, 43 skipped on one or more platforms, 0 skipped on all platforms, 0 failed on one or more platforms
Test manifest: /home/gbrown/src/netwerk/cookie/test/browser/browser.ini
  1 tests, 0 skipped on one or more platforms, 0 skipped on all platforms, 0 failed on one or more platforms
Total: 418 tests, 65 skipped on one or more platforms, 2 skipped on all platforms, 3 failed on one or more platforms


Known limitations:
 - doesn't understand reftest annotations
 - doesn't count reftests "properly"

Also, it would be nice to generate a report for a bugzilla component (list of components for an owner); I don't know how to do that.
ideally I would like to have a dashboard that does this- in fact davehunt has a dashboard that queries activedata and generates some related information:
http://fxtestr.us-west-2.elasticbeanstalk.com/unittest?branch=autoland&path=netwerk&since=today-1week

It might have different limitations based on the data/availability.

Overall this would be useful- especially if you could query further:
./mach test-report netwerk --verbose
<lists the tests and related skipped configs>

./mach test-report netwerk --verbose --suite=xpcshell
<only lists xpcshell related tests>


the 4 different data points here are interesting:
Total: 418 tests, 65 skipped on one or more platforms, 2 skipped on all platforms, 3 failed on one or more platforms

I am curious what the failed status is?  is that a different item in a manifest?
(In reply to Joel Maher ( :jmaher) (UTC-5) from comment #2)

Thanks for the feedback -- much appreciated.

> I am curious what the failed status is?  is that a different item in a
> manifest?

This is the difference between skip-if and fail-if. With proper support for reftests, it would get more complicated: fuzzy-if, random-if, etc...not sure how important all these distinctions are.
Attached patch wip / prototype (obsolete) — Splinter Review
This version support --component:

$ ./mach test-report --component Core::Layout
Test manifest: /home/gbrown/src/layout/reftests/backgrounds/reftest.list
  216 tests, 0 skipped on one or more platforms, 0 skipped on all platforms, 0 failed on one or more platforms
Test manifest: /home/gbrown/src/layout/reftests/css-blending/reftest.list
  120 tests, 0 skipped on one or more platforms, 0 skipped on all platforms, 0 failed on one or more platforms
Test manifest: /home/gbrown/src/layout/reftests/border-dotted/reftest.list
  12 tests, 0 skipped on one or more platforms, 0 skipped on all platforms, 0 failed on one or more platforms
...

The implementation is naive / brute-force-ish and performance is not great, but not too bad either: a few seconds for component lookup on the top src dir, in my experience.
Attachment #8913825 - Attachment is obsolete: true
this seems useful to maybe include in |./mach test-info| ?  I would like to get these stats easier, and applying a patch to get them is not ideal :)
Attached patch wip / prototype (obsolete) — Splinter Review
Now provided as a subcommand of test-info:

  mach test-info report [--component product::component] [--verbose]

I'm not comfortable checking this in until I add proper reftest manifest support. Sorry.
Attachment #8914947 - Attachment is obsolete: true
Priority: -- → P3
Assignee: gbrown → nobody
Assignee: nobody → gbrown
See Also: → 1355568, 1352495
Attachment #8916198 - Attachment is obsolete: true
See Also: → 1568638, 1519171, 1548192
Blocks: 1568638
See Also: 1568638
Attachment #9080374 - Attachment description: Bug 1404472 - Add mach subcommand for per-component test-info report; r= → Bug 1404472 - Add mach subcommand for per-component test-info report; r=jmaher
Pushed by gbrown@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/d76bbec5d855
Add mach subcommand for per-component test-info report; r=jmaher
Status: NEW → RESOLVED
Closed: 3 months ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla70
Blocks: 1577302
You need to log in before you can comment on or make changes to this bug.