Closed Bug 996183 Opened 8 years ago Closed 7 years ago

write a tool which reports the total number of tests we have and which are skipped/disabled for all configurations

Categories

(Testing :: General, defect)

defect
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: jmaher, Assigned: ahal)

References

(Blocks 1 open bug)

Details

(Keywords: ateam-unittests-big)

We have so many disabled tests on various platforms and configurations.  Right now, we just dream of green runs when we land a patch.  This means the focus is not on complete test coverage, instead it is on running what looks good.

Knowing that, it is important for us to track the tests we have and what their expected state is.

While I don't expect this to be a simple this, this shouldn't be a 6 month project.

I envision something like this:
hg clone m-c
build
for config in ['linux', 'linux&debug', 'e10s', 'linux&e10s', 'b2g', ...]:
  tests = parseManifest(mozinfo=config)
  sqlinsert(config, tests.total, tests.active)


The exact coding for this could be much more complicated or at least more complete.

We could have a json api to query the stats, likewise a simple graph/webui.

Please comment in here to outline what details we need to account for.
Adding Armen, since he may help out with the implementation after this is scoped well enough.
Blocks: 996504
The skipped tests are something I'd like to capture as part of the metadata emitted by the build system (currently all-tests.json). I would eventually like to see --run-skipped (or similar) options on the mach testing commands to allow people to force run certain tests without having to update manifests.

We kinda/sorta get this for free if skipped tests aren't ignored by the build system. I think we only do that for xpcshell now, but I saw a bug to do that for mochitests too.
Depends on: 989583
Please also check bug 1029348 comment 6
Assignee: nobody → ahalberstadt
Depends on: 1068847
Duplicate of this bug: 1066279
Depends on: 1066735, 1066274
I'm working on this here:
https://github.com/ahal/test-informant
Here's an early version of what the report will look like:
http://people.mozilla.org/~ahalberstadt/report.html

Note that the data that populated it was from inbound, and the urls are hardcoded to mozilla-central.. but you get the idea. I'll also be adding more suites and platforms in the near future. For now, any feedback on what kind of data you'd like to see would be appreciated.
This looks great!  I think this is good for a rev 1.  Making the div wider for the tests might be nice, so that they don't wrap.
Yeah, good point. I guess I've been testing on my high-res desktop monitor. I updated report at the same url to be wider, made it scale a bit better with smaller resolutions and added some new mochitest flavours. The percentages for browser-chrome and devtools are skewed, that's a known issue.
Depends on: 1071642
Depends on: 1073592
Depends on: 1085389
Note: this is more or less done. I'll resolve it once the process of generating and posting reports is fully automated. This is blocked on some IT bugs.
Status: NEW → RESOLVED
Closed: 7 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.