650979 - Results: Implement list of Test Suites

Status: VERIFIED WONTFIX

(Mozilla QA Graveyard :: MozTrap, enhancement, P1)

Description

[Cameron Dawson [:camd]]

This list should be filterable with unified filtering interface
Paginated
Each line should show some summary data without expanding, such as:
- % complete
- Summary results values for that run (pass / fail / invalid / pending)

Expansion shows:

When expanded, it should show:
- link to all the contained testcases
- explore possibility of showing testcases that are part of the latest version of the Test Suite, but NOT included in this test run / cycle.
- team members
- environments

Comment 2

[Cameron Dawson [:camd]]

A Pivotal Tracker story has been created for this Bug: http://www.pivotaltracker.com/story/show/12751051

Comment 3

[Cameron Dawson [:camd]]

Cameron Dawson changed story state to unstarted in Pivotal Tracker

Comment 4

[Cameron Dawson [:camd]]

Eric Meyer changed story state to started in Pivotal Tracker

Comment 5

[Cameron Dawson [:camd]]

Eric Meyer added a comment in Pivotal Tracker:   
   
"explore possibility of showing testcases that are part of the latest version of the Test Suite, but NOT included in this test run / cycle"

I could use some detail on that. Are we just showing titles? How long do we expect the list to be? How would you phrase a title for this clearly and succinctly? Do we want to do something with them (add?), or are we just displaying them? What's the use-case for needing these?

Comment 6

[Cameron Dawson [:camd]]

Eric Meyer added a comment in Pivotal Tracker:   
   
delivering a first draft without the optional item mentioned above.

Comment 7

[Cameron Dawson [:camd]]

Carl Meyer added a comment in Pivotal Tracker:   
   
deployed what's there so far, but I don't think this wireframe reflects the additional complexity we discussed the other day on skype (of course, the description here doesn't either :-). Each row in this list actually represents a test-suite-as-included-in-a-test-run. If the same test suite is included in multiple test runs, it should appear by default collapsed into a single "generic" test suite row, expandable to see a row for each as-included-in-a-test-run, with results only shown on the latter expanded rows. Bumping this back to eric to add wireframe showing how each case would look (a test suite that's only included in one test run, thus all the data can appear on one row, vs collapsed suite included in multiple runs).

Comment 8

[Cameron Dawson [:camd]]

Cameron Dawson added a comment in Pivotal Tracker:   
   
I'm wondering if the "[multiple]" text should be in the "run" column, rather than right next to the name.

Comment 9

[Cameron Dawson [:camd]]

Cameron Dawson added a comment in Pivotal Tracker:   
   
Carl: so, as an implementation point, if filtering is on a particular testrun, then the [multiple] thing wouldn't show, right?  It would then just be a single row for the test suite?

Comment 10

[Cameron Dawson [:camd]]

Carl Meyer added a comment in Pivotal Tracker:   
   
Cam: on your first point, great idea. Checking with Eric to see if that's doable, but I don't see why not. On your question: yes, the only reason you'd ever see the whole multiple-runs combined business is if you aren't filtering your view on a particular test run.

Comment 11

[Cameron Dawson [:camd]]

Cameron Dawson added a comment in Pivotal Tracker:   
   
If do-able, that'd be great.  But regardless, it looks great to me!  Make it so!

Comment 12

[Cameron Dawson [:camd]]

Eric Meyer added a comment in Pivotal Tracker:   
   
doable. I'll push the change.

Comment 13

[Cameron Dawson [:camd]]

Carl Meyer added a comment in Pivotal Tracker:   
   
We also don't have summary-results-reporting in the platform for test suites. This I'd requested as part of https://bugzilla.mozilla.org/show_bug.cgi?id=636566 but I didn't notice that it wasn't included in the fix when that platform bug was marked fixed.

In filing platform bugs for this and/or summary reporting for included-test-cases, might be good to be aware that apparently uTest's position is that "all statistical reporting requirements should be addressed separately" - see https://bugzilla.mozilla.org/show_bug.cgi?id=636566#c1

If we can't get these summary reporting calls in the platform, I can fake it by querying all results and counting; not sure yet how slow this would be, but caching may make it feasible.

Comment 14

[Cameron Dawson [:camd]]

Carl Meyer added a comment in Pivotal Tracker:   
   
Just noting that I've bumped this story up to 3 points' complexity, because the platform has no inherent concept of an "included-test-suite" - the only way I can implement this story is to query all included test cases and then group them by suite for display as a list of suites. Doable, but adds additional complexity, especially in terms of getting the filtering UI working here.

Comment 15

[Cameron Dawson [:camd]]

Carl Meyer deleted the linked story in Pivotal Tracker

Comment 16

[Cameron Dawson [:camd]]

We decided to handle this differently.  We won't be providing a main list for Test Suites.  You can access this by filtering the list of Test Cases by the Test Suite.  This avoids requirements from the Platform that we're not likely to get, and helps simplify the UI.

Comment 17

[Matt Brandt [:mbrandt]]

Due to age of bug and comment 16, bumping to verified wontfix