Closed Bug 658706 Opened 14 years ago Closed 13 years ago

Summary results for included-test-cases and included-test-suites

Categories

(Mozilla QA :: Case Conductor-Platform, enhancement)

Product:
Mozilla QA
Component:
Case Conductor-Platform
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: carljm, Unassigned)

References

Details

UI needs to display summary result numbers for included test cases, and for included test suites (i.e. results for test cases from a given test suite within a given test run; the platform has no direct concept of an included-test-suite, but this data is still possible to retrieve). Currently UI implements this by searching for relevant TestResults by testSuiteId and testCaseId and counting the status values, but experiences on the staging server already demonstrate that with environment variations and multiple testers the number of results, even for a single included-test-case, rapidly grows to a level where this approach is not feasible, even with caching (the platform response is too slow and the network transfer too big).
Blocks: 658708
Severity: normal → enhancement
Target Milestone: --- → 1.0
A Pivotal Tracker story has been created for this Bug: http://www.pivotaltracker.com/story/show/17765745
Vadim Kisen added a comment in Pivotal Tracker: I consolidated coverage reporting under /testcycles/ resource path... Now to get coverage totals for test cycle just call the URL below without passing any additional Form parameters. To get results for Test Run - pass testRunId, for Test Suite - testSuiteId .../testcycles/{id}/reports/coverage/resultstatus/ <param name="testSuiteId" style="query" type="xs:int"/> <param name="testRunId" style="query" type="xs:int"/>
Vadim Kisen changed story state to delivered in Pivotal Tracker
Cameron Dawson added a comment in Pivotal Tracker: Vadim: how do we get summary for test cases? This will be important since there will often be multiple environment specific results for test cases
Vadim Kisen added a comment in Pivotal Tracker: OK, I can add another optional parameter to the call testCaseId
Carl Meyer added a comment in Pivotal Tracker: Vadim: it should probably be either testCaseVersionId or includedTestCaseId; aggregated results over multiple versions of a testcase are probably not meaningful. I'm not sure if its useful to aggregate results for a test case over multiple test runs, either; if not, it should be includedTestCaseId. Right now our View Results > Test Cases page actually lists includedTestCases because we had earlier concluded that seeing results per-test-run was most useful. So that would suggest includedTestCaseId. Cam, what's your thought here?
Vadim Kisen added a comment in Pivotal Tracker: Since we don't allowing adding multiple versions of the same test case into a test suite or test run, it would not make a difference while running at a test run level. But if you want to aggregate across multiple test runs withing a test cycle, you can supply test case version id...
Carl Meyer added a comment in Pivotal Tracker: Sure, having the option to supply testCaseId to aggregate across runs would be fine, as long as we also can supply includedTestCaseId to get results for a single case within just the one run.
Vadim Kisen added a comment in Pivotal Tracker: Carl, as my comment above stated, we.do not have multiple versions of the same test case within a single test run, so providing test case id would give you the same effect . Even more, test results don't have associations to Included test case id, only to test case id and test case version id, so searching on included id will not be possible...
Carl Meyer added a comment in Pivotal Tracker: I don't think it is the same effect. Versions are not relevant, the issue is test runs - how can we get aggregated results for a test case within only a single test run? If we provide testCaseId, it has no way of knowing which run we want results for.
Carl Meyer added a comment in Pivotal Tracker: Or can we provide multiple parameters at once, e.g. testCaseId and testRunId?
Vadim Kisen added a comment in Pivotal Tracker: no, API gives you testRunId, testSuiteId and testCaseId. You can supply all 3 of them or any number...
Carl Meyer added a comment in Pivotal Tracker: Ah, ok, if we can combine that's perfect. Guess that's kind of obvious, sorry :-)
Vadim Kisen added a comment in Pivotal Tracker: OK, so I'll add testCaseId then..It'll give users the ability to search for all versions of the same test case across all test runs for a test cycle, in case they want to....
Carl Meyer added a comment in Pivotal Tracker: That's great, thanks!
Vadim Kisen added a comment in Pivotal Tracker: Just pushed the changes..please pull...
Carl Meyer added a comment in Pivotal Tracker: This doesn't seem to be working. When I call the endpoint with no parameters but for various different test cycles, I get the same results back for all queries, even though some of the test cycles don't even have a single test run, so clearly have no results (in other words, it doesn't seem to be filtering on the test cycle ID from the URL). And when I pass testRunId=X, I get no data, even though I know that test run does have results i the system. Haven't tried with testCaseId or testSuiteId yet.
Carl Meyer changed story state to rejected in Pivotal Tracker
Vadim Kisen added a comment in Pivotal Tracker: That's strange...I tested the query fine.... When passing testRunId you have to make sure you're using the parent testcycle id at the endpoint. Otherwise it will not find anything....
Carl Meyer added a comment in Pivotal Tracker: Yes, I'm definitely passing a testRunId that is part of the test cycle for the endpoint I'm calling - still getting no results. Also, worth pointing out that the first bug I mentioned wouldn't show up unless you have more than one test cycle in the system and check with both.
Vadim Kisen changed story state to delivered in Pivotal Tracker
Vadim Kisen changed story state to started in Pivotal Tracker
Vadim Kisen changed story state to finished in Pivotal Tracker
Vadim Kisen changed story state to delivered in Pivotal Tracker
Cameron Dawson added a comment in Pivotal Tracker: Vadim, this is working for me with test cycles with one caveat: it doesn't seem to be honoring when results are deleted. In my specific case, when I set results for tests, it leaves behind a "deleted" result in state of "pending" So when I ask for summary results, I see 2 passed, 1 failed, 1 invalid but ALSO 4 pending. I shouldn't see the pending ones, since they're deleted.
Cameron Dawson added a comment in Pivotal Tracker: just to be clear: I DO expect those deleted items to exist in state pending. I just don't expect them to be returned with the summary results.
Carl Meyer added a comment in Pivotal Tracker: Hmm, maybe I'm missing something, but I wouldn't expect those deleted results to be left behind at all. When a result's status is updated, I'd expect that same result object to have a changed state, not for the pending one to be deleted and a new one created in its place. Seems like something is fishy here, either in the platform or in the test.
Cameron Dawson added a comment in Pivotal Tracker: That's a good point. I'm not deleting anything. I'm just changing the state. yeah, I guess you're right. Those items shouldn't exist in deleted state at all. I should just be updating an existing row. Though, I think one of my original points is still true: if there ARE deleted results, they shouldn't show in the results summary.
Carl Meyer added a comment in Pivotal Tracker: Yes, that is certainly true.
Vadim Kisen added a comment in Pivotal Tracker: You're right..It was on my list to do...Will check in the fix in 10 minutes...
Vadim Kisen added a comment in Pivotal Tracker: done...please pull the latest...
Cameron Dawson added a comment in Pivotal Tracker: Great! thanks, Vadim. That works with test cycles for me now. Will test with Test Runs as soon as I get my tests fixed.
Cameron Dawson added a comment in Pivotal Tracker: testRunResults.csv is a dump of what my table looks like after "executing" these tests.
Cameron Dawson added a comment in Pivotal Tracker: Vadim: upon closer examination, it looks like when status gets updated for a result, that you delete the old result, and create a new one with a new verison. But this also bumps the testCaseVersionId, which changes the URL for that result object. This sounds broken
Cameron Dawson added a comment in Pivotal Tracker: OK, I dug further into this. Looks like there is a result object that gets created as soon as both a user and a suite are added to a testrun. But when the result is assigned to a user, then they get their own result, and the original one is deleted. However, once this assignment happens, the URL is consistent for status changes on that result. So while it was unexpected, it's harmless. So no worries...
Carl Meyer changed story state to accepted in Pivotal Tracker
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.