Closed Bug 393467 Opened 17 years ago Closed 11 years ago

test cases should be randomized so users can cover all of them

Categories

(Other Applications Graveyard :: QA Companion, enhancement, P2)

x86
Windows XP
enhancement

Tracking

(Not tracked)

RESOLVED WONTFIX

People

(Reporter: damian.publicemail, Unassigned)

Details

Currently scenario for testing in litmus as well as in QA Community Extension is like this:
- user selects group and subgroup
- user defines which test case he wishes to verify

this means that some tests which are difficult to verify or require external resources (server, second machine) are not covered because user is not able to check them

my idea is to randomize test cases so users don't choose test cases which they want to verify but they are asked to verify test case which was choosen by system

there are some benefits of this feature:
- more tests are covered (verified)
- tests which are skiped often should be improved (probably users are not able to verify them, they don't unserstand what they are asked to do and so on), system can count them and report to qa
- if first step is to "restart and log in" user which verifies 3 similar tests may skip this step (I'm log in, I don't need to log in again!) which means that he does not really verify all steps
- testing different area makes testing less boring and helps finding new defects
Based on our conversation on IRC, I think this would operate very well as an optional mode: maybe a button in the select-tests dialog would let the user select a random set of testcases instead of choosing testrun --> testgroup --> subgroup. 

It'd be best if there was some litmus-side support for this, since it's not easy to determine which testcase_ids belong to which platform/branches, and finding out is probably kind of expensive. Since litmus updates tend to move slower than extension updates, I'll probably do some client-side hack first if I have time.

I do want to say that mixing up the testcases sounds like a great idea for improving coverage and keeping users interested. Who wants to do four migrations in a row?
Severity: normal → enhancement
Product: Webtools → Other Applications
Version: Trunk → unspecified
Priority: -- → P2
Assignee: zach → nobody
Litmus has been replaced by MozTrap. 

If we want an extension to integrate with MozTrap, it would be better to start from scratch with a new extension rather than trying to shoehorn that functionality into the >5 year old QAC.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → WONTFIX
Product: Other Applications → Other Applications Graveyard
You need to log in before you can comment on or make changes to this bug.