test cases should be randomized so users can cover all of them



Other Applications
QA Companion
11 years ago
5 years ago


(Reporter: Damian Szczepanik, Unassigned)


Firefox Tracking Flags

(Not tracked)




11 years ago
Currently scenario for testing in litmus as well as in QA Community Extension is like this:
- user selects group and subgroup
- user defines which test case he wishes to verify

this means that some tests which are difficult to verify or require external resources (server, second machine) are not covered because user is not able to check them

my idea is to randomize test cases so users don't choose test cases which they want to verify but they are asked to verify test case which was choosen by system

there are some benefits of this feature:
- more tests are covered (verified)
- tests which are skiped often should be improved (probably users are not able to verify them, they don't unserstand what they are asked to do and so on), system can count them and report to qa
- if first step is to "restart and log in" user which verifies 3 similar tests may skip this step (I'm log in, I don't need to log in again!) which means that he does not really verify all steps
- testing different area makes testing less boring and helps finding new defects

Comment 1

11 years ago
Based on our conversation on IRC, I think this would operate very well as an optional mode: maybe a button in the select-tests dialog would let the user select a random set of testcases instead of choosing testrun --> testgroup --> subgroup. 

It'd be best if there was some litmus-side support for this, since it's not easy to determine which testcase_ids belong to which platform/branches, and finding out is probably kind of expensive. Since litmus updates tend to move slower than extension updates, I'll probably do some client-side hack first if I have time.

I do want to say that mixing up the testcases sounds like a great idea for improving coverage and keeping users interested. Who wants to do four migrations in a row?


11 years ago
Severity: normal → enhancement
Component: QA Community Extension → QA Community Extension
Product: Webtools → Other Applications
Version: Trunk → unspecified


10 years ago
Priority: -- → P2


9 years ago
Assignee: zach → nobody

Comment 2

5 years ago
Litmus has been replaced by MozTrap. 

If we want an extension to integrate with MozTrap, it would be better to start from scratch with a new extension rather than trying to shoehorn that functionality into the >5 year old QAC.
Last Resolved: 5 years ago
Resolution: --- → WONTFIX
You need to log in before you can comment on or make changes to this bug.