359366 - Testing coverage stats should be presented at the OS level rather than platform

Status: RESOLVED FIXED

(Webtools Graveyard :: Litmus, defect)

Description

[Carsten Book [:Tomcat]]

-> Do and complete a BFT/Smoketest with a Windows XP

-> Go to Litmus again with the same Build and do test with Windows2000/ME/Vista (example: choose OS: Vista) on the Systeminformation Page

When you see the select group Testing overview you see on Personal Testing Coverage your results from XP.

Could be a Problem when people using multiple Windows OS Systems like XP as Host and Vista on a VM.

Comment 1

[Chris Cooper [:coop] (he/him)]

(In reply to comment #0)
> Could be a Problem when people using multiple Windows OS Systems like XP as
> Host and Vista on a VM.

We're not going to track whether the tester is running in a VM or not. That's not common enough in the community-at-large to make it worthwhile.

The coverage stats we do are consciously presented at the Platform (e.g. Windows) rather than the OS (e.g. XP) level. Historically this was done so that Mozilla testers could quickly triage their efforts. For example, if we already have some coverage on Windows (any OS), it is a better use of their time to test on another platform, given that most behavior will be the same between OS versions on the same platform.

If that's not still the case, I can change the coverage stats to report down to the OS level. What do people think?

Comment 2

[Zach Lipton [:zach]]

I think keeping the coverage stats at the platform level makes more sense. Unless the QA team is going to want to run all testcases on all OSes, it makes sense to keep the stats listed by platform. If this functionality would be useful, we could add a matrix view of test results by OS so we can get a more detailed view of coverage across all OSes. 

Comment 3

[Tracy Walker [:tracy]]

Unfortunately, there may be enough OS specific bugs to warrant this be at the OS level.  If we're tracking community coverage, our hopes would be that we're getting sufficient cross OS testing.  But the importance of testing Win 2k compared to Win XP has been diminishing. While the importance of testing Vista compared to Win XP will be increasing in the future. The same can be said for the various cats on Mac.  The number of Linux distros make it unreasonable to attempt to track.

While I think it's fine to keep reports at a platform level.  It's important we keep focused on the current primary OS without losing site of the fading OS's.

Comment 4

[Chris Cooper [:coop] (he/him)]

With test runs, coverage statistics are presented for the current configuration (down to the OS level) being tested, or for each set of criteria (including OS) individually in the test run report.