Testing coverage stats should be presented at the OS level rather than platform

RESOLVED FIXED

Status

Webtools Graveyard
Litmus
--
minor
RESOLVED FIXED
11 years ago
2 years ago

People

(Reporter: Tomcat, Assigned: coop)

Tracking

Details

(Reporter)

Description

11 years ago
-> Do and complete a BFT/Smoketest with a Windows XP

-> Go to Litmus again with the same Build and do test with Windows2000/ME/Vista (example: choose OS: Vista) on the Systeminformation Page

When you see the select group Testing overview you see on Personal Testing Coverage your results from XP.

Could be a Problem when people using multiple Windows OS Systems like XP as Host and Vista on a VM.
(Assignee)

Comment 1

11 years ago
(In reply to comment #0)
> Could be a Problem when people using multiple Windows OS Systems like XP as
> Host and Vista on a VM.

We're not going to track whether the tester is running in a VM or not. That's not common enough in the community-at-large to make it worthwhile.

The coverage stats we do are consciously presented at the Platform (e.g. Windows) rather than the OS (e.g. XP) level. Historically this was done so that Mozilla testers could quickly triage their efforts. For example, if we already have some coverage on Windows (any OS), it is a better use of their time to test on another platform, given that most behavior will be the same between OS versions on the same platform.

If that's not still the case, I can change the coverage stats to report down to the OS level. What do people think?

OS: Windows 2000 → All
Hardware: PC → All
Summary: Windows Versions are not handled separate for Testruns → Testing coverage stats should be presented at the OS level rather than platform

Comment 2

11 years ago
I think keeping the coverage stats at the platform level makes more sense. Unless the QA team is going to want to run all testcases on all OSes, it makes sense to keep the stats listed by platform. If this functionality would be useful, we could add a matrix view of test results by OS so we can get a more detailed view of coverage across all OSes. 

Comment 3

11 years ago
Unfortunately, there may be enough OS specific bugs to warrant this be at the OS level.  If we're tracking community coverage, our hopes would be that we're getting sufficient cross OS testing.  But the importance of testing Win 2k compared to Win XP has been diminishing. While the importance of testing Vista compared to Win XP will be increasing in the future. The same can be said for the various cats on Mac.  The number of Linux distros make it unreasonable to attempt to track.

While I think it's fine to keep reports at a platform level.  It's important we keep focused on the current primary OS without losing site of the fading OS's.
(Assignee)

Updated

11 years ago
Status: NEW → ASSIGNED
(Assignee)

Comment 4

11 years ago
With test runs, coverage statistics are presented for the current configuration (down to the OS level) being tested, or for each set of criteria (including OS) individually in the test run report.
Status: ASSIGNED → RESOLVED
Last Resolved: 11 years ago
Resolution: --- → FIXED
Product: Webtools → Webtools Graveyard
You need to log in before you can comment on or make changes to this bug.