662706 - Bulk compatibility test results view

Status: VERIFIED FIXED

(addons.mozilla.org Graveyard :: Compatibility Tools, defect, P3)

Description

[Justin Scott [:fligtar]]

Please provide a link with each validation job to view a table showing the compatibility tests and the number of hits for each. Right now we have to download the email log and open each results page. This will greatly improve QAing as well as being able to report back on the biggest compatibility issues.

The top should have a summary for all tests, followed by each add-on and the problems encountered.

Mock:

|-----------------------------------------------------------------------------|
| Test                          | # add-ons failing | total failures | passes |
|-----------------------------------------------------------------------------|
| Test 1 (error)                |                45 |             87 |   3422 |
| Test 2 (error)                |                45 |             87 |   3422 |
| Test 3 (warning)              |                45 |             87 |   3422 |
|-----------------------------------------------------------------------------|

Add-ons not bumped:
* Password Exporter (currently 4.0.*)
** Failed Test 1 (error) 32 times
** Failed Test 2 (error) 1 time

Add-ons bumped:
* ThreadBubble (now 6.*)
** Failed Test 3 (warning) 1 time

Comment 1

[Kumar McMillan [:kumar]]

Fixed https://github.com/jbalogh/zamboni/commit/ac67187

I just did the first part, a tally of tests per addons.  Now when a validation job finishes you'll get a link to a CSV for errors/warnings. I know CSVs are annoying but it's the easiest to implement. Could be a table view later.

I ran out of time to do the second part of this bug so feel to file another bug.  That is, there is no tally of what tests ran for *each* addon.

Also note that the errors/warnings CSV does not show passing tests because we currently do not track that it in the validator; we only track failures.

Comment 2

[krupa raj[:krupa]]

Validation tally file has the following info:

message_id: testcases_l10ncompleteness._aggregate_results.unexpected_encodings

message: Unexpected encodings in locale files

long_message: Localization files were encountered that used encodings that are not characteristic of those types of files.; Detected files:
locale/de-DE/goen.dtd
 (expected: ASCII, UTF_8); Localization files with the wrong encoding can cause issues with locales that include non-ASCII characters.

type: warning

addons_affected: 30

I filed bug 670001 for the features which were skipped.

Comment 3

[krupa raj[:krupa]]

Attachment: post-fix screenshot