Create improved subsuite HTML report

RESOLVED FIXED

Status

Firefox OS
Certification Suite
RESOLVED FIXED
4 years ago
4 years ago

People

(Reporter: jgraham, Unassigned)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments)

This can use the original HTML report as a template, but has at least one additional requirement; results that are better than expected should not be flagged up.
Created attachment 8447998 [details]
report.html

So I made a start at an improved report. Attached is a very basic sample. Feedback welcome before I spend too much time on this.
Much better, thanks!

I like the practice of prefacing unexpected failures with UNEXPECTED_.  I know it's redundant since we include the expected outcome, but I worry about partners (particularly those not super fluent in English) not understanding the difference between:

expected: FAIL, result: FAIL

and 

expected: PASS, result: FAIL

Using "UNEXPECTED_" gives them something more explicit to look for.
Well this won't ever say expected: FAIL result: FAIL, or even expected: FAIL result: PASS. So I will certainly experiment with adding that prefix, but I sort of suspect it will just make the page more noisy.
(In reply to James Graham [:jgraham] from comment #3)
> Well this won't ever say expected: FAIL result: FAIL, or even expected: FAIL
> result: PASS. So I will certainly experiment with adding that prefix, but I
> sort of suspect it will just make the page more noisy.

Oh we don't have expected failures or unexpected successes?  In that case, it's probably OK to leave it as-is.
Right, the idea was to not report expected failures or unexpected successes in order to concentrate on the issues most likely to be real cert-affecting problems.
In that case, it looks great!  Thanks.
Created attachment 8450450 [details] [review]
Updated report format
Attachment #8450450 - Flags: review?(jgriffin)
Btw, the look of the new report is a big improvement, IMO.
Comment on attachment 8450450 [details] [review]
Updated report format

I see ato has reviewed this in critic, so I'll give my r+ for the general look and format of the report.
Attachment #8450450 - Flags: review?(jgriffin) → review+
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.