1030766 - Create improved subsuite HTML report

Status: RESOLVED FIXED

(Firefox OS Graveyard :: Certification Suite, defect)

Description

[James Graham [:jgraham]]

This can use the original HTML report as a template, but has at least one additional requirement; results that are better than expected should not be flagged up.

Comment 1

[James Graham [:jgraham]]

Attachment: report.html

So I made a start at an improved report. Attached is a very basic sample. Feedback welcome before I spend too much time on this.

Comment 2

[Jonathan Griffin (:jgriffin)]

Much better, thanks!

I like the practice of prefacing unexpected failures with UNEXPECTED_.  I know it's redundant since we include the expected outcome, but I worry about partners (particularly those not super fluent in English) not understanding the difference between:

expected: FAIL, result: FAIL

and 

expected: PASS, result: FAIL

Using "UNEXPECTED_" gives them something more explicit to look for.

Comment 3

[James Graham [:jgraham]]

Well this won't ever say expected: FAIL result: FAIL, or even expected: FAIL result: PASS. So I will certainly experiment with adding that prefix, but I sort of suspect it will just make the page more noisy.

Comment 4

[Jonathan Griffin (:jgriffin)]

(In reply to James Graham [:jgraham] from comment #3)
> Well this won't ever say expected: FAIL result: FAIL, or even expected: FAIL
> result: PASS. So I will certainly experiment with adding that prefix, but I
> sort of suspect it will just make the page more noisy.

Oh we don't have expected failures or unexpected successes?  In that case, it's probably OK to leave it as-is.

Comment 5

[James Graham [:jgraham]]

Right, the idea was to not report expected failures or unexpected successes in order to concentrate on the issues most likely to be real cert-affecting problems.

Comment 6

[Jonathan Griffin (:jgriffin)]

In that case, it looks great!  Thanks.

Comment 7

[James Graham [:jgraham]]

Attachment: Updated report format

Comment 8

[Jonathan Griffin (:jgriffin)]

Btw, the look of the new report is a big improvement, IMO.

Comment 9

[Jonathan Griffin (:jgriffin)]

Comment on attachment 8450450 [details] [review]
Updated report format

I see ato has reviewed this in critic, so I'll give my r+ for the general look and format of the report.