636764 - Differentiate between exceptions and assertions for l10n tests

Status: RESOLVED FIXED

(Mozilla QA Graveyard :: Mozmill Result Dashboard, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

For l10n tests we have to differentiate between JUM assertions and exceptions. Otherwise some important information is not shown.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Patch v1

Only for l10n tests for now because no other testrun is using the JUM lib.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

This needs Mozmill 1.5.2. So adding dependency.

Comment 3

[u279076]

Comment on attachment 515108 [details] [diff] [review]
Patch v1

>-                failures.push(result.fails[j].comment);

>+                else
>+                  failures.push("unknown failure");

In the case of !exception && !assertion, I'd still like to see the failure.  "unknown failure" is not valuable.  Can you simply use result.fails[j].comment for the "unknown fail" case?

With that change, r+.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to comment #3)
> In the case of !exception && !assertion, I'd still like to see the failure. 
> "unknown failure" is not valuable.  Can you simply use result.fails[j].comment
> for the "unknown fail" case?

We do not have a result.fails[j].comment field anymore with the changes on bug 632403. We can really have only those two different types of failure. So there is really nothing I could change.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Anthony, any further objections? Otherwise I would like to land it.

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Due to no other objections I have pushed the change to all dashboard instances and landed the patch in the repository.

https://github.com/whimboo/mozmill-dashboard/commit/26436c6b2d435046cb36d14bb872e3866a6e762b