1053207 - Mach formatter has forgotten how to count

Status: RESOLVED FIXED

(Testing :: Mozbase, defect)

Description

[James Graham [:jgraham]]

It thinks that 0 subtests passed, always.

Comment 1

[James Graham [:jgraham]]

Attachment: Count number of passing subtests in mach formatter

Comment 2

[Chris Manchester (limited bugmail, email directly)]

Comment on attachment 8472341 [details] [diff] [review]
Count number of passing subtests in mach formatter

Review of attachment 8472341 [details] [diff] [review]:
-----------------------------------------------------------------

::: testing/mozbase/mozlog/mozlog/structured/formatters/machformatter.py
@@ +221,5 @@
>                                                             data["expected"],
>                                                             data.get("message", "")))
> +        if data["status"] == "PASS":
> +            self.status_buffer[test]["pass"] += 1
> +

It's a little confusing that pass + len(unexpected) == count except if we have expected failures/timeouts/etc. or unexpected passes. I can imagine looking at the summary line in test end and wondering why they don't add up.

Comment 3

[James Graham [:jgraham]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/a87da9366285

Agreed that the summary can be confusing if you don't realise it's trying to tell you two different things: how you did on the tests in an absolute sense (the number that passed) and how you did relative to expectations (the number of unexpecteds). I'm not sure if we can present those two things in a clearer way or if one of them isn't that useful.

Comment 4

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/a87da9366285