1867822 - Drop maximum code coverages in Karma to (at most) 80%

Status: RESOLVED FIXED

(Firefox :: Messaging System, task, P2)

Description

[Dan Mosedale (:dmosedale, :dmose)]

Dropping the required code coverage thresholds a bit to 80% is considered a best practice.

The idea here is that having thresholds higher than that puts too much focus on writing tests that meet a very low-level criterion, at the expense of sufficient focus on what's really important for users of both the product and the code.

Kat, do you have any other thoughts on the different sub-thresholds (eg statements vs branches vs functions vs lines) and/or links to any docs about code coverage best practices that you think are particularly good?

Comment 1

[Katrina Anderson]

(In reply to Dan Mosedale (:dmosedale, :dmose) from comment #0)

Dropping the required code coverage thresholds a bit to 80% is considered a best practice.

The idea here is that having thresholds higher than that puts too much focus on writing tests that meet a very low-level criterion, at the expense of sufficient focus on what's really important for users of both the product and the code.

Kat, do you have any other thoughts on the different sub-thresholds (eg statements vs branches vs functions vs lines) and/or links to any docs about code coverage best practices that you think are particularly good?

The 80% rule generally applies to branch coverage (if it's available) in my experience. I will also add that 80% should cover critical paths, so a social contract between contributors will have to be established, where they use their best judgment to determine the ~20% of coverage which has diminishing returns.

Let me think on the docs suggestions and also poke my teammate Peter deHann on that too.

Comment 2

[Peter deHaan [:pdehaan]]

Not sure I have any specific guidance, apart from what :dmose and :kanderson already suggested. ~80% is a great target, since 100% either takes too long to be beneficial or just leads to weird tests just to hit specific lines without being great tests.

https://testing.googleblog.com/2020/08/code-coverage-best-practices.html is an interesting read:

In general code coverage of a lot of products is below the bar; we should aim at significantly improving code coverage across the board. Although there is no “ideal code coverage number,” at Google we offer the general guidelines of 60% as “acceptable”, 75% as “commendable” and 90% as “exemplary.” However we like to stay away from broad top-down mandates and encourage every team to select the value that makes sense for their business needs.

https://saucelabs.com/resources/blog/code-coverage-vs-test-coverage outlines differences between code coverage vs test coverage (but doesn’t really offer %% targets).
Or https://blog.testproject.io/2022/05/10/code-coverage-vs-test-coverage/… Most everything I’m finding when searching for industry coverage stats points to 80%

Therefore, it is alright to keep the number higher (80% maybe), but writing test cases just for achieving 100% code coverage should be avoided.

and

With that being said, it is also important to note that 100% code coverage should not be aimed in any phase of SDLC or release cycles. It would just waste the time and resources, and we may miss out on important testing parts when short on time.

TLDR: Yeah, 80% sounds like a good target, as long as it's the right 80% of code.

Comment 3

[Mimi Jung]

Attachment: Bug 1867822 - Drop maximum code coverages in Karma to (at most) 80%

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/edb2cfa7960c
Drop maximum code coverages in Karma to (at most) 80% r=omc-reviewers,dmose

Comment 5

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/edb2cfa7960c