1749591 - Test that auto-tags for FOG metrics conform to expectation

Status: NEW

(Toolkit :: Telemetry, task, P3)

Description

[Chris H-C :chutten]

bug 1746941 adds initial support for semi-automatic tagging for FOG metrics with the product and component the metric belongs to.

Since it's only semi-automatic we should write tests to guard against devs making accidental changes to the tree that cause the tags to go wonky.

Specifically:

• All metrics should have at least one tag
• (( Not needed: a test to check that the tags are valid. glean_parser will fail the build for us there, right? Double-check that. ))
• All metrics in a metrics.yaml should share 1 tag
• The tag that all metrics in a metrics.yaml share should be the same as what's reported by mach file-info bugzilla-component for that metrics.yaml
• That the tags list is up-to-date with additions/removals of products+components from the tree (ie, that mach update-glean-tags would result in 0 changes to the checked-in tags.yaml)

(( The first two will be subsumed by a file-level tag test eventually (see bug 1745283), but we can't do that yet ))

I propose the tests live in toolkit/components/glean/tests/pytest/ with the other pythonic tests in FOG.

Comment 1

[Chris H-C :chutten]

(In reply to Chris H-C :chutten from comment #0)

• (( Not needed: a test to check that the tags are valid. glean_parser will fail the build for us there, right? Double-check that. ))

Double-checked and confirmed:

 1:10.55 script.ParserError: glinter nits found during parse
 1:10.55 make[3]: *** [backend.mk:636: toolkit/components/glean/api/src/.deps/metrics.rs.stub] Error 1
 1:10.55 Sorry, Glean found some glinter nits:
 1:10.55 ERROR: INVALID_TAGS: fog.failed_idle_registration: Invalid tags specified in metric: BAD TAG
 1:10.55 ERROR: INVALID_TAGS: fog.initialization: Invalid tags specified in metric: BAD TAG
 1:10.55 ERROR: INVALID_TAGS: fog.ipc.buffer_sizes: Invalid tags specified in metric: BAD TAG
 1:10.55 ERROR: INVALID_TAGS: fog.ipc.flush_durations: Invalid tags specified in metric: BAD TAG
 1:10.55 ERROR: INVALID_TAGS: fog.ipc.flush_failures: Invalid tags specified in metric: BAD TAG
 1:10.55 ERROR: INVALID_TAGS: fog.ipc.replay_failures: Invalid tags specified in metric: BAD TAG