1945662 - test_profilerMarkers.js and test_UsageReporting.js are permafailing cross-platform for artifact builds

Status: RESOLVED FIXED

(Toolkit :: Telemetry, task)

Description

[:Gijs (he/him)]

toolkit/components/glean/tests/xpcshell/test_ProfilerMarkers.js:

[task 2025-01-16T17:51:17.941Z] 17:51:17  WARNING -  TEST-UNEXPECTED-FAIL | toolkit/components/glean/tests/xpcshell/test_ProfilerMarkers.js | test_fog_counter_markers - [test_fog_counter_markers : 86] [{"type":"IntLikeMetric","id":"id not found","val":31}] deepEqual [{"type":"IntLikeMetric","id":"testOnly.badCode","val":31}]

and toolkit/components/telemetry/tests/unit/test_UsageReporting.js:

TEST-UNEXPECTED-FAIL | toolkit/components/glean/tests/xpcshell/test_ProfilerMarkers.js | test_fog_counter_markers - [test_fog_counter_markers : 116] [] deepEqual [{"type":"IntLikeMetric","id":"testOnly.badCode","val":31},{"type":"IntLikeMetric","id":"testOnly.mabelsKitchenCounters","label":"near_the_sink","val":1},{"type":"IntLikeMetric","id":"testOnly.mabelsKitchenCounters","label":"with_junk_on_them","val":2},{"type":"IntLikeMetric","id":"te

Sample push

Sample log

Given it's a permafailure it probably reproduces locally with an artifact build (and the strategy described in bug 1775795 comment 110 may be viable here, too).

Comment 1

[Chris H-C :chutten]

I think this is expected because the profiler markers can't figure out metric names in artifact builds. :aabh, can you confirm?

Comment 2

[Adam Brouwers-Harries [:aabh] [he/him] ⌚GMT]

:chutten - yes, I think this is probably an issue with how we're looking up metric names. I wasn't aware of the limitations around artefact builds and turning static metrics into JoG metrics, though I'm surprised that this hasn't been failing before. I think the easiest (read, quickest) fix right now is to disable all the profiler marker tests when we're running an artefact build (a-la ride-along pings), and then see if the new metric-lookup capabilities will fix it.

Comment 3

[Adam Brouwers-Harries [:aabh] [he/him] ⌚GMT]

Quick update: I can reproduce this with a local artefact build.

Comment 4

[:Gijs (he/him)]

FWIW I copy-paste failed in comment 0, the UsageReporting failure looks like:

[task 2025-02-03T15:12:37.414Z] 15:12:37  WARNING -  TEST-UNEXPECTED-FAIL | toolkit/components/telemetry/tests/unit/test_UsageReporting.js | test_prefs - [test_prefs : 31] "decafdec-afde-cafd-ecaf-decafdecafde" == null

which is from here, which is maybe because of the setEnabled stuff right above it or something, which I think Jan-Erik may be fixing in bug 1945663?

Anyway, per the previous comments I'll put up a patch to disable both tests in artifact builds and you can let me know if that is not the right thing because I accidentally misled you (sorry!).

Comment 5

[:Gijs (he/him)]

Attachment: Bug 1945662 - skip failing glean tests in artifact builds, r?chutten,aabh

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/11c48d4259bc
skip failing glean tests in artifact builds, r=aabh

Comment 7

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/11c48d4259bc