2040417 - Frequent permafails related to telemetry/glean that may be related to artifact builds

Status: NEW

(Toolkit :: Telemetry, defect, P4)

Description

[Alexandre Poirot [:ochameau]]

xpcshell test jobs are frequently orange since at least April.
It may be specific to artifact jobs only.
I blocked a few bugs that may relate to this class of failures.

Beyond polluting try results, it seems to require frontend developer attention to workaround some issues specific to artifact builds.
Like bug 2027887.

Two active failures are bug 2040084 and bug 2036960.

• Event should have been recorded and ping submitted https://treeherder.mozilla.org/logviewer?job_id=567040784&repo=try&task=aI69KwI_TlydvnERM8tS8Q.0
• Ping did not submit immediately https://treeherder.mozilla.org/logviewer?job_id=567124791&repo=try&task=Uim4cwNMSOy-Yxf-LEa0pg.0

Is this a known issue? Do not hesitate to dupe this bug.
If not, is there some systemic tweak to be done to improve try results, while reducing pressure on frontend devs?

Comment 1

[Chris H-C :chutten]

Per https://firefox-source-docs.mozilla.org/toolkit/components/glean/user/instrumentation_tests.html#tests-and-artifact-builds there are two situations that are known to cause problems in testing on artifact builds:

• You are changing a metric or ping that is used in instrumentation in the compiled code, or
• You are using testSubmission or testBeforeNextSubmit in JavaScript for a ping submitted in the compiled code.

#1 is things like "I changed the acceptable event extras, but am still getting the old stuff from recordings in C++" or "I adjusted the time_unit of my timing_distribution, but am getting higher-resolution samples from Rust". #2 is things like "GleanPings.myPing.testSubmission will not be called by C++'s mozilla::glean_pings::MyPing.Submit()".

If these tests are of this sort of shape, they can be skipped whenever FOG's artifact build support is enabled (instructions in the docs)

I do have an idea for fixing #2, which I will file shortly and add as a See Also here.

Comment 2

[Chris H-C :chutten]

I'll double-check that these are only due to known causes.

Comment 3

[BugBot [:suhaib / :marco]]

The severity field is not set for this bug.
:chutten, could you have a look please?

For more information, please visit BugBot documentation.

Comment 4

[Chris H-C :chutten]

Confirmed that test_dcb_bad_magic_header_telemetry is testing the "content-decoding-error" ping which is submitted in C++ and test_new_profile_ping_sent is testing the "new-profile" ping which is the same. Both are the second known case, and so both should be fixable after bug 1974314