1531369 - Set telemetry-tests-client suite to tier 1

Status: RESOLVED FIXED

(Toolkit :: Telemetry, enhancement, P1)

Description

[Raphael Aurich [:raphael] UTC+02:00]

This will provide clarity for Firefox Telemetry engineers by automatically generating bugs for new intermittent failures and have breaking changes backed out by sheriffs.

Requirements for the tier 1 are listed here https://wiki.mozilla.org/Sheriffing/Job_Visibility_Policy#Additional_requirements_for_Tier_1_jobs

Comment 1

[Chris H-C :chutten]

Setting to "P1" (to get it out of triage) since it is being worked on.

Comment 2

[Raphael Aurich [:raphael] UTC+02:00]

I wanted to run the telemetry-tests-client suite on CI tier 1 on try using ./mach try fuzzy -q "'telemetry-tests-client".

This resulted in the following error:
Exception: test-linux64-ccov/debug-telemetry-tests-client (tier 1) cannot depend on build-linux64-ccov/debug (tier 2)

Ryan, can you please let me know what I can do now or refer me to someone who might know? Thank you!

Comment 3

[Ryan VanderMeulen [:RyanVM]]

We don't want to be a situation where a test suite is considered Tier 1 but the builds they depend on aren't. Does making it |tier: default| work?

Comment 4

[Raphael Aurich [:raphael] UTC+02:00]

Attachment: Bug 1531369 - Set telemetry-tests-client suite to tier default; r?gfritzsche

Comment 5

[Raphael Aurich [:raphael] UTC+02:00]

Hi Sebastian,

We're looking to move the integration test suite for Firefox Client Telemetry from CI tier 2 to CI tier 1, so that breaking changes related to the telemetry component result in a quick follow-up push or a backout.

We've worked towards meeting the requirements for CI tier 1 jobs from the wiki:

• the suite has a low intermittent failure rate
• the suite can be run locally and on try via mach
• we've added documentation for the suite to MDN

Can you think of any other measures to implement before we set the suite's tier to default? Thank you!

Comment 6

[Raphael Aurich [:raphael] UTC+02:00]

try run for "telemetry-tests-client" with the tier set to default:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=c6c3f5ad75f27d0bb08c6f3e22c79f7b30e10101

Comment 7

[Raphael Aurich [:raphael] UTC+02:00]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #3)

We don't want to be a situation where a test suite is considered Tier 1 but the builds they depend on aren't. Does making it |tier: default| work?

That works. Thank you, Ryan!

Comment 8

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Thank, this looks good to ship. Will there be macOS support eventually?

Comment 9

[Raphael Aurich [:raphael] UTC+02:00]

(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #8)

Thank, this looks good to ship. Will there be macOS support eventually?

Thank you for the feedback, Sebastian. I created a new bug for running the suite on macOS: bug 1546614

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/f83f1665555b
Set telemetry-tests-client suite to tier default; r=gfritzsche

Comment 11

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/f83f1665555b