1598436 - automate check_bouncer.py in taskcluster cron

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, task)

Description

[Aki Sasaki (not active)]

This check is currently running on bb01, running the script https://hg.mozilla.org/build/nagios-tools/file/tip/nagios_tools/scripts/check_bouncer.py . We'd like to shut down bb01, so let's update the code to use py3 and schedule it in taskcluster cron.

Related:

mbrandt:
@aki that's a trip down memory lane .. I was pulled off go-bouncer about a year ago. Both rail and oremj were the last two ppl I spoke to during the handoff
@aki it might also be valuable to investigate if these checks are still being run by someone https://github.com/mozilla-services/go-bouncer/tree/master/tests/e2e

Oremj said these aren't running anywhere we know of, so until we have an alternate solution, let's port the bb01 check to taskcluster cron.

Comment 1

[Aki Sasaki (not active)]

Hm, Mihai pointed out Simon did this, and I misread. sfraser++

https://searchfox.org/mozilla-central/source/taskcluster/ci/cron-bouncer-check/kind.yml#28
https://treeherder.mozilla.org/#/jobs?repo=mozilla-beta&selectedJob=277570085&searchStr=ckbouncer
https://firefoxci.taskcluster-artifacts.net/CkR2_E_cR8SHpENZ07Brrw/0/public/logs/live_backing.log

Should we resolve this bug?

Comment 2

[Simon Fraser [:sfraser] ⌚️GMT]

Do we need to do anything special to turn off the bb01 check? If not then let's just let it pass into the hall of fame and close this bug

Comment 3

[Aki Sasaki (not active)]

I'm going to be shutting off bb01 after l10n bumper is fully ported to taskcluster cron in bug 1481916.
Let's call this fixed.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

You may have missed that the products checked are different. The nagios check uses the latest products like firefox-beta-latest-ssl while in-tree cron uses versioned products like Firefox-70.0b12-SSL. I think we should use the latest style as that's what www.mozilla.org uses for installers.

A set of changes like

diff --git a/testing/mozharness/configs/releases/bouncer_firefox_beta.py b/testing/mozharness/configs/releases/bouncer_firefox_beta.py
--- a/testing/mozharness/configs/releases/bouncer_firefox_beta.py
+++ b/testing/mozharness/configs/releases/bouncer_firefox_beta.py
@@ -5,1 +5,1 @@ config = {
-            "product-name": "Firefox-%(version)s",
+            "product-name": "Firefox-beta-latest",
@@ -17,1 +17,1 @@ config = {
-            "product-name": "Firefox-%(version)s-SSL",
+            "product-name": "Firefox-beta-latest-SSL",
@@ -29,1 +29,1 @@ config = {
-            "product-name": "Firefox-%(version)s-msi-SSL",
+            "product-name": "Firefox-beta-msi-latest-ssl",
@@ -37,1 +37,1 @@ config = {
-            "product-name": "Firefox-%(version)s-stub",
+            "product-name": "Firefox-beta-stub",

across the branches would use the newer test but check what we use in prod (we'd leave the complete update check alone).

I'm not sure what we should do for the nightly case, maybe some extra work there to enable the check there in .cron.yml etc.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Perhaps I was a bit hasty. The configs are used in a couple of different places

1. the release-bouncer-check-firefox kind during the push phase of a release, where it's passed current and previous versions as arguments. It makes sense to check the versioned products here, and it includes full checks on the update products
2. the cron-bouncer-check kind once a day, where I was thinking we should use the latest products

Looks like we'd have to duplicate the configs for the two cases because of how the script works.

Comment 6

[Aki Sasaki (not active)]

Are we able to add the missing configs to the cron-bouncer-check, update the timing, and call it done?

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Bug 1598436 - add checks for 'latest' bouncer products to cron (beta, deved, release), r=aki

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Bug 1598436 - add ESR bouncer checks to cron, r=aki

Fixes up ESR bouncer aliases now that ESR60 is EOL.
Adds check for latest products used by www.mozilla.org.
Enables the cron checks.
Fixes up a script error which meant we weren't checking all partial platforms in release automation.

Depends on D54656

Comment 9

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: Bug 1598436 - add nightly bouncer checks to cron, r=aki

Depends on D54657

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e165b244de59
add checks for 'latest' bouncer products to cron (beta, deved, release), r=aki
https://hg.mozilla.org/integration/autoland/rev/1b24d899e04f
add ESR bouncer checks to cron, r=aki
https://hg.mozilla.org/integration/autoland/rev/3310bcfea608
add nightly bouncer checks to cron, r=aki

Comment 11

[Oana Pop-Rus]

https://hg.mozilla.org/mozilla-central/rev/e165b244de59
https://hg.mozilla.org/mozilla-central/rev/1b24d899e04f
https://hg.mozilla.org/mozilla-central/rev/3310bcfea608

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

TODO - uplift to release (71) and esr68 after we release.

Comment 13

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/releases/mozilla-release/rev/feaf62cb9b9d
https://hg.mozilla.org/releases/mozilla-release/rev/3c3760cc2ebe
https://hg.mozilla.org/releases/mozilla-release/rev/290d570566d6

Comment 14

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/releases/mozilla-esr68/rev/35bc3746ef35
https://hg.mozilla.org/releases/mozilla-esr68/rev/15f9e0a61e78
https://hg.mozilla.org/releases/mozilla-esr68/rev/16ca1384f658

Comment 15

[Nick Thomas [:nthomas] (UTC+12)]

The first checks after the code changes look good on esr68 and release.