1609174 - [tracking] switch release automation to use bouncer Nazgul

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Bouncer, task)

Description

[Mihai Tabara [:mtabara]⌚️GMT]

At some point in Q1 we'd like to take care of this lose end and fully rollout to bouncer Nazgul and eventually remove the existing bounceradmin.

In order for that to happen, we need to do a couple of things:

1. identify what's left from our side in order to submit to it across all branches/products (that includes Thunderbird, ESR68, etc)
2. how much work is needed from our side
3. what's currently broken so that all our nazgul jobs are busted everywhere.
4. what else do we need to do, outside of releng, to make this work end to end (bedrock/product-details Mozilla.org integration?)

Comment 1

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: [gecko] Add nazgul counterparts for remaining bouncer jobs

Add nazgul counterpart for bouncer-locations

Add nazgul counterpart for partnew-repack-bouncer-sub

Comment 2

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: [ciadmin] Bug 1609174 - enable bouncer-locations for nazgul. r=rail

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c800f4e37592
Add nazgul counterparts for remaining bouncer jobs r=rail

Comment 4

[Arthur Iakab [arthur_iakab]]

https://hg.mozilla.org/mozilla-central/rev/c800f4e37592

Comment 5

[Rail Aliiev [:rail]]

The automation code looks good, we just need to bump the version manually somehow in order to make the automation work next time.

https://firefoxci.taskcluster-artifacts.net/ORkABFkwTn2GHduLdCwwkQ/0/public/logs/live_backing.log

2020-04-01 09:07:25,895 - asyncio - DEBUG - Using selector: EpollSelector
2020-04-01 09:07:25,897 - bouncerscript.task - INFO - Action types: ['locations']
2020-04-01 09:07:25,898 - scriptworker.client - DEBUG - Task is validated against this schema: {'title': 'Taskcluster Bouncer locations task minimal schema', 'type': 'object', 'properties': {'dependencies': {'type': 'array', 'minItems': 1, 'uniqueItems': True, 'items': {'type': 'string'}}, 'scopes': {'type': 'array', 'minItems': 1, 'uniqueItems': True, 'items': {'type': 'string'}}, 'payload': {'type': 'object', 'properties': {'bouncer_products': {'type': 'array', 'minItems': 1, 'uniqueItems': True, 'items': {'type': 'string'}}, 'version': {'type': 'string', 'minItems': 1}, 'product': {'type': 'string', 'minItems': 1}}, 'required': ['bouncer_products', 'version', 'product'], 'additionalProperties': False}}, 'required': ['dependencies', 'scopes', 'payload']}
2020-04-01 09:07:25,899 - bouncerscript.task - INFO - Servers scopes: ['project:releng:bouncer:server:production-nazgul']
2020-04-01 09:07:25,900 - bouncerscript.task - INFO - Action types: ['locations']
2020-04-01 09:07:25,901 - bouncerscript.script - INFO - Preparing to update locations information in bouncer
2020-04-01 09:07:25,901 - bouncerscript.script - INFO - Sanity check bouncer products before updating ...
2020-04-01 09:07:25,901 - bouncerscript.script - INFO - All bouncer products look good before updating them!
2020-04-01 09:07:25,902 - bouncerscript.script - INFO - Sanity checking the version math makes sense ...
2020-04-01 09:07:25,902 - bouncerscript.script - INFO - In-tree version from payload looks good!
2020-04-01 09:07:25,902 - bouncerscript.script - INFO - Sanity check to ensure product exists...
2020-04-01 09:07:25,903 - bouncerscript.utils - INFO - Calling product_show?product=firefox-nightly-latest with data: {}
2020-04-01 09:07:25,903 - bouncerscript.utils - INFO - Performing a GET request to https://bounceradmin-default.prod.mozaws.net/api/product_show?product=firefox-nightly-latest with kwargs {'timeout': 60}
2020-04-01 09:07:26,060 - bouncerscript.utils - INFO - Server response: <?xml version="1.0" encoding="utf-8"?><products><product id="2005" name="firefox-nightly-latest"/></products>
2020-04-01 09:07:26,070 - bouncerscript.utils - INFO - Products found: 1
2020-04-01 09:07:26,071 - bouncerscript.utils - INFO - Calling location_show?product=firefox-nightly-latest with data: {}
2020-04-01 09:07:26,072 - bouncerscript.utils - INFO - Performing a GET request to https://bounceradmin-default.prod.mozaws.net/api/location_show?product=firefox-nightly-latest with kwargs {'timeout': 60}
2020-04-01 09:07:26,148 - bouncerscript.utils - INFO - Server response: <?xml version="1.0" encoding="utf-8"?><locations><product id="2005" name="firefox-nightly-latest"><location id="9379" os="win">/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.win32.installer.exe</location><location id="23647" os="linux">/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.linux-i686.tar.bz2</location><location id="23645" os="osx">/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.mac.dmg</location><location id="23646" os="linux64">/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.linux-x86_64.tar.bz2</location><location id="20140" os="win64">/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.win64.installer.exe</location><location id="51281" os="win64-aarch64">/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.win64-aarch64.installer.exe</location></product></locations>
2020-04-01 09:07:26,149 - bouncerscript.utils - DEBUG - Locations info: [{'os': 'win', 'id': '9379', 'path': '/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.win32.installer.exe'}, {'os': 'linux', 'id': '23647', 'path': '/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.linux-i686.tar.bz2'}, {'os': 'osx', 'id': '23645', 'path': '/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.mac.dmg'}, {'os': 'linux64', 'id': '23646', 'path': '/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.linux-x86_64.tar.bz2'}, {'os': 'win64', 'id': '20140', 'path': '/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.win64.installer.exe'}, {'os': 'win64-aarch64', 'id': '51281', 'path': '/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.win64-aarch64.installer.exe'}]
2020-04-01 09:07:26,149 - bouncerscript.script - INFO - Sanity check product firefox-nightly-latest platform win, path /firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.win32.installer.exe before bumping its version ...
2020-04-01 09:07:26,151 - scriptworker.client - ERROR - Failed to run async_main
Traceback (most recent call last):
  File "/app/lib/python3.8/site-packages/scriptworker/client.py", line 204, in _handle_asyncio_loop
    await async_main(context)
  File "/app/lib/python3.8/site-packages/bouncerscript/script.py", line 160, in async_main
    await action_map[context.action](context)
  File "/app/lib/python3.8/site-packages/bouncerscript/script.py", line 125, in bouncer_locations
    check_versions_are_successive(current_version, payload_version, product)
  File "/app/lib/python3.8/site-packages/bouncerscript/task.py", line 160, in check_versions_are_successive
    _successive_sanity(current_bouncer_version.major_number, candidate_version.major_number)
  File "/app/lib/python3.8/site-packages/bouncerscript/task.py", line 152, in _successive_sanity
    raise ScriptWorkerTaskException(err_msg)
scriptworker.exceptions.ScriptWorkerTaskException: In-tree version 76.0a1 can't be greater than current bouncer by more than 1 digit
exit code: 1

Comment 6

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Rail Aliiev [:rail] from comment #5)

The automation code looks good, we just need to bump the version manually somehow in order to make the automation work next time.

https://firefoxci.taskcluster-artifacts.net/ORkABFkwTn2GHduLdCwwkQ/0/public/logs/live_backing.log

2020-04-01 09:07:25,895 - asyncio - DEBUG - Using selector: EpollSelector
2020-04-01 09:07:25,897 - bouncerscript.task - INFO - Action types: ['locations']
2020-04-01 09:07:25,898 - scriptworker.client - DEBUG - Task is validated against this schema: {'title': 'Taskcluster Bouncer locations task minimal schema', 'type': 'object', 'properties': {'dependencies': {'type': 'array', 'minItems': 1, 'uniqueItems': True, 'items': {'type': 'string'}}, 'scopes': {'type': 'array', 'minItems': 1, 'uniqueItems': True, 'items': {'type': 'string'}}, 'payload': {'type': 'object', 'properties': {'bouncer_products': {'type': 'array', 'minItems': 1, 'uniqueItems': True, 'items': {'type': 'string'}}, 'version': {'type': 'string', 'minItems': 1}, 'product': {'type': 'string', 'minItems': 1}}, 'required': ['bouncer_products', 'version', 'product'], 'additionalProperties': False}}, 'required': ['dependencies', 'scopes', 'payload']}
2020-04-01 09:07:25,899 - bouncerscript.task - INFO - Servers scopes: ['project:releng:bouncer:server:production-nazgul']
2020-04-01 09:07:25,900 - bouncerscript.task - INFO - Action types: ['locations']
2020-04-01 09:07:25,901 - bouncerscript.script - INFO - Preparing to update locations information in bouncer
2020-04-01 09:07:25,901 - bouncerscript.script - INFO - Sanity check bouncer products before updating ...
2020-04-01 09:07:25,901 - bouncerscript.script - INFO - All bouncer products look good before updating them!
2020-04-01 09:07:25,902 - bouncerscript.script - INFO - Sanity checking the version math makes sense ...
2020-04-01 09:07:25,902 - bouncerscript.script - INFO - In-tree version from payload looks good!
2020-04-01 09:07:25,902 - bouncerscript.script - INFO - Sanity check to ensure product exists...
2020-04-01 09:07:25,903 - bouncerscript.utils - INFO - Calling product_show?product=firefox-nightly-latest with data: {}
2020-04-01 09:07:25,903 - bouncerscript.utils - INFO - Performing a GET request to https://bounceradmin-default.prod.mozaws.net/api/product_show?product=firefox-nightly-latest with kwargs {'timeout': 60}
2020-04-01 09:07:26,060 - bouncerscript.utils - INFO - Server response: <?xml version="1.0" encoding="utf-8"?><products><product id="2005" name="firefox-nightly-latest"/></products>
2020-04-01 09:07:26,070 - bouncerscript.utils - INFO - Products found: 1
2020-04-01 09:07:26,071 - bouncerscript.utils - INFO - Calling location_show?product=firefox-nightly-latest with data: {}
2020-04-01 09:07:26,072 - bouncerscript.utils - INFO - Performing a GET request to https://bounceradmin-default.prod.mozaws.net/api/location_show?product=firefox-nightly-latest with kwargs {'timeout': 60}
2020-04-01 09:07:26,148 - bouncerscript.utils - INFO - Server response: <?xml version="1.0" encoding="utf-8"?><locations><product id="2005" name="firefox-nightly-latest"><location id="9379" os="win">/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.win32.installer.exe</location><location id="23647" os="linux">/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.linux-i686.tar.bz2</location><location id="23645" os="osx">/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.mac.dmg</location><location id="23646" os="linux64">/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.linux-x86_64.tar.bz2</location><location id="20140" os="win64">/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.win64.installer.exe</location><location id="51281" os="win64-aarch64">/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.win64-aarch64.installer.exe</location></product></locations>
2020-04-01 09:07:26,149 - bouncerscript.utils - DEBUG - Locations info: [{'os': 'win', 'id': '9379', 'path': '/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.win32.installer.exe'}, {'os': 'linux', 'id': '23647', 'path': '/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.linux-i686.tar.bz2'}, {'os': 'osx', 'id': '23645', 'path': '/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.mac.dmg'}, {'os': 'linux64', 'id': '23646', 'path': '/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.linux-x86_64.tar.bz2'}, {'os': 'win64', 'id': '20140', 'path': '/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.win64.installer.exe'}, {'os': 'win64-aarch64', 'id': '51281', 'path': '/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.win64-aarch64.installer.exe'}]
2020-04-01 09:07:26,149 - bouncerscript.script - INFO - Sanity check product firefox-nightly-latest platform win, path /firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.win32.installer.exe before bumping its version ...
2020-04-01 09:07:26,151 - scriptworker.client - ERROR - Failed to run async_main
Traceback (most recent call last):
  File "/app/lib/python3.8/site-packages/scriptworker/client.py", line 204, in _handle_asyncio_loop
    await async_main(context)
  File "/app/lib/python3.8/site-packages/bouncerscript/script.py", line 160, in async_main
    await action_map[context.action](context)
  File "/app/lib/python3.8/site-packages/bouncerscript/script.py", line 125, in bouncer_locations
    check_versions_are_successive(current_version, payload_version, product)
  File "/app/lib/python3.8/site-packages/bouncerscript/task.py", line 160, in check_versions_are_successive
    _successive_sanity(current_bouncer_version.major_number, candidate_version.major_number)
  File "/app/lib/python3.8/site-packages/bouncerscript/task.py", line 152, in _successive_sanity
    raise ScriptWorkerTaskException(err_msg)
scriptworker.exceptions.ScriptWorkerTaskException: In-tree version 76.0a1 can't be greater than current bouncer by more than 1 digit
exit code: 1

We're trying to bump 76 against bouncer-stored 70 so this is expected. Likely the DB sync that was used to populate Nazgul is very outdated. Added this to my list to ask Jeremy tomorrow at the mtg.

Comment 7

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Bug 1609174 - enable bouncer-locations for nazgul. r=rail

Comment 8

[Mihai Tabara [:mtabara]⌚️GMT]

Doing some bug triaging and trello board updates - let's keep this bug until we have Nazul in production.

Comment 9

[Mihai Tabara [:mtabara]⌚️GMT]

Hm, I see Thunderbird still doesn't have the bouncer-aliases-nazgul counterpart in the ship graph, looking at the last graph https://firefox-ci-tc.services.mozilla.com/tasks/groups/UI5qapvqTyGJr50xcMe6bA from today.
@rjl: have the patches from this bug landed on thunderbird at some point and still riding the trains?

Comment 10

[Rob Lemley [:rjl]]

They're on beta, but I haven't uplifted to ESR68. I'll get that process started for the next one.

Comment 11

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Rob Lemley [:rjl] from comment #10)

They're on beta, but I haven't uplifted to ESR68. I'll get that process started for the next one.

Bah, my fault I think, I might've forgotten to uplift this to ESR68 :/ I'll get back to this by EOD.

Comment 12

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 9135147 [details]
[gecko] Add nazgul counterparts for remaining bouncer jobs

ESR Uplift Approval Request

• If this is not a sec:{high,crit} bug, please state case for ESR consideration: We are currently doing a migration for the admin part of the bouncer. Enabling this on ESR68 allows us to migrate sooner.
• User impact if declined:
• Fix Landed on Version:
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky):
• String or UUID changes made by this patch:

Comment 13

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #11)

(In reply to Rob Lemley [:rjl] from comment #10)

They're on beta, but I haven't uplifted to ESR68. I'll get that process started for the next one.

Bah, my fault I think, I might've forgotten to uplift this to ESR68 :/ I'll get back to this by EOD.

Indeed we don't have it on ESR68, mea culpa. Once we have it there, we can fix TB esr too and then we're ready to migrate to Nazgul completely.

Comment 14

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9135147 [details]
[gecko] Add nazgul counterparts for remaining bouncer jobs

Needed for release things. Approved for ESR68.

Comment 15

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr68/rev/0cc38b766974

Comment 16

[Justin Wood (:Callek)]

Attachment: Bug 1609174 - Followup, make sure bouncer is in worker aliases and support by-release-level on esr, initially from D58018. r=rail

Patch is only slated for ESR68

Comment 17

[Mihai Tabara [:mtabara]⌚️GMT]

Callek landed https://hg.mozilla.org/releases/mozilla-esr68/rev/444efed2d7106fae1b61b014034478102d20de07 as follow-up fix for my previous patch. Thanks again!

Comment 18

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #13)

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #11)

(In reply to Rob Lemley [:rjl] from comment #10)

They're on beta, but I haven't uplifted to ESR68. I'll get that process started for the next one.

Bah, my fault I think, I might've forgotten to uplift this to ESR68 :/ I'll get back to this by EOD.

Indeed we don't have it on ESR68, mea culpa. Once we have it there, we can fix TB esr too and then we're ready to migrate to Nazgul completely.

@rjl: just to confirm, has this made it to Thunderbird?

Comment 19

[Rob Lemley [:rjl]]

I have it queued up for the next 68 release which is planned for late this week.

Comment 20

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Rob Lemley [:rjl] from comment #19)

I have it queued up for the next 68 release which is planned for late this week.

Awesome! With the TB migration done, we'll be ready to make the back-end switch after Firefox 77.0 ships in two weeks!