Closed Bug 1609174 Opened 7 months ago Closed 3 months ago

[tracking] switch release automation to use bouncer Nazgul

Categories

(Release Engineering :: Release Automation: Bouncer, task)

task
Not set
normal

Tracking

(firefox-esr68 fixed, firefox76 fixed)

RESOLVED FIXED
Tracking Status
firefox-esr68 --- fixed
firefox76 --- fixed

People

(Reporter: mtabara, Assigned: rail)

References

(Blocks 1 open bug)

Details

Attachments

(3 files, 1 obsolete file)

At some point in Q1 we'd like to take care of this lose end and fully rollout to bouncer Nazgul and eventually remove the existing bounceradmin.

In order for that to happen, we need to do a couple of things:

  1. identify what's left from our side in order to submit to it across all branches/products (that includes Thunderbird, ESR68, etc)
  2. how much work is needed from our side
  3. what's currently broken so that all our nazgul jobs are busted everywhere.
  4. what else do we need to do, outside of releng, to make this work end to end (bedrock/product-details Mozilla.org integration?)
Depends on: 1599385

Add nazgul counterpart for bouncer-locations

Add nazgul counterpart for partnew-repack-bouncer-sub

Attachment #9135147 - Attachment description: Add nazgul counterpart for bouncer-aliases → Add nazgul counterparts for remaining bouncer jobs
Depends on: 1624369
Assignee: mtabara → rail
Pushed by raliiev@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/c800f4e37592
Add nazgul counterparts for remaining bouncer jobs r=rail
Status: NEW → RESOLVED
Closed: 5 months ago
Resolution: --- → FIXED

The automation code looks good, we just need to bump the version manually somehow in order to make the automation work next time.

https://firefoxci.taskcluster-artifacts.net/ORkABFkwTn2GHduLdCwwkQ/0/public/logs/live_backing.log

2020-04-01 09:07:25,895 - asyncio - DEBUG - Using selector: EpollSelector
2020-04-01 09:07:25,897 - bouncerscript.task - INFO - Action types: ['locations']
2020-04-01 09:07:25,898 - scriptworker.client - DEBUG - Task is validated against this schema: {'title': 'Taskcluster Bouncer locations task minimal schema', 'type': 'object', 'properties': {'dependencies': {'type': 'array', 'minItems': 1, 'uniqueItems': True, 'items': {'type': 'string'}}, 'scopes': {'type': 'array', 'minItems': 1, 'uniqueItems': True, 'items': {'type': 'string'}}, 'payload': {'type': 'object', 'properties': {'bouncer_products': {'type': 'array', 'minItems': 1, 'uniqueItems': True, 'items': {'type': 'string'}}, 'version': {'type': 'string', 'minItems': 1}, 'product': {'type': 'string', 'minItems': 1}}, 'required': ['bouncer_products', 'version', 'product'], 'additionalProperties': False}}, 'required': ['dependencies', 'scopes', 'payload']}
2020-04-01 09:07:25,899 - bouncerscript.task - INFO - Servers scopes: ['project:releng:bouncer:server:production-nazgul']
2020-04-01 09:07:25,900 - bouncerscript.task - INFO - Action types: ['locations']
2020-04-01 09:07:25,901 - bouncerscript.script - INFO - Preparing to update locations information in bouncer
2020-04-01 09:07:25,901 - bouncerscript.script - INFO - Sanity check bouncer products before updating ...
2020-04-01 09:07:25,901 - bouncerscript.script - INFO - All bouncer products look good before updating them!
2020-04-01 09:07:25,902 - bouncerscript.script - INFO - Sanity checking the version math makes sense ...
2020-04-01 09:07:25,902 - bouncerscript.script - INFO - In-tree version from payload looks good!
2020-04-01 09:07:25,902 - bouncerscript.script - INFO - Sanity check to ensure product exists...
2020-04-01 09:07:25,903 - bouncerscript.utils - INFO - Calling product_show?product=firefox-nightly-latest with data: {}
2020-04-01 09:07:25,903 - bouncerscript.utils - INFO - Performing a GET request to https://bounceradmin-default.prod.mozaws.net/api/product_show?product=firefox-nightly-latest with kwargs {'timeout': 60}
2020-04-01 09:07:26,060 - bouncerscript.utils - INFO - Server response: <?xml version="1.0" encoding="utf-8"?><products><product id="2005" name="firefox-nightly-latest"/></products>
2020-04-01 09:07:26,070 - bouncerscript.utils - INFO - Products found: 1
2020-04-01 09:07:26,071 - bouncerscript.utils - INFO - Calling location_show?product=firefox-nightly-latest with data: {}
2020-04-01 09:07:26,072 - bouncerscript.utils - INFO - Performing a GET request to https://bounceradmin-default.prod.mozaws.net/api/location_show?product=firefox-nightly-latest with kwargs {'timeout': 60}
2020-04-01 09:07:26,148 - bouncerscript.utils - INFO - Server response: <?xml version="1.0" encoding="utf-8"?><locations><product id="2005" name="firefox-nightly-latest"><location id="9379" os="win">/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.win32.installer.exe</location><location id="23647" os="linux">/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.linux-i686.tar.bz2</location><location id="23645" os="osx">/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.mac.dmg</location><location id="23646" os="linux64">/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.linux-x86_64.tar.bz2</location><location id="20140" os="win64">/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.win64.installer.exe</location><location id="51281" os="win64-aarch64">/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.win64-aarch64.installer.exe</location></product></locations>
2020-04-01 09:07:26,149 - bouncerscript.utils - DEBUG - Locations info: [{'os': 'win', 'id': '9379', 'path': '/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.win32.installer.exe'}, {'os': 'linux', 'id': '23647', 'path': '/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.linux-i686.tar.bz2'}, {'os': 'osx', 'id': '23645', 'path': '/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.mac.dmg'}, {'os': 'linux64', 'id': '23646', 'path': '/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.linux-x86_64.tar.bz2'}, {'os': 'win64', 'id': '20140', 'path': '/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.win64.installer.exe'}, {'os': 'win64-aarch64', 'id': '51281', 'path': '/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.win64-aarch64.installer.exe'}]
2020-04-01 09:07:26,149 - bouncerscript.script - INFO - Sanity check product firefox-nightly-latest platform win, path /firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.win32.installer.exe before bumping its version ...
2020-04-01 09:07:26,151 - scriptworker.client - ERROR - Failed to run async_main
Traceback (most recent call last):
  File "/app/lib/python3.8/site-packages/scriptworker/client.py", line 204, in _handle_asyncio_loop
    await async_main(context)
  File "/app/lib/python3.8/site-packages/bouncerscript/script.py", line 160, in async_main
    await action_map[context.action](context)
  File "/app/lib/python3.8/site-packages/bouncerscript/script.py", line 125, in bouncer_locations
    check_versions_are_successive(current_version, payload_version, product)
  File "/app/lib/python3.8/site-packages/bouncerscript/task.py", line 160, in check_versions_are_successive
    _successive_sanity(current_bouncer_version.major_number, candidate_version.major_number)
  File "/app/lib/python3.8/site-packages/bouncerscript/task.py", line 152, in _successive_sanity
    raise ScriptWorkerTaskException(err_msg)
scriptworker.exceptions.ScriptWorkerTaskException: In-tree version 76.0a1 can't be greater than current bouncer by more than 1 digit
exit code: 1
Status: RESOLVED → REOPENED
Resolution: FIXED → ---

(In reply to Rail Aliiev [:rail] from comment #5)

The automation code looks good, we just need to bump the version manually somehow in order to make the automation work next time.

https://firefoxci.taskcluster-artifacts.net/ORkABFkwTn2GHduLdCwwkQ/0/public/logs/live_backing.log

2020-04-01 09:07:25,895 - asyncio - DEBUG - Using selector: EpollSelector
2020-04-01 09:07:25,897 - bouncerscript.task - INFO - Action types: ['locations']
2020-04-01 09:07:25,898 - scriptworker.client - DEBUG - Task is validated against this schema: {'title': 'Taskcluster Bouncer locations task minimal schema', 'type': 'object', 'properties': {'dependencies': {'type': 'array', 'minItems': 1, 'uniqueItems': True, 'items': {'type': 'string'}}, 'scopes': {'type': 'array', 'minItems': 1, 'uniqueItems': True, 'items': {'type': 'string'}}, 'payload': {'type': 'object', 'properties': {'bouncer_products': {'type': 'array', 'minItems': 1, 'uniqueItems': True, 'items': {'type': 'string'}}, 'version': {'type': 'string', 'minItems': 1}, 'product': {'type': 'string', 'minItems': 1}}, 'required': ['bouncer_products', 'version', 'product'], 'additionalProperties': False}}, 'required': ['dependencies', 'scopes', 'payload']}
2020-04-01 09:07:25,899 - bouncerscript.task - INFO - Servers scopes: ['project:releng:bouncer:server:production-nazgul']
2020-04-01 09:07:25,900 - bouncerscript.task - INFO - Action types: ['locations']
2020-04-01 09:07:25,901 - bouncerscript.script - INFO - Preparing to update locations information in bouncer
2020-04-01 09:07:25,901 - bouncerscript.script - INFO - Sanity check bouncer products before updating ...
2020-04-01 09:07:25,901 - bouncerscript.script - INFO - All bouncer products look good before updating them!
2020-04-01 09:07:25,902 - bouncerscript.script - INFO - Sanity checking the version math makes sense ...
2020-04-01 09:07:25,902 - bouncerscript.script - INFO - In-tree version from payload looks good!
2020-04-01 09:07:25,902 - bouncerscript.script - INFO - Sanity check to ensure product exists...
2020-04-01 09:07:25,903 - bouncerscript.utils - INFO - Calling product_show?product=firefox-nightly-latest with data: {}
2020-04-01 09:07:25,903 - bouncerscript.utils - INFO - Performing a GET request to https://bounceradmin-default.prod.mozaws.net/api/product_show?product=firefox-nightly-latest with kwargs {'timeout': 60}
2020-04-01 09:07:26,060 - bouncerscript.utils - INFO - Server response: <?xml version="1.0" encoding="utf-8"?><products><product id="2005" name="firefox-nightly-latest"/></products>
2020-04-01 09:07:26,070 - bouncerscript.utils - INFO - Products found: 1
2020-04-01 09:07:26,071 - bouncerscript.utils - INFO - Calling location_show?product=firefox-nightly-latest with data: {}
2020-04-01 09:07:26,072 - bouncerscript.utils - INFO - Performing a GET request to https://bounceradmin-default.prod.mozaws.net/api/location_show?product=firefox-nightly-latest with kwargs {'timeout': 60}
2020-04-01 09:07:26,148 - bouncerscript.utils - INFO - Server response: <?xml version="1.0" encoding="utf-8"?><locations><product id="2005" name="firefox-nightly-latest"><location id="9379" os="win">/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.win32.installer.exe</location><location id="23647" os="linux">/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.linux-i686.tar.bz2</location><location id="23645" os="osx">/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.mac.dmg</location><location id="23646" os="linux64">/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.linux-x86_64.tar.bz2</location><location id="20140" os="win64">/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.win64.installer.exe</location><location id="51281" os="win64-aarch64">/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.win64-aarch64.installer.exe</location></product></locations>
2020-04-01 09:07:26,149 - bouncerscript.utils - DEBUG - Locations info: [{'os': 'win', 'id': '9379', 'path': '/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.win32.installer.exe'}, {'os': 'linux', 'id': '23647', 'path': '/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.linux-i686.tar.bz2'}, {'os': 'osx', 'id': '23645', 'path': '/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.mac.dmg'}, {'os': 'linux64', 'id': '23646', 'path': '/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.linux-x86_64.tar.bz2'}, {'os': 'win64', 'id': '20140', 'path': '/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.win64.installer.exe'}, {'os': 'win64-aarch64', 'id': '51281', 'path': '/firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.win64-aarch64.installer.exe'}]
2020-04-01 09:07:26,149 - bouncerscript.script - INFO - Sanity check product firefox-nightly-latest platform win, path /firefox/nightly/latest-mozilla-central-l10n/firefox-70.0a1.:lang.win32.installer.exe before bumping its version ...
2020-04-01 09:07:26,151 - scriptworker.client - ERROR - Failed to run async_main
Traceback (most recent call last):
  File "/app/lib/python3.8/site-packages/scriptworker/client.py", line 204, in _handle_asyncio_loop
    await async_main(context)
  File "/app/lib/python3.8/site-packages/bouncerscript/script.py", line 160, in async_main
    await action_map[context.action](context)
  File "/app/lib/python3.8/site-packages/bouncerscript/script.py", line 125, in bouncer_locations
    check_versions_are_successive(current_version, payload_version, product)
  File "/app/lib/python3.8/site-packages/bouncerscript/task.py", line 160, in check_versions_are_successive
    _successive_sanity(current_bouncer_version.major_number, candidate_version.major_number)
  File "/app/lib/python3.8/site-packages/bouncerscript/task.py", line 152, in _successive_sanity
    raise ScriptWorkerTaskException(err_msg)
scriptworker.exceptions.ScriptWorkerTaskException: In-tree version 76.0a1 can't be greater than current bouncer by more than 1 digit
exit code: 1

We're trying to bump 76 against bouncer-stored 70 so this is expected. Likely the DB sync that was used to populate Nazgul is very outdated. Added this to my list to ask Jeremy tomorrow at the mtg.

Regressions: 1627996
Attachment #9139558 - Attachment is obsolete: true
Blocks: 1629944
Depends on: 1629969

Doing some bug triaging and trello board updates - let's keep this bug until we have Nazul in production.

Blocks: 1635159

Hm, I see Thunderbird still doesn't have the bouncer-aliases-nazgul counterpart in the ship graph, looking at the last graph https://firefox-ci-tc.services.mozilla.com/tasks/groups/UI5qapvqTyGJr50xcMe6bA from today.
@rjl: have the patches from this bug landed on thunderbird at some point and still riding the trains?

Flags: needinfo?(rob)

They're on beta, but I haven't uplifted to ESR68. I'll get that process started for the next one.

Flags: needinfo?(rob)

(In reply to Rob Lemley [:rjl] from comment #10)

They're on beta, but I haven't uplifted to ESR68. I'll get that process started for the next one.

Bah, my fault I think, I might've forgotten to uplift this to ESR68 :/ I'll get back to this by EOD.

Attachment #9135150 - Attachment description: Bug 1609174 - enable bouncer-locations for nazgul. r=rail → [ciadmin] Bug 1609174 - enable bouncer-locations for nazgul. r=rail

Comment on attachment 9135147 [details]
[gecko] Add nazgul counterparts for remaining bouncer jobs

ESR Uplift Approval Request

  • If this is not a sec:{high,crit} bug, please state case for ESR consideration: We are currently doing a migration for the admin part of the bouncer. Enabling this on ESR68 allows us to migrate sooner.
  • User impact if declined:
  • Fix Landed on Version:
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky):
  • String or UUID changes made by this patch:
Attachment #9135147 - Attachment description: Add nazgul counterparts for remaining bouncer jobs → [gecko] Add nazgul counterparts for remaining bouncer jobs
Attachment #9135147 - Flags: approval-mozilla-esr68?

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #11)

(In reply to Rob Lemley [:rjl] from comment #10)

They're on beta, but I haven't uplifted to ESR68. I'll get that process started for the next one.

Bah, my fault I think, I might've forgotten to uplift this to ESR68 :/ I'll get back to this by EOD.

Indeed we don't have it on ESR68, mea culpa. Once we have it there, we can fix TB esr too and then we're ready to migrate to Nazgul completely.

Blocks: 1635525

Comment on attachment 9135147 [details]
[gecko] Add nazgul counterparts for remaining bouncer jobs

Needed for release things. Approved for ESR68.

Attachment #9135147 - Flags: approval-mozilla-esr68? → approval-mozilla-esr68+

Callek landed https://hg.mozilla.org/releases/mozilla-esr68/rev/444efed2d7106fae1b61b014034478102d20de07 as follow-up fix for my previous patch. Thanks again!

Status: REOPENED → RESOLVED
Closed: 5 months ago3 months ago
Resolution: --- → FIXED

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #13)

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #11)

(In reply to Rob Lemley [:rjl] from comment #10)

They're on beta, but I haven't uplifted to ESR68. I'll get that process started for the next one.

Bah, my fault I think, I might've forgotten to uplift this to ESR68 :/ I'll get back to this by EOD.

Indeed we don't have it on ESR68, mea culpa. Once we have it there, we can fix TB esr too and then we're ready to migrate to Nazgul completely.

@rjl: just to confirm, has this made it to Thunderbird?

Flags: needinfo?(rob)

I have it queued up for the next 68 release which is planned for late this week.

Flags: needinfo?(rob)

(In reply to Rob Lemley [:rjl] from comment #19)

I have it queued up for the next 68 release which is planned for late this week.

Awesome! With the TB migration done, we'll be ready to make the back-end switch after Firefox 77.0 ships in two weeks!

You need to log in before you can comment on or make changes to this bug.