1519054 - add AArch64 as part of the bouncer locations being updated after each mergeduty

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect, P1)

Description

[Treeherder Bug Filer]

Filed by: archaeopteryx [at] coole-files.de

https://treeherder.mozilla.org/logviewer.html#?job_id=221049621&repo=mozilla-central

https://queue.taskcluster.net/v1/task/Gs4qOS-ySQa3ryv9y8E9Nw/runs/0/artifacts/public/logs/live_backing.log

2019-01-10 10:13:19,578 - scriptworker.client - DEBUG - Task is validated against this schema: {'title': 'Taskcluster Bouncer locations task minimal schema', 'type': 'object', 'properties': {'dependencies': {'type': 'array', 'minItems': 1, 'uniqueItems': True, 'items': {'type': 'string'}}, 'scopes': {'type': 'array', 'minItems': 1, 'uniqueItems': True, 'items': {'type': 'string'}}, 'payload': {'type': 'object', 'properties': {'bouncer_products': {'type': 'array', 'minItems': 1, 'uniqueItems': True, 'items': {'type': 'string'}}, 'version': {'type': 'string', 'minItems': 1}}, 'required': ['bouncer_products', 'version'], 'additionalProperties': False}}, 'required': ['dependencies', 'scopes', 'payload']}
2019-01-10 10:13:19,581 - bouncerscript.task - INFO - Servers scopes: ['project:releng:bouncer:server:production']
2019-01-10 10:13:19,582 - bouncerscript.task - INFO - Action types: ['locations']
2019-01-10 10:13:19,582 - bouncerscript.script - INFO - Preparing to update locations information in bouncer
2019-01-10 10:13:19,583 - bouncerscript.script - INFO - Sanity check products before updating ...
2019-01-10 10:13:19,583 - bouncerscript.script - INFO - All bouncer products look good before updating them!
2019-01-10 10:13:19,584 - bouncerscript.script - INFO - Sanity checking the version math makes sense ...
2019-01-10 10:13:19,584 - bouncerscript.script - INFO - In-tree version from payload looks good!
2019-01-10 10:13:19,585 - bouncerscript.script - INFO - Sanity check to ensure product exists...
2019-01-10 10:13:19,586 - bouncerscript.utils - INFO - Calling product_show?product=firefox-nightly-latest with data: {}
2019-01-10 10:13:19,586 - bouncerscript.utils - INFO - Performing a GET request to https://bounceradmin.mozilla.com/api/product_show?product=firefox-nightly-latest with kwargs {'timeout': 60}
2019-01-10 10:13:20,099 - bouncerscript.utils - INFO - Server response: <?xml version="1.0" encoding="utf-8"?><products><product id="2005" name="firefox-nightly-latest"/></products>
2019-01-10 10:13:20,103 - bouncerscript.utils - INFO - Products found: 1
2019-01-10 10:13:20,103 - bouncerscript.utils - INFO - Calling location_show?product=firefox-nightly-latest with data: {}
2019-01-10 10:13:20,104 - bouncerscript.utils - INFO - Performing a GET request to https://bounceradmin.mozilla.com/api/location_show?product=firefox-nightly-latest with kwargs {'timeout': 60}
2019-01-10 10:13:20,231 - bouncerscript.utils - INFO - Server response: <?xml version="1.0" encoding="utf-8"?><locations><product id="2005" name="firefox-nightly-latest"><location id="9379" os="win">/firefox/nightly/latest-mozilla-central-l10n/firefox-66.0a1.:lang.win32.installer.exe</location><location id="23647" os="linux">/firefox/nightly/latest-mozilla-central-l10n/firefox-66.0a1.:lang.linux-i686.tar.bz2</location><location id="23645" os="osx">/firefox/nightly/latest-mozilla-central-l10n/firefox-66.0a1.:lang.mac.dmg</location><location id="23646" os="linux64">/firefox/nightly/latest-mozilla-central-l10n/firefox-66.0a1.:lang.linux-x86_64.tar.bz2</location><location id="20140" os="win64">/firefox/nightly/latest-mozilla-central-l10n/firefox-66.0a1.:lang.win64.installer.exe</location><location id="51281" os="win64-aarch64">/firefox/nightly/latest-mozilla-central-l10n/firefox-66.0a1.:lang.win64-aarch64.installer.exe</location></product></locations>
2019-01-10 10:13:20,233 - scriptworker.client - ERROR - Failed to run async_main
Traceback (most recent call last):
File "/builds/scriptworker/lib/python3.6/site-packages/scriptworker/client.py", line 203, in _handle_asyncio_loop
await async_main(context)
File "/builds/scriptworker/lib/python3.6/site-packages/bouncerscript/script.py", line 162, in async_main
await action_mapcontext.action
File "/builds/scriptworker/lib/python3.6/site-packages/bouncerscript/script.py", line 109, in bouncer_locations
locations_info = await get_locations_info(context, product_name)
File "/builds/scriptworker/lib/python3.6/site-packages/bouncerscript/utils.py", line 161, in get_locations_info
locations_info = process_locations_from_xml(xml_doc.getElementsByTagName("location"))
File "/builds/scriptworker/lib/python3.6/site-packages/bouncerscript/utils.py", line 177, in process_locations_from_xml
raise ScriptWorkerTaskException(err_msg)
scriptworker.exceptions.ScriptWorkerTaskException: Unexpected os found in bouncer. Found win64-aarch64 while expected ['linux', 'linux64', 'osx', 'win', 'win64', 'android-x86', 'android'].
exit code: 1

Comment 1

[Intermittent Failures Robot]

13 failures in 2627 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 13

Platform breakdown:

• firefox-release: 13

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1519054&startday=2019-01-07&endday=2019-01-13&tree=all

Comment 2

[Mihai Tabara [:mtabara]⌚️GMT]

Resurfacing this bug from last week.

The errors are expected. It's the reverse effect described in 1519941.

Comment 4

[Mihai Tabara [:mtabara]⌚️GMT]

• need to add the platform as available in the constants.py
• add this as a good unit test to cover

Comment 5

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #4)

• need to add the platform as available in the constants.py
• add this as a good unit test to cover

Turns out we already have tests for this in here[1].
Following-up the PR to add the win64 platform only.

[1] https://github.com/mozilla-releng/bouncerscript/blob/master/bouncerscript/test/test_utils.py#L523

Comment 6

[Mihai Tabara [:mtabara]⌚️GMT]

Note to self: let's track here the in-tree patch needed as well in order to update this with each locations job per nightly graphs

Comment 7

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: [bouncerscript] Add support for AArch64 to fix bouncer locations nightly job

Comment 8

[Mihai Tabara [:mtabara]⌚️GMT]

This solves the current issue that we've been hitting in the jobs.
a) we still need to solve adding the new location in-tree payload
b) make sure no other amendments are needed in bouncerscript. (potentially another integration test or amend existing to include the new locations as well)

Will continue tomorrow morning.

Comment 9

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 9036424 [details] [review]
[bouncerscript] Add support for AArch64 to fix bouncer locations nightly job

r+'ed by jlorenzo in PR.
Landed + bumped bouncerscript.

Comment 10

[Mihai Tabara [:mtabara]⌚️GMT]

Tasks are now green again.

Comment 11

[Intermittent Failures Robot]

6 failures in 2378 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 6

Platform breakdown:

• firefox-release: 6

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1519054&startday=2019-01-14&endday=2019-01-20&tree=all