1800611 - firefox-android: Use gecko version numbers

Status: RESOLVED FIXED

(Release Engineering :: General, task)

Description

[Johan Lorenzo [:jlorenzo]]

For historical reasons, Android versions don't follow the same pattern. Fenix and Focus were moved to the regular Gecko pattern in bug 1777255, but Android-Components remained on its own pattern. I think it's time we change it too. It will be a problem to solve once we move focus to the Android monorepo (bug 1797704) but it also impacted yesterday's release. Treescript didn't manage to bump version 108.0.0 to 108.0.1[1]:

2022-11-14 20:36:59,669 - treescript.versionmanip - INFO - Reading /app/workdir/src/version.txt for version information.
2022-11-14 20:36:59,669 - treescript.versionmanip - INFO - Could not determine version class based on file path. Falling back to source_repo
2022-11-14 20:36:59,669 - treescript.versionmanip - INFO - Contents:
2022-11-14 20:36:59,669 - treescript.versionmanip - INFO -  108.0.0
Traceback (most recent call last):
  File "/app/bin/treescript", line 8, in <module>
    sys.exit(main())
  File "/app/lib/python3.9/site-packages/treescript/script.py", line 132, in main
    return sync_main(async_main, default_config=get_default_config())
  File "/app/lib/python3.9/site-packages/scriptworker_client/client.py", line 127, in sync_main
    loop.run_until_complete(_handle_asyncio_loop(async_main, config, task))
  File "/usr/local/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
    return future.result()
  File "/app/lib/python3.9/site-packages/scriptworker_client/client.py", line 182, in _handle_asyncio_loop
    await async_main(config, task)
  File "/app/lib/python3.9/site-packages/treescript/script.py", line 105, in async_main
    await retry_async(do_actions, args=(config, task, actions_to_perform, repo_path), retry_exceptions=(CheckoutError, PushError))
  File "/app/lib/python3.9/site-packages/scriptworker_client/aio.py", line 322, in retry_async
    return await func(*args, **kwargs)
  File "/app/lib/python3.9/site-packages/treescript/script.py", line 75, in do_actions
    num_changes += await bump_version(config, task, repo_path, repo_type)
  File "/app/lib/python3.9/site-packages/treescript/versionmanip.py", line 101, in bump_version
    changed = await do_bump_version(config, repo_path, bump_info["files"], bump_info["next_version"], source_repo)
  File "/app/lib/python3.9/site-packages/treescript/versionmanip.py", line 145, in do_bump_version
    curr_version = get_version(file_, repo_path, source_repo)
  File "/app/lib/python3.9/site-packages/treescript/versionmanip.py", line 78, in get_version
    return VersionClass.parse(lines[-1])
  File "/app/lib/python3.9/site-packages/mozilla_version/mobile.py", line 114, in parse
    mobile_version = super(MobileVersion, cls).parse(
  File "/app/lib/python3.9/site-packages/mozilla_version/version.py", line 56, in parse
    return cls(**kwargs)
  File "<attrs generated init mozilla_version.mobile.MobileVersion>", line 11, in __init__
  File "/app/lib/python3.9/site-packages/mozilla_version/mobile.py", line 109, in __attrs_post_init__
    raise PatternNotMatchedError(self, patterns=error_messages)
  File "/app/lib/python3.9/site-packages/mozilla_version/errors.py", line 18, in __init__
    message = '"{}" does not match the pattern: {}'.format(string, patterns[0])
  File "/app/lib/python3.9/site-packages/mozilla_version/mobile.py", line 163, in __str__
    string = str(GeckoVersion(
  File "<attrs generated init mozilla_version.gecko.GeckoVersion>", line 14, in __init__
  File "/app/lib/python3.9/site-packages/mozilla_version/gecko.py", line 223, in __attrs_post_init__
    raise PatternNotMatchedError(self, patterns=error_messages)
mozilla_version.errors.PatternNotMatchedError: "108.0.0" does not match the pattern: Minor number and patch number cannot be both equal to 0
exit code: 1

I think we should stop using X.0.0 version numbers in A-C. X.0 numbers remain valid in the maven realm[2]:

• version if you distribute it, then you can choose any typical version with numbers and dots (1.0, 1.1, 1.0.1, ...).

[1] https://firefox-ci-tc.services.mozilla.com/tasks/eI-rwmP5TZ-KX8CCZkvTYw/runs/1/logs/public/logs/live_backing.log
[2] https://maven.apache.org/guides/mini/guide-naming-conventions.html

Comment 1

[BMO Automation]

Attachment: [mozilla-mobile/firefox-android] Bug 1800611 - Use gecko version numbers (#214)

Comment 2

[Johan Lorenzo [:jlorenzo]]

Attachment: [mozilla-releng/mozilla-version] Allow mobile version numbers to be X.0.0 up to v108

Comment 3

[Johan Lorenzo [:jlorenzo]]

Attachment: [mozilla-releng/shipit] Use Mobile version numbers for Android-Components

Comment 4

[Johan Lorenzo [:jlorenzo]]

Attachment: [mozilla-releng/scriptworker-scripts] Support Gecko version numbers starting Android-Components v109

Comment 5

[Johan Lorenzo [:jlorenzo]]

Attachment: [mozilla-mobile/relbot] Let Android-Components version numbers use the Gecko scheme

Comment 6

[Johan Lorenzo [:jlorenzo]]

All pieces have landed!

Comment 7

[Johan Lorenzo [:jlorenzo]]

Reopening issue, we missed several spots.

Comment 8

[Johan Lorenzo [:jlorenzo]]

Attachment: [mozilla-mobile/firefox-android] Bump the versions for the Sync Strings GitHub actions

Comment 9

[BMO Automation]

Attachment: [mozilla-releng/maven-lambda] Bug 1800611 - Support Gecko version pattern for Android-Components >= 109 (#31)

Comment 10

[Johan Lorenzo [:jlorenzo]]

Hi :jbuck! Based on this mana page[1], the Services/Purple SRE team is in change of AWS. I see you took care of a similar request in bug 1752809. Before then[2], the process was ad-hoc too.

Could you update the AWS Lambda function in charge of updating the Maven metadata in (nightly.)maven.mozilla.org? The new function is available here[3].

Thanks in advance!

[1] https://mozilla-hub.atlassian.net/wiki/spaces/SRE/pages/27919434/Contacting+Services+SRE#ContactingServicesSRE-CloudOpsInfraOwners/Admins
[2] E.g.: bug 1589065 comment 27
[3] https://community-tc.services.mozilla.com/api/queue/v1/task/UgmdKgM5SWi7VgZElFKdVQ/runs/0/artifacts/public%2Fbuild%2Ffunction.zip

Comment 11

[Ryan VanderMeulen [:RyanVM]]

Tom, I believe that jbuck isn't available to reply to this request at the moment? This is blocking our ability to update Android Components in Fenix, which in turns is blocking our release pipeline. Can you please redirect this request to someone who can help?

Comment 12

[Dustin Lactin [:dlactin]]

I ran this deploy in stage:

CLOUDOPS_INFRA_ROOT=../.. ./tasks deploy-maven-lambda --realm nonprod --env stage bug-1800611 https://community-tc.services.mozilla.com/api/queue/v1/task/UgmdKgM5SWi7VgZElFKdVQ/runs/0/artifacts/public%2Fbuild%2Ffunction.zip

Can you test this in stage before we deploy to prod?

Comment 13

[Johan Lorenzo [:jlorenzo]]

Thank you for the deployment, :dlactin!

I tested the updated lamdba function on staging but it run as expected. We don't see the new beta versions here[1] for example. :jcristau and I wanted to have a look at the logs based on the information we have. However, I don't have an account anymore (I used to in bug 1600995, but I left then rejoined Mozilla). :jcristau has an account but we were unable to find the location of the stage log groups. We've only found the production ones[2]. Bug 1470937 comment 9 talks about the staging bucket but no logs groups.

:dlactin, what are the stage logs groups?

Clearing NI to Tom Healy because we got some help from Dustin 🙂

[1] https://maven-default.stage.mozaws.net/maven2/org/mozilla/components/browser-engine-gecko/maven-metadata.xml
[2] https://github.com/mozilla-services/cloudops-infra/blob/3b970fe8ce954375dd9de71ea0102a49bb67d1c0/projects/maven/README.md?plain=1#L26-L29

Comment 14

[Jon Buckley [:jbuck]]

I found the following error in the logs:

[ERROR] AttributeError: module 'copy' has no attribute 'deepcopy'
Traceback (most recent call last):
  File "/var/lang/lib/python3.7/imp.py", line 234, in load_module
    return load_source(name, filename, file)
  File "/var/lang/lib/python3.7/imp.py", line 171, in load_source
    module = _load(spec)
  File "<frozen importlib._bootstrap>", line 696, in _load
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 728, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/var/task/copy.py", line 1, in <module>
    import boto3
  File "/var/task/boto3/__init__.py", line 16, in <module>
    from boto3.session import Session
  File "/var/task/boto3/session.py", line 17, in <module>
    import botocore.session
  File "/var/task/botocore/session.py", line 28, in <module>
    import botocore.configloader
  File "/var/task/botocore/configloader.py", line 19, in <module>
    from botocore.compat import six
  File "/var/task/botocore/compat.py", line 148, in <module>
    import xml.etree.cElementTree
  File "/var/lang/lib/python3.7/xml/etree/cElementTree.py", line 3, in <module>
    from xml.etree.ElementTree import *
  File "/var/lang/lib/python3.7/xml/etree/ElementTree.py", line 1660, in <module>
    from _elementtree import *

:jlorenzo noted that Python 3.9 was now required. I click-ops this change and got a new, slightly different error:

[ERROR] AttributeError: partially initialized module 'copy' has no attribute 'deepcopy' (most likely due to a circular import)
Traceback (most recent call last):
  File "/var/lang/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/var/task/copy.py", line 1, in <module>
    import boto3
  File "/var/task/boto3/__init__.py", line 16, in <module>
    from boto3.session import Session
  File "/var/task/boto3/session.py", line 17, in <module>
    import botocore.session
  File "/var/task/botocore/session.py", line 28, in <module>
    import botocore.configloader
  File "/var/task/botocore/configloader.py", line 19, in <module>
    from botocore.compat import six
  File "/var/task/botocore/compat.py", line 148, in <module>
    import xml.etree.cElementTree
  File "/var/lang/lib/python3.9/xml/etree/cElementTree.py", line 3, in <module>
    from xml.etree.ElementTree import *
  File "/var/lang/lib/python3.9/xml/etree/ElementTree.py", line 2082, in <module>
    from _elementtree import *

Still trying to figure out what's going on here

Comment 15

[BMO Automation]

Attachment: [mozilla-releng/maven-lambda] Bug 1800611 - part 4: Create package in order not to override the st… (#32)

Comment 16

[Johan Lorenzo [:jlorenzo]]

Attachment: [mozilla-services/cloudops-infra] Bug 1800611 - Upgrade maven.m.o lambda functions (#4588)

Comment 17

[Johan Lorenzo [:jlorenzo]]

:jcristau found the root cause of this bug. It's actually a change that was introduced in Python 3.7[1] and backported to Python 3.6.3+[2]. In [3], I bumped the Python code to Python 3.9. :jbuck, does this new function[4] make things better?

[1] https://github.com/python/cpython/pull/2647
[2] https://github.com/python/cpython/pull/2649
[3] https://github.com/mozilla-releng/maven-lambda/pull/31/commits/710375383ca946124e8b7cec2d0ee3fdc4cff830
[4] https://community-tc.services.mozilla.com/api/queue/v1/task/Yujo6EMcQrOUzekVlFmeDA/runs/0/artifacts/public%2Fbuild%2Ffunction.zip

Comment 18

[Jon Buckley [:jbuck]]

After checking out this PR I deployed this to stage with CLOUDOPS_INFRA_ROOT=../.. ./tasks deploy-maven-lambda --realm nonprod --env stage bug-1800611 https://community-tc.services.mozilla.com/api/queue/v1/task/Yujo6EMcQrOUzekVlFmeDA/runs/0/artifacts/public%2Fbuild%2Ffunction.zip

Comment 19

[Jon Buckley [:jbuck]]

:jlorenzo and I found that if you're updating the code, the app_version always needs to change as well, so running CLOUDOPS_INFRA_ROOT=../.. ./tasks deploy-maven-lambda --realm nonprod --env stage bug-1800611-2 https://community-tc.services.mozilla.com/api/queue/v1/task/Yujo6EMcQrOUzekVlFmeDA/runs/0/artifacts/public%2Fbuild%2Ffunction.zip (notice the -2) actually updated the code properly

Comment 20

[Jon Buckley [:jbuck]]

Alright, after landing :jlorenzo's PR I ran the following commands to update stage & prod:

CLOUDOPS_INFRA_ROOT=../.. ./tasks deploy-maven-lambda --realm nonprod --env stage bug-1800611-3 https://community-tc.services.mozilla.com/api/queue/v1/task/GFypnhmWR-W8elPTFGfgOg/runs/0/artifacts/public%2Fbuild%2Ffunction.zip

CLOUDOPS_INFRA_ROOT=../.. ./tasks deploy-maven-lambda --realm prod --env prod bug-1800611-3 https://community-tc.services.mozilla.com/api/queue/v1/task/GFypnhmWR-W8elPTFGfgOg/runs/0/artifacts/public%2Fbuild%2Ffunction.zip

Comment 21

[Johan Lorenzo [:jlorenzo]]

Thank you very much for your help, :jbuck! I confirm the fix worked in both staging and production. maven.mozilla.org now displays all version numbers including the beta ones. This fixed the Update A-C Action on Fenix[1]:

2022-12-21 20:16:03,532 - util.update_android_components_release:510 - INFO - Current A-C version in fenix is 109.0b2
2022-12-21 20:16:03,680 - util.update_android_components_release:514 - INFO - Latest A-C version available is 109.0b3
2022-12-21 20:16:03,680 - util.update_android_components_release:524 - INFO - We are going to upgrade fenix 109 to Android-Components 109.0b3
2022-12-21 20:16:03,812 - util.update_android_components_release:545 - INFO - Last commit on releases_v109.0.0 is 545e05f6879c053f878c8b09ef5ebe0ebb84cc01
2022-12-21 20:16:03,812 - util.update_android_components_release:547 - INFO - Creating branch relbot/fenix-109 on 545e05f6879c053f878c8b09ef5ebe0ebb84cc01
2022-12-21 20:16:04,162 - util.update_android_components_release:552 - INFO - Updating AndroidComponents.kt from 109.0b2 to 109.0b3 on relbot/fenix-109
2022-12-21 20:16:04,786 - util.update_android_components_release:565 - INFO - Creating pull request
2022-12-21 20:16:05,845 - util.update_android_components_release:573 - INFO - Pull request at https://github.com/mozilla-mobile/fenix/pull/28282

As far as I know, all issues related to the gecko version scheme used in Android-Components are fixed. Closing this bug.

[1] https://github.com/mozilla-mobile/fenix/actions/runs/3752424759/jobs/6374771131

Comment 22

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ff6d3a69925f
part 1: Bump taskgraph python dependencies
https://hg.mozilla.org/integration/autoland/rev/7e7b07370397
part 2: Bump mozilla-version to 1.2.0
https://hg.mozilla.org/integration/autoland/rev/d15dbc39e89e
part 3: Use gecko version numbers
https://hg.mozilla.org/integration/autoland/rev/33ed9bf9719a
part 4: Do not strip out tasks when using `try_task_config.json`
https://hg.mozilla.org/integration/autoland/rev/4deefba14fdb
part 5: Bump relbot to 5.0.0