1637695 - [application-services] toolchains broken when building locally

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect)

Description

[Mihai Tabara [:mtabara]⌚️GMT]

• 24 days ago: last good one 132a8a8dd6bd636d8996c01063ac8e0911e1842e is created and it’s 200 https://firefox-ci-tc.services.mozilla.com/tasks/index/project.application-services.application-services.build.libs.android/132a8a8dd6bd636d8996c01063ac8e0911e1842e
• 19 days ago taskgraph support is added which skips this publishing
• 9 days ago libs changes sha via https://github.com/mozilla/application-services/commit/88cfed943f2487d415fba2833559997c134025b1 from git rev-parse HEAD:libs goes from 132a8a8dd6bd636d8996c01063ac8e0911e1842e -> a548e4c3da083f6e7de8c20f0da7218811d7c4a0 and nothing gets published to the index
• for the past 9 days: breakage (only when building locally as CI doesn’t use the downloaded libs).

This is for all toolchains.

Suggestions:

1. We adjust taskgraph to publish to those https://firefox-ci-tc.services.mozilla.com/tasks/index/project.application-services.application-services.build.libs. indexes
2. We adjust app-services in the gradle logic from (e.g https://github.com/mozilla/application-services/blob/ad809b6514f370b2b3f8ad10790d19f01d5503eb/build.gradle#L90) to point to the https://firefox-ci-tc.services.mozilla.com/tasks/index/project.application-services.cache.level-3.toolchains.v3.android indexes instead (hence no longer computing libs hash, etc). Tom suggested we turn something like https://searchfox.org/mozilla-central/rev/9f074fab9bf905fad62e7cc32faf121195f4ba46/python/mozbuild/mozbuild/artifact_commands.py#294-335 into a taskgraph subcommand, and using that in gradle.

Comment 1

[Mihai Tabara [:mtabara]⌚️GMT]

In creating the toolchains, we do take into account libs based on https://github.com/mozilla/application-services/blob/master/taskcluster/ci/toolchain/kind.yml#L22 and some logic from https://hg.mozilla.org/ci/taskgraph/file/tip/src/taskgraph/transforms/job/toolchain.py#l61 so we publish new cached tasks should anything changes in libs or in any of the toolchain scripts.

Options:

1. We know we can rely on the https://firefox-ci-tc.services.mozilla.com/tasks/index/project.application-services.cache.level-3.toolchains.v3 indexes so would it be easier to point gradle to them? It’d be consistent with what we have in the other repos + m-c on one hand, but more work to add it.

2. The other option is to publish to the old indexes https://firefox-ci-tc.services.mozilla.com/tasks/index/project.application-services.application-services.build.libs the current toolchains. But I don’t see how that’s trivial, the list of routes is computed under the hood in taskgraph, not in the application-services’s taskcluster logic. Doable though.

Motivation to go with 1 is to be consistent with standard routes as the other repos are + mozilla-central. We should aim for 1 anyway long-term.

Comment 2

[Mihai Tabara [:mtabara]⌚️GMT]

Boils down to:

1. turn https://searchfox.org/mozilla-central/rev/9f074fab9bf905fad62e7cc32faf121195f4ba46/python/mozbuild/mozbuild/artifact_commands.py#294-335 into taskgraph subcommand that can be reused in gradle so that we point to the standard caches
2. add git rev-parse HEAD:libs logic in taskgraph and enrich the toolchains to publish to the old routes too? Not sure how this looks though, we’d have to make the changes in taskgraph itself. Or rather an additional transform that adds those routes.

Plan going forward:
2) for now to unblock. 1) for longer term once we're fixed.

Comment 3

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: [application-services] temporarily add old index to allow local building of appsv

Comment 4

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #2)

Boils down to:

1. turn https://searchfox.org/mozilla-central/rev/9f074fab9bf905fad62e7cc32faf121195f4ba46/python/mozbuild/mozbuild/artifact_commands.py#294-335 into taskgraph subcommand that can be reused in gradle so that we point to the standard caches

Bug 1563169 tracks this once we've migrated fully to taskgraph.

2. add git rev-parse HEAD:libs logic in taskgraph and enrich the toolchains to publish to the old routes too? Not sure how this looks though, we’d have to make the changes in taskgraph itself. Or rather an additional transform that adds those routes.

Plan going forward:
2) for now to unblock. 1) for longer term once we're fixed.

PR attached in this bug.

Comment 5

[Mihai Tabara [:mtabara]⌚️GMT]

:eoger adding https://github.com/mozilla/application-services/pull/3118 to force-toolchain tasks to be-rebuilt and published in the old-index too.

Comment 6

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: [application-services] fix download libs url

Comment 7

[Mihai Tabara [:mtabara]⌚️GMT]

:eoger confirmed works now:

> Task :downloadAndroidLibs
Download https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/project.application-services.application-services.build.libs.android.13f21bb7ae734bc3b0de9907df5f9df121d2d4ec/artifacts/public/build/android.tar.gz
> Task :downloadDesktopLibs
Download https://firefox-ci-tc.services.mozilla.com/api/index/v1/task/project.application-services.application-services.build.libs.desktop.macos.13f21bb7ae734bc3b0de9907df5f9df121d2d4ec/artifacts/public/build/macos.tar.gz