1615248 - [android-components] add independent nightly releases to compete with snapshots

Status: RESOLVED FIXED

(Release Engineering :: General, task)

Description

[Mihai Tabara [:mtabara]⌚️GMT]

We're currently investigating whether we can switch off AC snapshots and instead built nightlies (which are regular Github releases happening twice a day, with a "buildid" instead of "snapshot").

This should help with reproducible builds and eliminating the caching issues as the current snapshots overlap in the latest locations in maven.mozilla.org.

Comment 1

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: [android-components] add support for nightly-releases

Comment 2

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: [android-components] add missing .cron.yml entry

Comment 3

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Bug 1615248 - Add initial resources for AC nightly. r=jlorenzo

Comment 4

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: Bug 1615248 - Add initial resources for AC nightly. r=jlorenzo

Comment 5

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: [android-components] fixes build paths and adds signing

Comment 6

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: [beetmoverscript] add support for nightly MMA

Comment 7

[Mihai Tabara [:mtabara]⌚️GMT]

I have updated SOPS as well with the credentials.

Comment 8

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: [cloudops-infra] add beetmover variables for AC nightly maven

Comment 9

[Mihai Tabara [:mtabara]⌚️GMT]

Nightly graphs successfully pushed to the S3 upload buckets - https://firefox-ci-tc.services.mozilla.com/tasks/groups/S_NqnH8ZRky3MY7ucv3n_g

So we upload correctly, but we have some issues with the lamda workers not replicating the data or the serving bucket being misconfigured. Either way, we're very close to victory.

Comment 10

[Mihai Tabara [:mtabara]⌚️GMT]

Aaaaand ... we're live: https://nightly.maven.mozilla.org/?prefix=maven2/org/mozilla/components/

Comment 11

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: [android-components] same .cron.yml cadence for nightlies as for snapshots

Comment 12

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: [android-components] nightly graphs version override

Comment 13

[Mihai Tabara [:mtabara]⌚️GMT]

Attachment: [android-components] moar fixes nightly graphs version override

Comment 14

[Mihai Tabara [:mtabara]⌚️GMT]

https://firefox-ci-tc.services.mozilla.com/tasks/groups/H8ln-zW0T8-nL07WZ4iItg looks good, with the pom files pointing at the right versions finally.

Comment 15

[Mihai Tabara [:mtabara]⌚️GMT]

Nightlies looking good but we now have a problem with lambda workers. E.g. https://nightly.maven.mozilla.org/maven2/org/mozilla/components/browser-session/maven-metadata.xml was giving different results to me vs Sebastian.

Plan of action:

1. nuke everything up in the nightly MMO buckets + clear the caches
2. push a fresh new AC nightly
3. check the logs for lambda workers and spot the problem
4. fix and deploy the lambda workers
5. celebrate

Comment 16

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #15)

Nightlies looking good but we now have a problem with lambda workers. E.g. https://nightly.maven.mozilla.org/maven2/org/mozilla/components/browser-session/maven-metadata.xml was giving different results to me vs Sebastian.

Plan of action:

1. nuke everything up in the nightly MMO buckets + clear the caches

DONE

2. push a fresh new AC nightly

DONE - latest release is https://firefox-ci-tc.services.mozilla.com/tasks/groups/OZWTUfrITIaANMK3XqgUuQ
published under nightly.maven.mozilla.org

on to debug lambda workers.

Comment 17

[Mihai Tabara [:mtabara]⌚️GMT]

After the big nuke, things seem to be fine. I double-check with latest nightly earlier today, the ones at 13 UTC.
I think we can declare victory here.

:Sebastian and Johan have a plan to gradually roll this out - they are pinning the nightly version RB first and then consequently in Fenix in bug 1620190 via automation.

Let's declare victory here and re-surface if issues come up along the way.

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c74cc9d9e542
[components] Bug 1615248 - add missing nightly entry in .cron.yml
https://hg.mozilla.org/integration/autoland/rev/0fc427303a00
[components] Bug 1615248 - fix paths for components.
https://hg.mozilla.org/integration/autoland/rev/a0a96e3351df
[components] Bug 1615248 - add beetmover tasks to nightly graphs
https://hg.mozilla.org/integration/autoland/rev/3fffaf5ffa0d
[components] Bug 1615248 - nightly graphs version override.
https://hg.mozilla.org/integration/autoland/rev/5d79bb3cbe66
[components] Bug 1615248 - fix build paths in cron-nightly.
https://hg.mozilla.org/integration/autoland/rev/0f8f98c85466
[components] Bug 1615248 - add same cron entries for nightlies as we do for snapshots.