1763040 - Fenix and GeckoView Example nightlies should be kept around for longer than 1 year

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Botond Ballo [:botond]]

Steps to reproduce

For GeckoView Example: run mozregression -n gve --launch 2021-04-04. Observe that it fails, while mozregression -n gve --launch 2021-04-05 succeeds, suggesting that GVE nightlies are only kept for 1 year.

For Fenix: go to https://firefox-ci-tc.services.mozilla.com/tasks/index/mobile.v2.fenix.nightly.2021. Observe that the builds only go back as far as 2021-04-04, i.e. one year ago.

Expected results

Builds are retained for a longer period to facilitate finding regression windows.

For example, I'm currently chasing a regression that occurred sometime between 2020-09-20 and 2021-01-22, and it's unfortunate that we don't seem to have nightly builds for either Fenix or GVE in that range.

Comment 1

[Chris Peterson [:cpeterson]]

Moving to the Release Engineering component because this sounds like a Relng policy change.

We archive desktop Nightlies are forever, all the way back to 2004.

Comment 2

[Aki Sasaki (not active)]

Taskcluster artifacts for nightly tasks will only be kept for a maximum of 1 year; this is true for Firefox Desktop as well as Fenix.

We don't have a way of recovering the expired taskcluster artifacts, but we now beetmove Fenix nightlies to https://archive.mozilla.org/pub/fenix/nightly/, as of December 2021; these files will be archived forever.
That work was done in RELENG-489.

Comment 3

[Chris Peterson [:cpeterson]]

(In reply to Aki Sasaki [:aki] (he/him) (UTC-6) from comment #2)

Taskcluster artifacts for nightly tasks will only be kept for a maximum of 1 year; this is true for Firefox Desktop as well as Fenix.

We don't have a way of recovering the expired taskcluster artifacts, but we now beetmove Fenix nightlies to https://archive.mozilla.org/pub/fenix/nightly/, as of December 2021; these files will be archived forever.
That work was done in RELENG-489.

Zeid, does mozregression need to be updated to find the Fenix nightlies that are now beetmoved to archive.mozilla.org?

I understand that old Fenix nightlies on Taskcluster older than 1 year are lost, e.g. https://firefox-ci-tc.services.mozilla.com/tasks/index/mobile.v2.fenix.nightly.2021. I just want to make sure that mozregression is downloading from the right server going forward.

Comment 4

[Zeid [:zeid]]

There are multiple ways artifacts are fetched currently, depending on the application name / launcher that's selected. For GeckoView Example (and Fennec†), those are fetched exclusively from taskcluster artifacts. For Firefox and others, this is done via the archives. There doesn't appear to be any fallback mechanisms currently implemented.

It is possible to add support for using the archives. If it's possible to clone all the artifacts going back to April* into the archives, then we wouldn't need to introduce any new code to support this awkward period between April 2021* and December 2021.

† it's unclear whether anyone is using this option (-n fennec) at this time, but there is no fenix option currently supported.
* replace April with whatever arbitrary month we decide to go with depending on when or if we end up cloning those artifacts.

Zeid, does mozregression need to be updated to find the Fenix nightlies that are now beetmoved to archive.mozilla.org?

Short answer is yes.

Comment 5

[Botond Ballo [:botond]]

(In reply to Aki Sasaki [:aki] (he/him) (UTC-6) from comment #2)

We don't have a way of recovering the expired taskcluster artifacts, but we now beetmove Fenix nightlies to https://archive.mozilla.org/pub/fenix/nightly/, as of December 2021; these files will be archived forever.
That work was done in RELENG-489.

Is there some way to preserve nightlies that are currently not yet expired (e.g., as of today, from 2021-06-25 until the mentioned December 2021 date)? It seems unfortunate to me to let those expire.

Comment 6

[Chris Peterson [:cpeterson]]

Is there some way to preserve nightlies that are currently not yet expired (e.g., as of today, from 2021-06-25 until the mentioned December 2021 date)? It seems unfortunate to me to let those expire.

Yes, copying the remaining Fenix Nightly builds from https://firefox-ci-tc.services.mozilla.com/tasks/index/mobile.v2.fenix.nightly.2021 to https://archive.mozilla.org/pub/fenix/nightly/2021/ would be great.

https://archive.mozilla.org/pub/fenix/nightly/ has Fenix Nightly builds going back to 2021-12-21, but https://firefox-ci-tc.services.mozilla.com/tasks/index/mobile.v2.fenix.nightly.2021 has builds going back to 2021-06-23 (though we're losing another build from 2021 every day).

Comment 7

[Johan Lorenzo [:jlorenzo]]

Hi there! I remember we transferred some builds from Taskcluster to archive.m.o, in the past. That said, it was a manual process. Releng can likely make the script to get all nightly builds in the past year. However, we would need CloudOps to get them uploaded. :aki, who would be the right contact, in your opinion?

Comment 8

[Aki Sasaki (not active)]

:cpeterson, how valuable are these nightlies? If these take a week of Releng work to move, will they save developers more than a week of work?

:jlorenzo, we can probably use a script like https://github.com/escapewindow/beetmove-telemetry/tree/apidoc, though that moves a single set of manually downloaded builds to a location. We'd want all the nightlies for six months beetmoved, so we'd probably want to also write logic to download all the bits, ideally verify their shas, then beetmove them to archive.m.o. Releng can look at k8s-sops to get the aws creds to move these ourselves if needed.

Comment 9

[Chris Peterson [:cpeterson]]

(In reply to Aki Sasaki [:aki] (he/him) (UTC-6) from comment #8)

:cpeterson, how valuable are these nightlies? If these take a week of Releng work to move, will they save developers more than a week of work?

No. I don't think preserving six months of Fenix nightlies from 2021 is worth the Releng team investing more than one day of work (let alone an entire week of work).

Comment 10

[Botond Ballo [:botond]]

This bug is marked as fixed, and its title is "Fenix and GeckoView Example nightlies should be kept around for longer than 1 year".

I understand from comment 2 that Fenix nightlies have been "beetmoved" to archive.m.o since December 2021, and are available at https://archive.mozilla.org/pub/fenix/nightly/.

Has something similar been implemented for GeckoView Example nightlies, and if so where can I find those?

Comment 11

[Johan Lorenzo [:jlorenzo]]

Hi :botond! I just checked and I think you're right: even though the title mentioned GeckoView Example, we've never done it. For instance, this revision "beetmoved"[2] desktop and geckoview but not GeckoView Example. Unlike Fenix, I don't think we've ever made any nightly builds for GeckoView Example so we should start with that.

That being said, the Android teams and Release Engineering are migrating Fenix to the gecko repos (bug 1822248). We're currently aiming to migrate in January 2024. Just to understand the importance of GeckoView Example: are there any bugs we will find in GeckoView Example but not Fenix?

[1] https://treeherder.mozilla.org/jobs?repo=mozilla-central&revision=489e7952cd67fe4e1816d3452675332e76676a3e&searchStr=beetmove
[2] Beetmover is RelEng's Taskcluster worker in charge of moving bits ("beets") to archive.mozilla.org.

Comment 12

[Botond Ballo [:botond]]

(In reply to Johan Lorenzo [:jlorenzo] from comment #11)

Just to understand the importance of GeckoView Example: are there any bugs we will find in GeckoView Example but not Fenix?

Good question!

I'm not aware of any relevant functionality that GVE has but Fenix doesn't. Rather, my interest in GVE stems mostly from things I can do with GVE in mozregression that I can't do with Fenix.

Until recently, mozregression didn't support Fenix at all; that was added very recently in bug 1556042. However, a remaining gap is that I can't get a mozilla-central pushlog from a Fenix regression range, while I can get one from a GVE regression range; I filed bug 1857026 about this. If that's fixed then I think all my use cases for GVE nightlies are superseded by using Fenix nightlies and I'm happy to retract the request to retain GVE nightlies longer.

Comment 13

[Johan Lorenzo [:jlorenzo]]

Thanks for the detailed answer! Yeah, when Fenix is migrated to mozilla-central, then we'll be able to get an m-c pushlog. Please don't hesitate to NI me if that happens to be not the case 🙂

Comment 14

[Botond Ballo [:botond]]

(In reply to Johan Lorenzo [:jlorenzo] from comment #13)

when Fenix is migrated to mozilla-central, then we'll be able to get an m-c pushlog

To clarify, I'd like to be able to get m-c pushlogs for regression windows involving Fenix nightlies in the date range from December 2021 (when they started to be retained on archive.m.o) to January 2024 (the expected date of the Fenix migration into m-c) as well. I think this should be possible, as there should be a record somewhere of what m-c revision a Fenix nightly was built from.