1876704 - Distribute geckodriver releases in Mozilla's apt repository

Status: NEW

(Testing :: geckodriver, task, P2)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Originally filed as: https://github.com/mozilla/geckodriver/issues/2155

Right now we ship geckodriver releases from GitHub, crates.io, and as a snap package (bundled with Firefox). It would be helpful for users of geckodriver to also have it shipped via our apt repository. But I don't know what's required to get this task done.

Gabriel given that you wrote the post on the Nightly blog maybe you could help us or forward to the responsible person? Thanks!

Comment 1

[Gabriel Bustamante [:gabriel]]

Hi :whimboo,

I was browsing the geckodriver docs and they say that development happens in mozilla-central, then that geckodriver repository on Github is updated and a Github release is created manually with artifacts built on mozilla-central.

Making future releases from Mozilla’s CI infrastructure seems like a logical step, given that the development and artifact creation is already happening on mozilla-central.

To ship geckodriver to the Mozilla APT repository, we could repackage the geckodriver.tar.gz artifact on the gd-s tasks running on mozilla-central to build .deb packages. Then, we can upload these packages to the repository by running tasks similar to the ones that upload our Firefox .deb packages.

Could geckodriver ride the Firefox release train? Or maybe we could create a cron job that triggers the release flow?

Either of these would require automating the other release steps (so they execute in tandem with the APT upload.)

Creating a Github release and making changes to a external repositories from our CI infrastructure seems possible. IIRC we create Github releases automatically for Firefox on Android (but that project's development happens on Github), and I found CI tasks that sync an external Github repositories here: https://searchfox.org/mozilla-central/source/taskcluster/ci/github-sync/kind.yml

Comment 2

[Gabriel Bustamante [:gabriel]]

Mm, maybe those release artifacts could be hosted in the Mozilla FTP archive - like this https://ftp.mozilla.org/pub/firefox/releases/122.0/linux-x86_64/en-US/ - Do we need that external repository?

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Gabriel Bustamante [:gabriel] from comment #1)

Making future releases from Mozilla’s CI infrastructure seems like a logical step, given that the development and artifact creation is already happening on mozilla-central.

Thanks for your reply Gabriel! So it all looks like that work is needed first to get in-house releases done. As such it looks like that it might depend on bug 1372587. Sadly a release is not easy and a reason why we haven't had the time to actually do that work yet. :/

The current release process can be found here: firefox-source-docs.mozilla.org/testing/geckodriver/Releasing.html

Hereby please note that the final geckodriver binaries are not taken from the actual commit when the changes landed on autoland or mozilla-central but the central merge commit (see bug 1824713 comment 13 and following). It's needed to after the fact update the Changes.md file with the correct revision and date.

To ship geckodriver to the Mozilla APT repository, we could repackage the geckodriver.tar.gz artifact on the gd-s tasks running on mozilla-central to build .deb packages. Then, we can upload these packages to the repository by running tasks similar to the ones that upload our Firefox .deb packages.

Could geckodriver ride the Firefox release train? Or maybe we could create a cron job that triggers the release flow?

For us it will be important that we are not bound to Firefox release cycles. Compared to Chrome and chromedriver, which require a version match for each release, geckodriver can be used with a variety of Firefox releases at maximum back to the latest ESR release. If there are issues that need to be fixed and released quickly we shouldn't be blocked on getting beta/release approval to uplift patches.

As such a cron job might be a good idea, which then would pick the binaries from a given changeset of mozilla-central and makes them publicly available while keeping the artifacts forever.

Either of these would require automating the other release steps (so they execute in tandem with the APT upload.)

Creating a Github release and making changes to a external repositories from our CI infrastructure seems possible. IIRC we create Github releases automatically for Firefox on Android (but that project's development happens on Github), and I found CI tasks that sync an external Github repositories here: https://searchfox.org/mozilla-central/source/taskcluster/ci/github-sync/kind.yml

If I'm reading this right the GitHub sync here happens from mozilla-central to GitHub and not the other way around, right? Nevertheless to keep external users happy we might have to keep GitHub releases for a while until there is a way to also have a download page with release notes and the possibility to query for the latest or individual geckodriver releases + getting the right artifact for download.

(In reply to Gabriel Bustamante [:gabriel] from comment #2)

Mm, maybe those release artifacts could be hosted in the Mozilla FTP archive - like this https://ftp.mozilla.org/pub/firefox/releases/122.0/linux-x86_64/en-US/ - Do we need that external repository?

Given that we do not have a strict relation to Firefox versions (as mentioned above) so for releases a folder under https://ftp.mozilla.org/pub/geckodriver/ may be better.

Comment 4

[Gabriel Bustamante [:gabriel]]

The release process does not sound easy.

Actually, a cron job might not work here. I think we'd want the ability to choose the revision that is being shipped.

Could we run the project's build and release automation on the Github repository using Taskcluster? Like application-services?

We could reach out to SRE to upload a .deb from an index while we get to a better state, but we'd still need a task that does the repackage.

Agreed, the path ftp.mozilla.org/pub/geckodriver/ is better (since the project is versioning is not tied to the Firefox release cycle.)

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Gabriel Bustamante [:gabriel] from comment #4)

Could we run the project's build and release automation on the Github repository using Taskcluster? Like application-services?

We only sync with GitHub when a geckodriver release is due. Any other work is really happening in mozilla-central and that's not going to change. The ultimate goal is to actually get geckodriver released directly from mozilla-central (bug 1372587).

So it's not clear for us what might be the best way forward here and if we may have to event depend on bug 1372587.

If there is any simple kind of workaround for the time being we would be happy to know.

Comment 6

[Gabriel Bustamante [:gabriel]]

Any other work is really happening in mozilla-central and that's not going to change.

I see. In that case, I would recommend creating a custom push action to ship geckodriver from central pushes using treeherder (and a on-push task that creates a .deb package for the signed geckodriver build.) That custom push action would take the version as an input, and create the tasks to ship geckodriver. Release engineering can grant permissions for users to run this action from treeherder on central pushes.

Comment 7

[Gabriel Bustamante [:gabriel]]

Attachment: Screenshot 2024-02-06 at 08.46.35.png

Comment 8

[Gabriel Bustamante [:gabriel]]

Attachment: Screenshot 2024-02-06 at 08.47.24.png

Comment 9

[Gabriel Bustamante [:gabriel]]

Attachment: Screenshot 2024-02-06 at 08.47.37.png

This kind of thing.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Gabriel Bustamante [:gabriel] from comment #6)

I see. In that case, I would recommend creating a custom push action to ship geckodriver from central pushes using treeherder (and a on-push task that creates a .deb package for the signed geckodriver build.) That custom push action would take the version as an input, and create the tasks to ship geckodriver. Release engineering can grant permissions for users to run this action from treeherder on central pushes.

That is interesting! So it means that we could also upload all the binaries to archive.mozilla.org as well under the above mentioned location, and that these artifacts will be available forever? Should this first step maybe done via bug 1372587, and then we can follow-up with this bug?

I still don't think that we can have a full release automation given that for a new release a lot of things need to run. For that I might propose a special mach command in mozilla-central to prepare patches for geckodriver, get those landed and once we have a Nightly version that we can manually trigger the on-push task?

Comment 11

[Gabriel Bustamante [:gabriel]]

That is interesting! So it means that we could also upload all the binaries to archive.mozilla.org as well under the above mentioned location, and that these artifacts will be available forever? Should this first step maybe done via bug 1372587, and then we can follow-up with this bug?

Uploading to https://packages.mozilla.org/apt requires the .deb to be archived first. The package can then be imported from the archive to the repository. It will be available in the archive indefinitely. But, we delete packages that are a lot older than latest from the repository.

I still don't think that we can have a full release automation given that for a new release a lot of things need to run. For that I might propose a special mach command in mozilla-central to prepare patches for geckodriver, get those landed and once we have a Nightly version that we can manually trigger the on-push task?

Adding automation that prepares central for a geckodriver release is a good idea. Note, I don't think it blocks creating a custom push action. The on-push action can be triggered manually on the correct central revision when things are ready.

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Gabriel Bustamante [:gabriel] from comment #11)

Adding automation that prepares central for a geckodriver release is a good idea. Note, I don't think it blocks creating a custom push action. The on-push action can be triggered manually on the correct central revision when things are ready.

Just to wrap up given that it's not 100% clear to me. The following is what you are proposing:

1. We manually run our steps to prepare a new geckodriver release
2. When the geckodriver binaries are available on a given mozilla-central merge commit on Treeherder we need to define a push action that copies the geckodriver archives to archive.mozilla.org/pub/geckodriver/0.34.0/
3. When packages are available on archive.mozilla.org we need another push action which will take the Linux specific archives, convert them into .deb files and uploads them to packages.mozilla.org

If that sounds right I could file two bugs, one for each of the steps 2 and 3. Thanks!

Comment 13

[Gabriel Bustamante [:gabriel]]

Yeah. That would work.

With a on-push action we can add tasks to the revision on central that produces the shippable binaries, that is the builds that end up being uploaded to github, adhoc.

For the archival path I'd suggest:

archive.mozilla.org/pub/geckodriver/releases/<version>/<linux-x86_64|linux-i686|linux-aarch64>/geckodriver-v<version>-<linux-x86_64|linux-i686|linux-aarch64>.deb

The actions take in an input with a predefined schema, so we don't need two (they are pretty flexible.)
I think uploading to the archive and import into the apt repository could be done in succession and don't need to be separate steps.

I was browsing the tasks again, and iiuc, and just to restate it, what we need to do is:

1. Add a push action that can be triggered via firefox-ci/treeherder to add geckodriver release tasks to the revision we want to ship
2. That action adds tasks to the revision that
   a. Package geckodriver.tar.gz on geckodriver-signing into .deb packages ie. geckodriver.deb
   b. Upload the geckodriver.deb packages to the FTP archive.
   c. Import the geckodriver.deb packages to the apt repository.

While we are at it we could add tasks that upload the signed geckodriver.tar.gz build, the .asc file, and the CHECKSUMS to the archive.
We could also upload the builds for the other platforms to the archive. But maybe those are different issues.