1694127 - Change recommended version of Firefox for Puppeteer from Nightly to Beta/Release

Status: NEW

(Remote Protocol :: Agent, task, P3)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Originally filed as: https://github.com/puppeteer/puppeteer/issues/6894

We want to switch the recommended version of Firefox for Puppeteer from Firefox Nightly to Firefox Beta or even a release build. But that means we have to find a way how to get those builds installed. For them we do not offer zip archives as stated in bug 308048.

We should figure out how to get the .exe or .msi installed via node.js / npm, or maybe use the latest available release candidate build?

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Quick update here. I haven't gotten a reply on https://github.com/puppeteer/puppeteer/issues/6894 yet, so I asked again. Jan seems to be fine with the proposal to use the silent MSI installer, but we are still waiting for the reply from Mathias.

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+2]]

So the only option for zip archives will actually be the usage of the final release candidate build. But not sure if that is what we actually want to do. We should have a team consensus on it, and as such adding it for the next triage session.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+2]]

We discussed that yesterday in the WebDriver meeting. Generally we are all fine with using the zip files as long as we can be sure that the latest candidate build for a given Firefox (beta) release matches exactly the officially released binaries. And that seems to be the case.

But there was one concern raised by James we are not sure about how to handle it. Given that users of Puppeteer run their tests in CI, and based on our experience with Selenium/geckodriver, these CI projects are usually badly setup, which means that each of these running test jobs might pull and install the Firefox binary. As what I can see candidate builds are only available through archive.mozilla.org, and I assume that this is not backed by CDN? That means it would add a certain amount of load on this server, which might not be wanted.

Ben, could you help us with that? Do you have a proposal? If not who else could help? Basically we would only need a zip archive per platform for the en-US locale. Thanks!

Comment 4

[[e-mail] (:bhearsum)]

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+1] from comment #3)

We discussed that yesterday in the WebDriver meeting. Generally we are all fine with using the zip files as long as we can be sure that the latest candidate build for a given Firefox (beta) release matches exactly the officially released binaries. And that seems to be the case.

But there was one concern raised by James we are not sure about how to handle it. Given that users of Puppeteer run their tests in CI, and based on our experience with Selenium/geckodriver, these CI projects are usually badly setup, which means that each of these running test jobs might pull and install the Firefox binary. As what I can see candidate builds are only available through archive.mozilla.org, and I assume that this is not backed by CDN? That means it would add a certain amount of load on this server, which might not be wanted.

archive.mozilla.org is an S3 bucket fronted by cloudfront - it's fine to throw any amount of load at it.

Ben, could you help us with that? Do you have a proposal? If not who else could help? Basically we would only need a zip archive per platform for the en-US locale. Thanks!

I'm not clear on what exactly is needed. It sounds like "Firefox for Puppeteer" might be a new thing that we're publishing for Nightly already, and we're not sure where to host it for beta/release?

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to [e-mail] (:bhearsum) from comment #4)

archive.mozilla.org is an S3 bucket fronted by cloudfront - it's fine to throw any amount of load at it.

This is great to hear! Thanks.

Ben, could you help us with that? Do you have a proposal? If not who else could help? Basically we would only need a zip archive per platform for the en-US locale. Thanks!

I'm not clear on what exactly is needed. It sounds like "Firefox for Puppeteer" might be a new thing that we're publishing for Nightly already, and we're not sure where to host it for beta/release?

To give some more background... there is no specific Firefox build that we are going to ship. Right now it's a regular Firefox Nightly that is downloaded and used by Puppeteer if users decide to use Firefox instead of Chrome. With this bug we want to switch away from Nightly (which in some cases is fragile and not reliable for web testing) and make use of a beta or even final release of Firefox.

That means to get the most recent Firefox release version we would use LATEST_FIREFOX_VERSION from https://product-details.mozilla.org/1.0/firefox_versions.json.

Then we can mostly build the following URL:
https://archive.mozilla.org/pub/firefox/candidates/92.0-candidates/%build%/%platform%/en-US/firefox-92.0.zip

Hereby the platform is known, but the build number would have to be queried. I assume there is no query service available to do that via a JSON HTTP request, but we would have to actually parse the output from https://archive.mozilla.org/pub/firefox/candidates/92.0-candidates/ and take the latest build.

Comment 6

[[e-mail] (:bhearsum)]

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+1] from comment #5)

(In reply to [e-mail] (:bhearsum) from comment #4)

archive.mozilla.org is an S3 bucket fronted by cloudfront - it's fine to throw any amount of load at it.

This is great to hear! Thanks.

Ben, could you help us with that? Do you have a proposal? If not who else could help? Basically we would only need a zip archive per platform for the en-US locale. Thanks!

I'm not clear on what exactly is needed. It sounds like "Firefox for Puppeteer" might be a new thing that we're publishing for Nightly already, and we're not sure where to host it for beta/release?

To give some more background... there is no specific Firefox build that we are going to ship. Right now it's a regular Firefox Nightly that is downloaded and used by Puppeteer if users decide to use Firefox instead of Chrome. With this bug we want to switch away from Nightly (which in some cases is fragile and not reliable for web testing) and make use of a beta or even final release of Firefox.

That means to get the most recent Firefox release version we would use LATEST_FIREFOX_VERSION from https://product-details.mozilla.org/1.0/firefox_versions.json.

Then we can mostly build the following URL:
https://archive.mozilla.org/pub/firefox/candidates/92.0-candidates/%build%/%platform%/en-US/firefox-92.0.zip

Hereby the platform is known, but the build number would have to be queried. I assume there is no query service available to do that via a JSON HTTP request, but we would have to actually parse the output from https://archive.mozilla.org/pub/firefox/candidates/92.0-candidates/ and take the latest build.

Thanks for all the additional context! I actually have good news for you -- you can look up which build of a version we shipped through Ship It's public API. Eg: https://shipitapi-public.services.mozilla.com/releases?product=firefox&status=shipped&version=92.0 shows that we shipped build 3 for it.

Does this address your needs?

Comment 7

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Ben, I think that sounds great! So lets check again for the steps:

1. Find the most recent release version: https://product-details.mozilla.org/1.0/firefox_versions.json
2. Find the last build_number for version %version%: https://shipitapi-public.services.mozilla.com/releases?product=firefox&status=shipped&version=%version%
3. Download the build from https://archive.mozilla.org/pub/firefox/candidates/%version%-candidates/build%build_number%/%platform%/en-US/ whereby file extensions vary per platform between firefox-%version%.(tar.bz2|zip) and Firefox%20%version%.dmg

Ben, I assume there is no easier way to determine and distinct the archive name?

Comment 8

[[e-mail] (:bhearsum)]

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+1] from comment #7)

Ben, I think that sounds great! So lets check again for the steps:

1. Find the most recent release version: https://product-details.mozilla.org/1.0/firefox_versions.json

This is the authoritative place for the current version that has shipped - so if that's what you want, then yes, you need this. If yo u don't care whether or not a Firefox build is officially shipped yet, you could just use the API from step 2 and drop the version to find all versions that Ship It considers shipped. 99% of the time the latest one that Ship It thinks is shipped will match product details, so it's up to you what's acceptable.

2. Find the last build_number for version %version%: https://shipitapi-public.services.mozilla.com/releases?product=firefox&status=shipped&version=%version%

This is the authoritative place for build numbers, so you'll definitely need this endpoint.

3. Download the build from https://archive.mozilla.org/pub/firefox/candidates/%version%-candidates/build%build_number%/%platform%/en-US/ whereby file extensions vary per platform between firefox-%version%.(tar.bz2|zip) and Firefox%20%version%.dmg

Ben, I assume there is no easier way to determine and distinct the archive name?

Yes, you need the builds from archive. If you don't want to hard code specific file names, you could query BuildHub (https://buildhub.moz.tools/) for them instead - although I couldn't come up with an obvious way quickly.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Alright. Then that's the way to go. I proposed these steps to the Puppeteer folks.

Comment 10

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Actually before doing this change we will have to make sure that Puppeteer works with Fission enabled. The minimal required work happens over on bug 1601245.

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+2]]

With the Fission fixes riding the trains to release we can do that as earliest end of this year. And to minimize the amount of possible regressions for users it would be wise to not start before the Christmas break. So I would like to see it in the reserved list. Resetting P2 for next triage meeting.

Comment 12

[Henrik Skupin [:whimboo][⌚️UTC+2]]

We are going to remove this bug from our list of M3 bugs because having this change we might detect issues very late (during beta or release) and we wont be able to have a fix soon. As such keeping it working with Nightly would be our preferred way for now.

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Just saw bug 922241 which is about zip files, and there is the following comment:

Some people have mentioned that it's convenient to have them because they're a little easier to work with for testing, but our installers are very simple to extract by hand these days;

1. 7z x firefox-installer.exe
2. you have a runnable firefox in the "core" directory

Not sure if that only works with 7z or if other tools also support unpacking, but it might be an option. Also msiexec seem to be able to unpack an executable to a specific target directory: msiexec /a msifilepath/qb TARGETDIR=targetpath. Something I might have to test.