Open Bug 1694127 Opened 7 months ago Updated 5 days ago

Change recommended version of Firefox for Puppeteer from Nightly to Beta/Release

Categories

(Remote Protocol :: Agent, task, P2)

task
Points:
8

Tracking

(Not tracked)

People

(Reporter: whimboo, Unassigned)

References

(Blocks 1 open bug, )

Details

(Whiteboard: [bidi-m2-mvp])

Originally filed as: https://github.com/puppeteer/puppeteer/issues/6894

We want to switch the recommended version of Firefox for Puppeteer from Firefox Nightly to Firefox Beta or even a release build. But that means we have to find a way how to get those builds installed. For them we do not offer zip archives as stated in bug 308048.

We should figure out how to get the .exe or .msi installed via node.js / npm, or maybe use the latest available release candidate build?

Summary: Change recommended version of Firefox for Puppeteer rom Nightly to Beta/Release → Change recommended version of Firefox for Puppeteer from Nightly to Beta/Release
Points: --- → 8
Priority: -- → P2
Priority: P2 → P3

Quick update here. I haven't gotten a reply on https://github.com/puppeteer/puppeteer/issues/6894 yet, so I asked again. Jan seems to be fine with the proposal to use the silent MSI installer, but we are still waiting for the reply from Mathias.

So the only option for zip archives will actually be the usage of the final release candidate build. But not sure if that is what we actually want to do. We should have a team consensus on it, and as such adding it for the next triage session.

Whiteboard: [bidi-m1-mvp] → [bidi-m1-mvp][webdriver:triage]

We discussed that yesterday in the WebDriver meeting. Generally we are all fine with using the zip files as long as we can be sure that the latest candidate build for a given Firefox (beta) release matches exactly the officially released binaries. And that seems to be the case.

But there was one concern raised by James we are not sure about how to handle it. Given that users of Puppeteer run their tests in CI, and based on our experience with Selenium/geckodriver, these CI projects are usually badly setup, which means that each of these running test jobs might pull and install the Firefox binary. As what I can see candidate builds are only available through archive.mozilla.org, and I assume that this is not backed by CDN? That means it would add a certain amount of load on this server, which might not be wanted.

Ben, could you help us with that? Do you have a proposal? If not who else could help? Basically we would only need a zip archive per platform for the en-US locale. Thanks!

Flags: needinfo?(bhearsum)
Whiteboard: [bidi-m1-mvp][webdriver:triage] → [bidi-m1-mvp]

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+1] from comment #3)

We discussed that yesterday in the WebDriver meeting. Generally we are all fine with using the zip files as long as we can be sure that the latest candidate build for a given Firefox (beta) release matches exactly the officially released binaries. And that seems to be the case.

But there was one concern raised by James we are not sure about how to handle it. Given that users of Puppeteer run their tests in CI, and based on our experience with Selenium/geckodriver, these CI projects are usually badly setup, which means that each of these running test jobs might pull and install the Firefox binary. As what I can see candidate builds are only available through archive.mozilla.org, and I assume that this is not backed by CDN? That means it would add a certain amount of load on this server, which might not be wanted.

archive.mozilla.org is an S3 bucket fronted by cloudfront - it's fine to throw any amount of load at it.

Ben, could you help us with that? Do you have a proposal? If not who else could help? Basically we would only need a zip archive per platform for the en-US locale. Thanks!

I'm not clear on what exactly is needed. It sounds like "Firefox for Puppeteer" might be a new thing that we're publishing for Nightly already, and we're not sure where to host it for beta/release?

Flags: needinfo?(bhearsum) → needinfo?(hskupin)

(In reply to bhearsum@mozilla.com (:bhearsum) from comment #4)

archive.mozilla.org is an S3 bucket fronted by cloudfront - it's fine to throw any amount of load at it.

This is great to hear! Thanks.

Ben, could you help us with that? Do you have a proposal? If not who else could help? Basically we would only need a zip archive per platform for the en-US locale. Thanks!

I'm not clear on what exactly is needed. It sounds like "Firefox for Puppeteer" might be a new thing that we're publishing for Nightly already, and we're not sure where to host it for beta/release?

To give some more background... there is no specific Firefox build that we are going to ship. Right now it's a regular Firefox Nightly that is downloaded and used by Puppeteer if users decide to use Firefox instead of Chrome. With this bug we want to switch away from Nightly (which in some cases is fragile and not reliable for web testing) and make use of a beta or even final release of Firefox.

That means to get the most recent Firefox release version we would use LATEST_FIREFOX_VERSION from https://product-details.mozilla.org/1.0/firefox_versions.json.

Then we can mostly build the following URL:
https://archive.mozilla.org/pub/firefox/candidates/92.0-candidates/%build%/%platform%/en-US/firefox-92.0.zip

Hereby the platform is known, but the build number would have to be queried. I assume there is no query service available to do that via a JSON HTTP request, but we would have to actually parse the output from https://archive.mozilla.org/pub/firefox/candidates/92.0-candidates/ and take the latest build.

Flags: needinfo?(hskupin) → needinfo?(bhearsum)

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+1] from comment #5)

(In reply to bhearsum@mozilla.com (:bhearsum) from comment #4)

archive.mozilla.org is an S3 bucket fronted by cloudfront - it's fine to throw any amount of load at it.

This is great to hear! Thanks.

Ben, could you help us with that? Do you have a proposal? If not who else could help? Basically we would only need a zip archive per platform for the en-US locale. Thanks!

I'm not clear on what exactly is needed. It sounds like "Firefox for Puppeteer" might be a new thing that we're publishing for Nightly already, and we're not sure where to host it for beta/release?

To give some more background... there is no specific Firefox build that we are going to ship. Right now it's a regular Firefox Nightly that is downloaded and used by Puppeteer if users decide to use Firefox instead of Chrome. With this bug we want to switch away from Nightly (which in some cases is fragile and not reliable for web testing) and make use of a beta or even final release of Firefox.

That means to get the most recent Firefox release version we would use LATEST_FIREFOX_VERSION from https://product-details.mozilla.org/1.0/firefox_versions.json.

Then we can mostly build the following URL:
https://archive.mozilla.org/pub/firefox/candidates/92.0-candidates/%build%/%platform%/en-US/firefox-92.0.zip

Hereby the platform is known, but the build number would have to be queried. I assume there is no query service available to do that via a JSON HTTP request, but we would have to actually parse the output from https://archive.mozilla.org/pub/firefox/candidates/92.0-candidates/ and take the latest build.

Thanks for all the additional context! I actually have good news for you -- you can look up which build of a version we shipped through Ship It's public API. Eg: https://shipitapi-public.services.mozilla.com/releases?product=firefox&status=shipped&version=92.0 shows that we shipped build 3 for it.

Does this address your needs?

Flags: needinfo?(bhearsum)
Whiteboard: [bidi-m1-mvp] → [bidi-m2-mvp]
Priority: P3 → --

Ben, I think that sounds great! So lets check again for the steps:

  1. Find the most recent release version: https://product-details.mozilla.org/1.0/firefox_versions.json
  2. Find the last build_number for version %version%: https://shipitapi-public.services.mozilla.com/releases?product=firefox&status=shipped&version=%version%
  3. Download the build from https://archive.mozilla.org/pub/firefox/candidates/%version%-candidates/build%build_number%/%platform%/en-US/ whereby file extensions vary per platform between firefox-%version%.(tar.bz2|zip) and Firefox%20%version%.dmg

Ben, I assume there is no easier way to determine and distinct the archive name?

Flags: needinfo?(bhearsum)

(In reply to Henrik Skupin (:whimboo) [⌚️UTC+1] from comment #7)

Ben, I think that sounds great! So lets check again for the steps:

  1. Find the most recent release version: https://product-details.mozilla.org/1.0/firefox_versions.json

This is the authoritative place for the current version that has shipped - so if that's what you want, then yes, you need this. If yo u don't care whether or not a Firefox build is officially shipped yet, you could just use the API from step 2 and drop the version to find all versions that Ship It considers shipped. 99% of the time the latest one that Ship It thinks is shipped will match product details, so it's up to you what's acceptable.

  1. Find the last build_number for version %version%: https://shipitapi-public.services.mozilla.com/releases?product=firefox&status=shipped&version=%version%

This is the authoritative place for build numbers, so you'll definitely need this endpoint.

  1. Download the build from https://archive.mozilla.org/pub/firefox/candidates/%version%-candidates/build%build_number%/%platform%/en-US/ whereby file extensions vary per platform between firefox-%version%.(tar.bz2|zip) and Firefox%20%version%.dmg

Ben, I assume there is no easier way to determine and distinct the archive name?

Yes, you need the builds from archive. If you don't want to hard code specific file names, you could query BuildHub (https://buildhub.moz.tools/) for them instead - although I couldn't come up with an obvious way quickly.

Flags: needinfo?(bhearsum)

Alright. Then that's the way to go. I proposed these steps to the Puppeteer folks.

Priority: -- → P2
You need to log in before you can comment on or make changes to this bug.