1936246 - Switch from bz2 to xz for Firefox releases on https://ftp.mozilla.org/ breaks Puppeteer

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Henrik Skupin [:whimboo][⌚️UTC+2]]

The hard switch to using the xz archive for Firefox releases as done over on bug 1710599 caused a regression for Puppeteer which relies on the bz2 archive format. Switching over to xz is not really possible at the moment because of certain factors:

• There is only a native package with xz (lzma) support available on npm and a native package is not wanted regarding the extra dependencies and compilation step on the target system
• They need a streaming solution to chain with tar
• They don't want to depend on packages not updated in the last years

That means that right now downloading Firefox is broken for Nightly builds. Here a failure from their CI:
https://github.com/puppeteer/puppeteer/actions/runs/12251886667/job/34178310393#step:8:46

[Tracking Requested - why for this release]:
Blocks at least one major test framework from using Firefox in upcoming releases for automation purposes. It's not clear which other tools are affected as well, which all rely on bz2.

Comment 1

[alexrudenko]

Puppeteer is a Node.js library and there is no built-in support for xz in Node. For bz2 there is also no support so we used a JavaScript implementation of bz2 (https://github.com/regular/unbzip2-stream). For xz, there seems to be no Node module available that is not a binding to libzma that requires more host dependencies or pre-built binaries for all platforms and that supports streaming API to integrate with tar streams.

I think the best way forward to require the archive extraction tool to be installed on the host for Firefox and invoke it as a subprocess.

Comment 2

[Ryan VanderMeulen [:RyanVM]]

Not sure why I was NIed on this. Redirecting to the manager of the team who did the implementation work if there's concerns around ship-readiness.

Comment 3

[Sylvestre Ledru [:Sylvestre]]

According to https://github.com/puppeteer/puppeteer/pull/13391/files
it seems that it won't be an issue for too long

Comment 4

[Calixte Denizet (:calixte)]

That means that right now downloading Firefox is broken for Nightly builds. Here a failure from their CI:
https://github.com/puppeteer/puppeteer/actions/runs/12251886667/job/34178310393#step:8:46

and it's causing some failures in the pdf.js CI too:
https://github.com/mozilla/pdf.js/actions/runs/12257676270/job/34195691801#step:4:28

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+2]]

(In reply to Calixte Denizet (:calixte) from comment #4)

and it's causing some failures in the pdf.js CI too:
https://github.com/mozilla/pdf.js/actions/runs/12257676270/job/34195691801#step:4:28

And as mentioned by Calixte on Slack they are some versions behind the most recent release of Puppeteer because of some intermittent test failures that pop up with the update. As such an upgrade isn't quickly done even if Puppeteer releases a new version with proper support for xz archives.

Comment 6

[Johan Lorenzo [:jlorenzo]]

Hello there!

Thanks for raising this issue. We announced and made the change 2 weeks ago. I suspect this was noticed yesterday as the 134 binaries disappeared from the latest-mozilla-central folder.

We expected this type of breakage. That's the main reason why we landed it at the beginning of the Nightly cycle. This is one of the purposes of Nightly: we enable third-parties to adapt to some breaking changes we make.

As :sylvestre mentioned in comment 3, puppeteer has a patch that, as of this writing, was merged and is now being released as v23.10.3. Given that we don't have a good way of programmatically warn downstream users about such changes (with a deprecation warning for instance), turning tests to red is sadly the most effective way. As a matter of proof: puppeteer noticed this problem today instead of 2 weeks ago. Therefore, I don't believe putting bz2 back is going to help people - they wouldn't notice until the next breakage.

An alternative could be to hold xz on Nightly for one more 4-week cycle. The main problem with this approach is: it may require downstream consumers to change their logic one more time. Puppeteer has a simple if condition on the major version number. They would have to change this logic again if the change doesn't ride the train and I suspect they wouldn't be the only ones.

Thus, I'd be in favor of not changing anything and letting xz ride the train. xz will reach beta on January 6th, 2025. That said, let's reassess this before then.

In the meantime, I'm clearing the tracking request as we don't have any actions to make until early January.

Comment 7

[BugBot [:suhaib / :marco/ :calixte]]

:hneiva, since you are the author of the regressor, bug 1710599, could you take a look? Also, could you set the severity field?

For more information, please visit BugBot documentation.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+2]]

Well, I think that we can close this bug given that Puppeteer released a new version with support for that new compression type.