Closed Bug 1945900 Opened 1 year ago Closed 1 year ago

update-verify fails to unpack lzma-compressed dmg

Categories

(Release Engineering :: Release Automation, defect, P1)

Product:
Release Engineering
Component:
Release Automation
Type:
defect

Tracking

(firefox-esr115 unaffected, firefox-esr128 unaffected, firefox135 unaffected, firefox136+ fixed, firefox137+ fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox-esr115 --- unaffected
firefox-esr128 --- unaffected
firefox135 --- unaffected
firefox136 + fixed
firefox137 + fixed

People

(Reporter: jcristau, Assigned: hneiva)

References

(Regression)

Details

(Keywords: regression)

Attachments

(1 file)

Bug 1945900 - Support LZMA DMGs in Update Verify r?#releng
48 bytes, text/x-phabricator-request
Details | Review

tools/update-verify/release/common/unpack.sh uses 7z x to unpack a dmg, and expects something like Firefox/Firefox.app/ inside.
But since 136.0b1 it's only seeing:

   Date      Time    Attr         Size   Compressed  Name
------------------- ----- ------------ ------------  ------------------------
                    .....          512           76  0.ddm
                    .....        32256          260  1.Apple_partition_map
                    .....         4096           88  2.Apple_Driver_ATAPI
                    .....    452984832    132230176  3.hfsx
                    .....         5120            0  4.free
------------------- ----- ------------ ------------  ------------------------
                             453026816    132230600  5 files

Looks to be from the lzma compression.

7z l output from https://archive.mozilla.org/pub/firefox/nightly/2024/12/2024-12-03-21-40-41-mozilla-central/firefox-135.0a1.en-US.mac.dmg vs https://archive.mozilla.org/pub/firefox/nightly/2024/12/2024-12-04-09-26-16-mozilla-central/firefox-135.0a1.en-US.mac.dmg is wildly different.

Keywords: regressionwindow-wanted
Regressed by: 1935015

:vasi, since you are the author of the regressor, bug 1935015, could you take a look?

For more information, please visit BugBot documentation.

Flags: needinfo?(dave)
Summary: update-verify fails to unpack dmg → update-verify fails to unpack lzma-compressed dmg

Taking a look.

Do we know where this script is running? A 7zz from 7zip upstream seems to work ok, maybe we can just upgrade the 7z version.

Flags: needinfo?(dave)

(We could also consider using libdmg's dmg and hfsplus scripts for unpacking.)

Flags: needinfo?(jcristau)

It's running in a docker image based on ubuntu 20.04, and 7z is from the p7zip-full package.

https://searchfox.org/mozilla-central/source/taskcluster/docker/update-verify

Flags: needinfo?(jcristau)

Thanks for the info. OOC, how can I tell that's based on 20.04? I don't know how to figure out what DOCKER_IMAGE_PARENT refers to. Ubuntu 20.04 is nearly EOL, so that's exciting.

Anyhow, it looks like only Ubuntu 24.11 oracular has the new 7zip in the repo. (The package has also changed names in modern Ubuntu versions.) I don't see us switching to a non-LTS version.

Some options:

  1. Fetch a moden 7z binary from eg: upstream 7zip. These seem to run fine on 20.04, and can handle lzma-compressed Firefox.
  2. Use libdmg to unpack the dmg, instead of 7zip. We'd have to get the libdmg toolchain into the docker image.
  3. Figure out if it's possible to make lzma dmgs acceptable to an old version of 7zip, debugging the 7zip code to figure out what it expects.

I'm most inclined towards option 1. Does that seem reasonable to folks?

The docker image is:
https://searchfox.org/mozilla-central/source/taskcluster/docker/update-verify/Dockerfile

Which is referenced https://searchfox.org/mozilla-central/rev/6fc1a107c44e011a585614cbbe8826f9e01ec802/taskcluster/kinds/docker-image/kind.yml#255 (you can see the parent there)

So could be relevant to update that parent to ubuntu2204-base or ubuntu2404-base. What I'm not sure about though, is if this was pinned at 2004-base due to wanting to validate on "oldest linux Firefox supports" due to glibc/etc for the updater binary itself.

Adding a 7z fetch to the task graph and using it instead of using the distro package seems like a good approach to me too.

Maybe add something like this in taskcluster/kinds/fetch/toolchains.yml and use that in UV?

@@ -743,3 +743,14 @@ uv:
         type: git
         repo: https://github.com/astral-sh/uv
         revision: 0652800cb018eb086fbbf220a1f73af7e320be54
+
+7z-2409-linux64:
+    description: 7zip 24.09 for x64 Linux
+    fetch:
+        type: static-url
+        url: https://github.com/ip7z/7zip/releases/download/24.09/7z2409-linux-x64.tar.xz
+        sha256: 914c7e20ad5ef8e4d3cf08620ff8894b28fe11b7eb99809d6930870fbe48a281
+        size: 1565344
+        artifact-name: 7z.tar.zst
+        strip-components: 1
+        add-prefix: 7z/

I'm having trouble finding any existing examples of adding a fetched artifact to a docker image for use in taskcluster. How would I do that?

Also, is there a job I can run on Try to repro this, and later test that it's fixed? "release-update-verify-firefox-macosx64-*/30" looked plausible, but it didn't surface any errors.

(In reply to Gabriel Bustamante [:gabriel] from comment #10)

Maybe add something like this in taskcluster/kinds/fetch/toolchains.yml and use that in UV?

Good news: I see bug 1863288 added the right fetch and toolchain tasks to make this exact version of 7zip available to downstream tasks.

I'm currently testing which approach would work best between upgrading the Ubuntu base image or just upgrading 7z in the update-verify tasks.

Depends on: 1863288

Marking this bug as P1/S1 because this bug creates a hole in our test coverage in the release pipeline.

Severity: -- → S1
Priority: -- → P1

The bug is marked as tracked for firefox136 (beta) and tracked for firefox137 (nightly). However, the bug still isn't assigned.

:jlorenzo, could you please find an assignee for this tracked bug? Given that it is a regression and we know the cause, we could also simply backout the regressor. If you disagree with the tracking decision, please talk with the release managers.

For more information, please visit BugBot documentation.

Flags: needinfo?(jlorenzo)

(In reply to Dave Vasilevsky [:vasi] from comment #7)

Anyhow, it looks like only Ubuntu 24.11 oracular has the new 7zip in the repo. (The package has also changed names in modern Ubuntu versions.) I don't see us switching to a non-LTS version.

I confirm you're right. 24.04 doesn't have it. This try commit still outputs:

[task 2025-02-05T11:37:06.519Z] Retrieving 'http://ftp.stage.mozaws.net/pub/firefox/candidates/136.0b1-candidates/build2/mac/en-US/Firefox 136.0b1.dmg' from cache...
[task 2025-02-05T11:37:06.645Z] ERROR: Dangerous link path was ignored : Firefox/[] : /Applications
[task 2025-02-05T11:37:12.724Z] ERROR: Unsupported Method : 0.ddm
[task 2025-02-05T11:37:12.724Z] ERROR: Unsupported Method : 1.Apple_partition_map
[task 2025-02-05T11:37:12.724Z] ERROR: Unsupported Method : 2.Apple_Driver_ATAPI
[task 2025-02-05T11:37:13.193Z] ERROR: Unsupported Method : 3.hfsx
[task 2025-02-05T11:37:13.196Z] Couldn't find .app package

I could repro the behavior locally and here's the version I got:

$ 7z -h

7-Zip 23.01 (x64) : Copyright (c) 1999-2023 Igor Pavlov : 2023-06-20
 64-bit locale=en_US.UTF-8 Threads:10 OPEN_MAX:1048576

Some options:

  1. Fetch a moden 7z binary from eg: upstream 7zip. These seem to run fine on 20.04, and can handle lzma-compressed Firefox.

I tried locally and this approach works on the 24.04 image but not the 20.04. Here's what I get:

$ ./7zz Firefox\ 136.0b1.dmg
./7zz: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by ./7zz)
./7zz: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by ./7zz)
./7zz: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by ./7zz)

That makes sense to me because 20.04 turns out to ship libc6 v2.31.

So I guess we need to upgrade both the Ubuntu base image and download 7zip 24.09.

Flags: needinfo?(jlorenzo)

I just chatted with :bhearsum and :dmeehan. We are going to back bug 1863288 out of beta in order to unblock update-verify.

For the long term fix, :bhearsum suggested we go with that option instead:

(In reply to Dave Vasilevsky [:vasi] from comment #7)

  1. Use libdmg to unpack the dmg, instead of 7zip. We'd have to get the libdmg toolchain into the docker image.

:bhearsum mentioned we already have a toolchain for it which I just found.

The following should work fine to extract them:

./dmg extract target.dmg extracted
./hfsplus extracted extractall

And will end up with a .app bundle in the cwd. This is a notable change from extracting with 7z, which creates a directory with the .app bundle inside. This logic will need to be adjusted to cope with that.

Yeah. Also, the DMG seems to be just like... a disk image?

We could use dmg2img (which might be in the 20.04 distro repo)

And do something like:

dmg2img "firefox.dmg" "output.img"

And then mount that.

That should also get us a .app inside the mnt/ directory.

Could be tricky if the image has multiple partitions tho...

Ah it doesn't seem like dmg2img it's in the distro package :/
and it seems very outdated never-mind.

(In reply to Gabriel Bustamante [:gabriel] from comment #19)

Yeah. Also, the DMG seems to be just like... a disk image?

We could use dmg2img (which might be in the 20.04 distro repo)

And do something like:

dmg2img "firefox.dmg" "output.img"

And then mount that.

That should also get us a .app inside the mnt/ directory.

Could be tricky if the image has multiple partitions tho...

Theoretically this could work, yes. I think it might be best to stick to unpacking with the same tools with package with though? (this could help avoid future bustage if we make additional enhancements in libdmg-hfsplus.)

We should try libdmg from what I can see it hasn’t seen active maintenance for years either.

Ah well if we package it with that... it... should... work

If the patch it's getting backed out maybe we have time to update both Ubuntu and 7z. That's probably closer to the current thing.

(In reply to Gabriel Bustamante [:gabriel] from comment #22)

We should try libdmg from what I can see it hasn’t seen active maintenance for years either.

We maintain it in https://github.com/mozilla/libdmg-hfsplus :)

(In reply to bhearsum@mozilla.com (:bhearsum) from comment #25)

(In reply to Gabriel Bustamante [:gabriel] from comment #22)

We should try libdmg from what I can see it hasn’t seen active maintenance for years either.

We maintain it in https://github.com/mozilla/libdmg-hfsplus :)

TIL (we have a libdmg fork)

Assignee: nobody → hneiva
Status: NEW → ASSIGNED
Pushed by hneiva@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/f38a3b88c66a Support LZMA DMGs in Update Verify r=releng-reviewers,taskgraph-reviewers,bhearsum

https://hg.mozilla.org/mozilla-central/rev/f38a3b88c66a

Status: ASSIGNED → RESOLVED
Closed: 1 year ago
status-firefox137: affected → fixed
Resolution: --- → FIXED
No longer depends on: 1863288
See Also: → 1966435
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: