release-snap-repackage-firefox fails: /snap/bin/snapcraft: 3: exec: /snap/snapcraft/current/usr/bin/python3: not found
Categories
(Release Engineering :: Release Automation: Snap, defect)
Tracking
(firefox-esr91 fixed, firefox-esr102 fixed, firefox104blocking fixed, firefox105 fixed)
People
(Reporter: gbrown, Assigned: jlorenzo)
References
Details
Attachments
(2 files)
|
138.15 KB,
text/plain
|
Details | |
|
48 bytes,
text/x-phabricator-request
|
diannaS
:
approval-mozilla-beta+
diannaS
:
approval-mozilla-release+
RyanVM
:
approval-mozilla-esr102+
RyanVM
:
approval-mozilla-esr91+
|
Details | Review |
https://firefox-ci-tc.services.mozilla.com/tasks/ZJLcv00LQPiXDZEFYd6I1Q
https://firefoxci.taskcluster-artifacts.net/ZJLcv00LQPiXDZEFYd6I1Q/1/public/logs/live_backing.log
Reading package lists... Done
+ snapcraft
/snap/bin/snapcraft: 3: exec: /snap/snapcraft/current/usr/bin/python3: not found
[taskcluster 2022-08-02 22:13:29.378Z] === Task Finished ===
[taskcluster 2022-08-02 22:13:29.447Z] Unsuccessful task run with exit code: 127 completed in 141.528 seconds
release-snap-repackage-firefox was running fine on beta on July 31.
|
Reporter | |
Comment 1•2 years ago
|
||
Maybe related to the image rebuild, https://firefox-ci-tc.services.mozilla.com/tasks/DTppit47RNmcBdHe72QGLw ?
|
||
Updated•2 years ago
|
|
|
Reporter | |
Comment 2•2 years ago
|
||
gbrown@mozpad2:~/src$ ./mach taskcluster-load-image --task-id DTppit47RNmcBdHe72QGLw
Downloading from https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/DTppit47RNmcBdHe72QGLw/artifacts/public/image.tar.zst
e722d396f503: Loading layer [==================================================>] 65.52MB/65.52MB
2280e40b0eab: Loading layer [==================================================>] 3.518GB/3.518GB
Loaded image: firefox-snap:latest
Found docker image: docker.io/library/firefox-snap:latest
Try: docker run -ti --rm docker.io/library/firefox-snap:latest bash
gbrown@mozpad2:~/src$ docker run -ti --rm docker.io/library/firefox-snap:latest bash
root@e774abc43394:/scripts# python3
Python 3.6.9 (default, Jun 29 2022, 11:45:57)
[GCC 8.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> quit()
root@e774abc43394:/scripts# snapcraft
/snap/bin/snapcraft: 3: exec: /snap/snapcraft/current/usr/bin/python3: not found
root@e774abc43394:/scripts# ls -l /snap/snapcraft/current/usr/bin/py*
-rwxr-xr-x 1 root root 7816 Jul 8 11:26 /snap/snapcraft/current/usr/bin/py3clean
-rwxr-xr-x 1 root root 12123 Jul 8 11:26 /snap/snapcraft/current/usr/bin/py3compile
lrwxrwxrwx 1 root root 31 Mar 13 2020 /snap/snapcraft/current/usr/bin/py3versions -> ../share/python3/py3versions.py
lrwxrwxrwx 1 root root 9 Mar 13 2020 /snap/snapcraft/current/usr/bin/python3 -> python3.8
-rwxr-xr-x 1 root root 5494544 Jul 8 11:28 /snap/snapcraft/current/usr/bin/python3.8
lrwxrwxrwx 1 root root 10 Mar 13 2020 /snap/snapcraft/current/usr/bin/pyvenv -> pyvenv-3.8
root@e774abc43394:/scripts# /snap/snapcraft/current/usr/bin/python3
bash: /snap/snapcraft/current/usr/bin/python3: No such file or directory
root@e774abc43394:/scripts# /snap/snapcraft/current/usr/bin/python3.8
bash: /snap/snapcraft/current/usr/bin/python3.8: No such file or directory
root@e774abc43394:/scripts# file /snap/snapcraft/current/usr/bin/python3.8
/snap/snapcraft/current/usr/bin/python3.8: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /snap/core20/current/lib64/ld-linux-x86-64.so.2, for GNU/Linux 3.2.0, BuildID[sha1]=1f3df9df2b5e575fdee41890fe17f6de614f93f6, stripped
|
|
Reporter | |
Comment 3•2 years ago
|
||
:jlorenzo - I'm not sure this is making sense! Do you have an idea of what's gone wrong?
|
Assignee | |
Comment 4•2 years ago
|
||
Thank you very much for the ping and for all the pointers 👍 TIL ./mach taskcluster-load-image 💯
First lead followed: core20
I ran the image image on my machine and interestingly, I have an additional log line:
root@8d5161d86921:/scripts# /snap/snapcraft/current/usr/bin/python3
qemu-x86_64: Could not open '/snap/core20/current/lib64/ld-linux-x86-64.so.2': No such file or directory
Is core20 the actual core issue?
I don't know for sure whether that's because I'm running an M1 Macbook or if that's really tied to the issue we see. I'd lean towards the latter because of a few additional things:
/snap/core20/[...]doesn't exist in the image but/snap/core18/[...]does!- Per the TC index[1], the
firefox-snapimage has been built 3 times for the past year. The last 2 being in the last week and the first one being exactly 1 year. - When comparing the logs between last year and this week, I see one particular difference around
core20. Let's deep dive into them!
snapcraft: I Know What You Did Last Summer[2]
We install snapcraft by using its snap package because back in bug 1591751, we needed the latest version of snap. Luckily, we log that version of snap is installed thanks to bug 1655669. Based on this: here's what snapcraft we installed this week[3]:
Running: [/bin/sh -c bash download_and_install_snap.sh 'snapcraft']
[..]
++ curl --header 'X-Ubuntu-Series: 16' 'https://api.snapcraft.io/api/v1/snaps/details/snapcraft?channel=stable'
+ SNAP_METADATA='{
[...]
"base": "core20",
"last_updated": "2022-07-08T11:29:16.074763+00:00",
"revision": 7923,
"version": "7.0.11",
}'
[...]
And here's what we got last year[4]:
Running: [/bin/sh -c bash download_and_install_snap.sh 'snapcraft']
[...]
++ curl --header 'X-Ubuntu-Series: 16' 'https://api.snapcraft.io/api/v1/snaps/details/snapcraft?channel=stable'
+ SNAP_METADATA='{
"base": "core18",
"last_updated": "2021-06-18T22:03:42.534110+00:00",
"revision": 6596,
"version": "4.8.3",
}'
[...]
Thus, something is wrong with the version of core.
Potential fixes
The immediate fix is likely to bump this line[5] to get this unblocked. A longer-term fix could be to stop hardcoding the version of core and reading it from the metadata at https://api.snapcraft.io/api/v1/snaps/details/snapcraft?channel=stable. If we do so, we will likely have to perform the same change for all snap-flavored dependencies of snapcraft. I also see we're building our snap on Ubuntu bionic (aka 18.04 LTS) which gets no additional free support in April. We may want to bump the snap to jammy, the latest LTS.
[1] https://firefox-ci-tc.services.mozilla.com/tasks/index/gecko.cache.level-3.docker-images.v2.firefox-snap.pushdate
[2] https://www.imdb.com/title/tt0119345/ 😁
[3] https://firefoxci.taskcluster-artifacts.net/DTppit47RNmcBdHe72QGLw/0/public/logs/live_backing.log
[4] See attached log - the one hosted on Taskcluster is about to expire.
[5] https://searchfox.org/mozilla-central/rev/560bbebd595cf2f6a9861f3f67dd0739f4c4c559/taskcluster/docker/firefox-snap/Dockerfile#40
|
|
Assignee | |
Comment 5•2 years ago
|
||
I don't remember, do we still use the snap repackages now that Canonical ships Firefox for us?
|
|
Assignee | |
Updated•2 years ago
|
|
||
Comment 6•2 years ago
|
||
(In reply to Johan Lorenzo [:jlorenzo] from comment #5)
I don't remember, do we still use the snap repackages now that Canonical ships Firefox for us?
We still build locally for CI/testing, but ours don't ship.
|
||
Comment 7•2 years ago
•
|
||
FWIW, it appears that core20 is being used for the upstream snap builds as well, all the way down the line to ESR91.
https://git.launchpad.net/~mozilla-snaps/firefox-snap/+git/firefox-snap/tree/snapcraft.yaml#n7
Appears they've been on core20 since the initial repo setup.
|
|
Assignee | |
Comment 8•2 years ago
|
||
Thanks for the information, Mike! 😃 Then, I'm looking into it to see if we can get repackages back on feet soon. The fact that core20 is used at Canonical is promising, thank you for the pointer, Ryan!
|
|
||
Comment 9•2 years ago
|
||
Upstream appears to be testing core22 as well:
https://git.launchpad.net/~mozilla-snaps/firefox-snap/+git/firefox-snap/log/?h=core22
|
|
Assignee | |
Comment 10•2 years ago
|
||
|
||
Comment 11•2 years ago
|
||
Pushed by jlorenzo@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/c5c9d4450697 Bump snap dependencies core18=>20 and gnome-3-34=>38 r=gerard-majax,gbrown
|
||
Comment 12•2 years ago
|
||
(In reply to Ryan VanderMeulen [:RyanVM] from comment #7)
FWIW, it appears that core20 is being used for the upstream snap builds as well, all the way down the line to ESR91.
https://git.launchpad.net/~mozilla-snaps/firefox-snap/+git/firefox-snap/tree/snapcraft.yaml#n7Appears they've been on core20 since the initial repo setup.
It's true that we've been using older versions of core and gnome than Canonical to package Firefox, but this bug is about something different: we're using the wrong hard-coded versions of those snaps for snapcraft itself, because we're “installing” it in the Docker image by manually downloading the snap files and unpacking the squashfs images, then doing some hacks to (apparently?) compensate for how snapd would have set up the filesystem environment. Both the dependency information and the hacks (see the part of D153734 about making the Python interpreter work) have diverged from upstream, giving us this bug. (And none of this is pinned to specific revisions of anything, so re-running the “same” job that previously worked can fail, as seen here.)
However, I see that D153734 also updates the dependencies for packaging Firefox.
(For context, I ran into all of this a few weeks ago while trying and failing to get that Dockerfile to work locally. I meant to file some bugs about it, but at the time it wasn't clear if we were even going to continue running these repackaging jobs; for example, see bug 1297531 comment #10.)
|
||
Comment 13•2 years ago
|
||
| bugherder | ||
| Comment hidden (Intermittent Failures Robot) |
|
|
Assignee | |
Comment 16•2 years ago
|
||
Thank you for the thorough input, Jed! I agree with you, the original is because we hardcode the version of core for snapcraft itself. I also agree that we diverged from upstream. However, D153734 has brought us closer to upstream which includes the hack on the Python interpreter. It actually comes from the upstream repo of snapcraft[1]. I left more details about which part comes from upstream in the Phabricator revision.
Speaking of using these repacking jobs, I concur with you. It's not clear to me how they're used. Do you :jld and :gerard-majax use them? If not, I wouldn't be surprised if no one does actually and I'd be super happy to retire this job.
|
|
Assignee | |
Comment 17•2 years ago
|
||
Comment on attachment 9288466 [details]
Bug 1782828 - Bump snap dependencies core18=>20 and gnome-3-34=>38 r=gerard-majax,gbrown
Beta/Release Uplift Approval Request
- User impact if declined: None to end-users. However our CI jobs will remain red on beta and release
- Is this code covered by automated tests?: No
- Has the fix been verified in Nightly?: No
- Needs manual test from QE?: No
- If yes, steps to reproduce:
- List of other uplifts needed: None
- Risk to taking this patch: Low
- Why is the change risky/not risky? (and alternatives if risky): Low risk because we don't ship this binary anymore. It's just a way to make CI green again. Patch was tested in try.
- String changes made/needed: None
- Is Android affected?: No
|
||
Comment 18•2 years ago
|
||
(In reply to Johan Lorenzo [:jlorenzo] from comment #16)
Thank you for the thorough input, Jed! I agree with you, the original is because we hardcode the version of
coreforsnapcraftitself. I also agree that we diverged from upstream. However, D153734 has brought us closer to upstream which includes the hack on the Python interpreter. It actually comes from the upstream repo ofsnapcraft[1]. I left more details about which part comes from upstream in the Phabricator revision.Speaking of using these repacking jobs, I concur with you. It's not clear to me how they're used. Do you :jld and :gerard-majax use them? If not, I wouldn't be surprised if no one does actually and I'd be super happy to retire this job.
As I mentionned, having that in sync with Ubuntu's package and running on mozilla-central + autoland might make bug 1763188 much easier to fix.
|
|
Assignee | |
Comment 19•2 years ago
|
||
Thanks for reminding me our conversation on Element, :gerard-majax! The long weekend made me forget about it 😅
|
|
||
Comment 20•2 years ago
|
||
Comment on attachment 9288466 [details]
Bug 1782828 - Bump snap dependencies core18=>20 and gnome-3-34=>38 r=gerard-majax,gbrown
Approved for 104.0b8
|
|
||
Comment 21•2 years ago
|
||
| bugherder uplift | ||
|
|
||
Comment 22•2 years ago
|
||
Comment on attachment 9288466 [details]
Bug 1782828 - Bump snap dependencies core18=>20 and gnome-3-34=>38 r=gerard-majax,gbrown
Approved for 103.0.2 dot release
|
|
||
Comment 23•2 years ago
|
||
Comment on attachment 9288466 [details]
Bug 1782828 - Bump snap dependencies core18=>20 and gnome-3-34=>38 r=gerard-majax,gbrown
Also approved for the ESRs.
|
|
||
Comment 24•2 years ago
|
||
| bugherder uplift | ||
|
|
||
Comment 25•2 years ago
|
||
| bugherder uplift | ||
|
|
||
Comment 26•2 years ago
|
||
I don't think I have any plans to use the CI repackage jobs themselves, but I do have some patches that use the snapcraft.yaml from those jobs to add a ./mach repackage snap command for repacking (and, optionally, installing as a Snap) a local build for development purposes.
Description
•