Generate multilocale Firefox flatpak
Categories
(Release Engineering :: Release Automation: Other, enhancement)
Tracking
(Not tracked)
People
(Reporter: Sylvestre, Assigned: mtabara)
References
(Blocks 1 open bug)
Details
Attachments
(2 files, 2 obsolete files)
17.49 KB,
patch
|
Details | Diff | Splinter Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review |
Flatpak, like Snap, should include all shipped locales. See https://bugzilla.mozilla.org/show_bug.cgi?id=1297519 for the original implementation
Reporter | ||
Updated•6 years ago
|
Assignee | ||
Comment 1•6 years ago
|
||
Started working on this, with ETA to get the packaging done before the All-hands.
Comment 2•6 years ago
|
||
See: https://bugzilla.mozilla.org/show_bug.cgi?id=1278719#c31
Assignee | ||
Comment 3•6 years ago
|
||
(In reply to Mike Kaply [:mkaply] from comment #2) > See: > > https://bugzilla.mozilla.org/show_bug.cgi?id=1278719#c31 Thanks for that heads-up. I started working on this but didn't manage to finish in time before the All-hands. It's on my list to complete before the end of Q2. I'll keep you updated.
Assignee | ||
Comment 4•6 years ago
|
||
Some other things took priority at the end of Q2, I'm sorry I have to push this by another week. ETA is now 10th of July.
Assignee | ||
Comment 5•6 years ago
|
||
Cleared up my queue in the meantime, I'm actively working on this bug effective yesterday. My testing will be on https://hg.mozilla.org/projects/maple. Some staging releases will come up there soon.
Assignee | ||
Comment 6•6 years ago
|
||
Status update: trying to make things work on my local machine (macbook) with Docker. I'm using stuff from https://bugzilla.mozilla.org/show_bug.cgi?id=1278719#c31. Once I get that working, I'll be adding tasks similarly to snaps, and have them be created within staging releases on maple.
Assignee | ||
Comment 7•6 years ago
|
||
Status update: * I've got Docker running the flatpacks from here[1] on my local machine * I've extracted some of the installation deps within a local Dockerfile which is later going to be used in-tree * am still hitting some issues when trying to run the flatpak-builder as I don't have the right SDK. Installing it chases the rabbit hole in various GPG, repo, and other related issues. Trying to debug all to figure things out. Once I get it working on my local machine, I'll try to break it down into pieces to add it in-tree and have a staging release re-do the steps. [1]: https://bugzilla.mozilla.org/show_bug.cgi?id=1278719#c31
Assignee | ||
Comment 8•6 years ago
|
||
Status update: after a bunch of hiccups with installation deps, docker privileges, fuse device not found, build & installation issues, I finally managed to reproduce the behavior from here[1] within a Docker container on my local macbook machine. One small step for me, giant leap for flatpaks :) I kept notes of the steps I've done so far and I've committed my containers so I can debug more if I need to. My next short-goal-small-victory is to obtain the Dockerfile in which we need to run the main script to bundle flatpaks. I'll reuse the same logic and approach from snaps. [1]: https://github.com/xhorak/firefox-devedition-flatpak/tree/master/org.mozilla.FirefoxUpstreamBinary
Assignee | ||
Comment 9•6 years ago
|
||
Okay, I managed to get a first version of Dockerfile which sets the groundwork and environment for `flatpak` and `flatpak-builder` commands and solves all the issues I've had earlier with installing the pre-req. Note to self - two quick issues: * the image is huge, 2.7Gb vs ~225 Mb for snaps counterpart. I need to find a way to install the bare minimum to make this work * make sure the flatpak-builder doesn't throw the sdk/platform * still need to find a way to tell Docker via Taskcluster somehow, to run the commands with `docker run --privileged`. This is going to be tricky but I'll poke #taskcluster for some help if I don't find a smart way.
Assignee | ||
Comment 10•6 years ago
|
||
Status update: I've managed to recreate via Dockerfile the environment I used the other day to obtain the first flatpaks and ... I get a nasty segmentation fault. No stacktrace, very difficult to trace. I'm out of ideas for now, I've tried various different ways to get a glimpse of the error, unsuccessfully. My last guess is it could be related to the SDK, potentially. .... Downloading sources Initializing build dir FB: Running 'flatpak build-init --arch=x86_64 /root/app org.mozilla.FirefoxUpstreamBinary org.gnome.Sdk org.gnome.Platform 3.26' Committing stage init to cache Segmentation fault ...
Comment 11•6 years ago
|
||
(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #9) > Created attachment 8993187 [details] [diff] [review] > Dockerfile > > Okay, I managed to get a first version of Dockerfile which sets the > groundwork and environment for `flatpak` and `flatpak-builder` commands and > solves all the issues I've had earlier with installing the pre-req. > > Note to self - two quick issues: > * the image is huge, 2.7Gb vs ~225 Mb for snaps counterpart. I need to find > a way to install the bare minimum to make this work This is likely due to the full GNOME SDK and Platform which are included. The base system for simply running flatpak-builder is tiny. Normally when building a manifest, flatpak-builder pulls all of the required SDK/platform/extension build dependencies. Is it necessary that the SDK is pulled during the Dockerfile build for some reason? If you want to cache the SDK inside your Docker image, you can make this trade, but it has a space cost as you can see. Relatedly: depending how much GNOME stuff you need, you might reduce the build size (and runtime dependency size) by using the freedesktop.org platform, rather than the GNOME platform. The gnome.org platform really does contain every last GNOME library you can imagine - freedesktop.org has the basics like Gtk, GIO, GSettings, Gvfs, GStreamer/ffmpeg, etc. For instance, LibreOffice is able to build/run OK against the freedesktop SDK. Finally, the current correct remote for building/publishing the main (Freedesktop, GNOME and KDE) runtimes is Flathub. You can replace the https://sdk.gnome.org/gnome.flatpakrepo with https://dl.flathub.org/repo/flathub.flatpakrepo (and conventionally this remote is named as flathub). > * make sure the flatpak-builder doesn't throw the sdk/platform What do you mean by this? > * still need to find a way to tell Docker via Taskcluster somehow, to run > the commands with `docker run --privileged`. This is going to be tricky but > I'll poke #taskcluster for some help if I don't find a smart way.
Comment 12•6 years ago
|
||
(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #10) > Status update: I've managed to recreate via Dockerfile the environment I > used the other day to obtain the first flatpaks and ... I get a nasty > segmentation fault. No stacktrace, very difficult to trace. I'm out of ideas > for now, I've tried various different ways to get a glimpse of the error, > unsuccessfully. My last guess is it could be related to the SDK, potentially. > > .... > Downloading sources > Initializing build dir > FB: Running 'flatpak build-init --arch=x86_64 /root/app > org.mozilla.FirefoxUpstreamBinary org.gnome.Sdk org.gnome.Platform 3.26' > Committing stage init to cache > Segmentation fault > ... Nasty! I've not seen this before. At this stage the SDK might not even have done anything - to me this feels more like a problem with the platform/tools than what is being built - like ostree could be doing some FUSE stuff here. Does this problem reproduce on a "normal" Linux box Docker? I've not used the Mac OS X Docker stuff. If you add "main/dbgsym" to the apt sources.list line for the PPA, you should be able to find the -dbgsym packages for ostree and flatpak, which might help getting a backtrace from gdb. (https://wiki.ubuntu.com/DebuggingProgramCrash#Installing_dbgsym_packages_from_a_PPA) The other thing that might help with some "contextual clues" is G_MESSAGES_DEBUG="all" which should give any debug output from ostree, flatpak, etc. Otherwise my normal debug ideas here would be `strace -f -e file` the whole thing to see roughly where it might be failing.
Comment 13•6 years ago
|
||
(In reply to Sylvestre Ledru [:sylvestre] from comment #0) > Flatpak, like Snap, should include all shipped locales. > See https://bugzilla.mozilla.org/show_bug.cgi?id=1297519 for the original > implementation For completeness, it's relatively trivial to do the right thing for splitting up Locales in Flatpaks. The way that Locales would be split out in a Flatpak is to relocate everything into /app/share/runtime/locale/<base_lang_code> folders which was language dependent (where the base lang code is en, pt, etc without any _variants or @dialects) and then symlinking back from wherever Firefox expected it into those paths. You can then turn the .Locale extension on in the Flatpak manifest (separate-locales: true) and it will create an extension which will be subsetted at install time to those languages in use by the user/system. A reasonable example of this is in LibreOffice, where two different trees of language-dependent stuff are moved into subdirectories of the /app/share/runtime/locale folder and symlinked back. On typical 1-2 language systems 100s of MB of download/install space was saved. https://cgit.freedesktop.org/libreoffice/core/tree/solenv/bin/assemble-flatpak.sh#n46 As you can see - one set of locales (C/en) should be kept built-in because the entire Locale extension (ie /app/share/runtime/locale folder and all children) is optional, and the only other caveat is that Firefox must be OK with dangling symlinks where locale stuff is normally found. LibreOffice needed a small patch to skip a dangling symlink in the locale data, for this situation.
Assignee | ||
Comment 14•6 years ago
|
||
@robert: thanks a lot for all your replies, this is very helpful! I'll give it a try using your suggestions. I was traveling on Thursday/Friday, hence the delays in answering, mea culpa.
Assignee | ||
Comment 15•6 years ago
|
||
(In reply to robert from comment #11) > (In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #9) > > Created attachment 8993187 [details] [diff] [review] > > Dockerfile > > > > Okay, I managed to get a first version of Dockerfile which sets the > > groundwork and environment for `flatpak` and `flatpak-builder` commands and > > solves all the issues I've had earlier with installing the pre-req. > > > > Note to self - two quick issues: > > * the image is huge, 2.7Gb vs ~225 Mb for snaps counterpart. I need to find > > a way to install the bare minimum to make this work > > This is likely due to the full GNOME SDK and Platform which are included. > The base system for simply running flatpak-builder is tiny. Normally when > building a manifest, flatpak-builder pulls all of the required > SDK/platform/extension build dependencies. Is it necessary that the SDK is > pulled during the Dockerfile build for some reason? If you want to cache the > SDK inside your Docker image, you can make this trade, but it has a space > cost as you can see. > > Relatedly: depending how much GNOME stuff you need, you might reduce the > build size (and runtime dependency size) by using the freedesktop.org > platform, rather than the GNOME platform. The gnome.org platform really does > contain every last GNOME library you can imagine - freedesktop.org has the > basics like Gtk, GIO, GSettings, Gvfs, GStreamer/ffmpeg, etc. For instance, > LibreOffice is able to build/run OK against the freedesktop SDK. > > Finally, the current correct remote for building/publishing the main > (Freedesktop, GNOME and KDE) runtimes is Flathub. You can replace the > https://sdk.gnome.org/gnome.flatpakrepo with > https://dl.flathub.org/repo/flathub.flatpakrepo (and conventionally this > remote is named as flathub). This worked like a charm, thanks a lot for suggesting this! I wasn't sure how to tell flatpak-builder to do that but sounds like https://github.com/flatpak/flatpak/issues/1445#issuecomment-373347884 was what I was missing. I tweaked the template manifest to point to the freedesktop sdk/platform installations in conjungtion to telling flatpak-builder to use `--install-deps-from=flathub`, where flathub is defined in Dockerfile via `flatpak remote-add --from flathub https://dl.flathub.org/repo/flathub.flatpakrepo` flatpak-builder --install-deps-from=flathub --assumeyes --disable-rofiles-fuse --disable-cache --force-clean --repo=repo app org.mozilla.FirefoxUpstreamBinary.json Image contains the bare minimum at is down to 474 Mb! Platform/SDK installation now happen at runtime which is exactly what I wanted. This is no longer an issue, moving on to the next one. Still end up with segmentation fault so I'll proceed with your suggestions in improving the stacktrack debugging. > > * still need to find a way to tell Docker via Taskcluster somehow, to run > > the commands with `docker run --privileged`. This is going to be tricky but > > I'll poke #taskcluster for some help if I don't find a smart way. Right now, I start the docker containers with `--privileged` which spares me of some nasty fuse errors. With Taskcluster, I won't be able to run that given the mechanics under the hood so I'll likely end up with those fuse errors. But so far I have the segmentation fault issue to deal with first.
Assignee | ||
Comment 16•6 years ago
|
||
Note to self: I could try using a different version too. Maybe there's something wrong with the existing installer builders.
Assignee | ||
Comment 17•6 years ago
|
||
Could be something wrong with my Macbook-based Docker containers. Johan helped me test this on his native Linux box and sounds like he's getting the following error: Building module firefox in /work/firefox-devedition-flatpak/org.mozilla.FirefoxUpstreamBinary/.flatpak-builder/build/firefox-1 ======================================================================== tar (child): bzip2: Cannot exec: No such file or directory tar (child): Error is not recoverable: exiting now tar: Child returned status 2 tar: Error is not recoverable: exiting now Error: module firefox: Child process exited with code 2 + flatpak build-bundle repo org.mozilla.FirefoxUpstreamBinary.flatpak org.mozilla.FirefoxUpstreamBinary error: 'repo' is not a valid repository * which still fails but moves past the seg fault I was getting. * he's also using Docker version 18.05.0-ce, build f150324782 while I have Docker version 18.03.1-ce, build 9ee9f40 so there could be something related to that as well, not sure. * he ran all without the privileged flag which is great news! Next steps: * move stuff in-tree and rely on docker-workers for remaining of the testing * iterate there to make these operational I'll be back with updates, work-in-progress.
Assignee | ||
Comment 18•6 years ago
|
||
Status update: * I managed to get them working on my local environments on Macbook too. * I've added tasks to get them in-tree and I'm currently iterating on maple within staging releases * Docker-image-generation task is green[1], creates the correct image here[2] * flatpak bundle-generation task is here[3]. Looks promising so far but hasn't run as we have some unrelated issues in the staging environment Task should go green and generate a single-locale flatpak. I'm working on adding support for multi-locale now. [1]: https://tools.taskcluster.net/groups/RYL82EquQDyilRvUdALHSg/tasks/KaTBqnEtTrqVf5LbNpATVg/details [2]: https://queue.taskcluster.net/v1/task/KaTBqnEtTrqVf5LbNpATVg/runs/0/artifacts/public/image.tar.zst [3]: https://tools.taskcluster.net/groups/Svy8QGRdSLigc1ON0kyI7w/tasks/DpbURMhdSiGg8cBcIy7Lwg/details
Comment 19•6 years ago
|
||
(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #18) > Status update: > * I managed to get them working on my local environments on Macbook too. > * I've added tasks to get them in-tree and I'm currently iterating on maple > within staging releases > * Docker-image-generation task is green[1], creates the correct image here[2] > * flatpak bundle-generation task is here[3]. Looks promising so far but > hasn't run as we have some unrelated issues in the staging environment > > Task should go green and generate a single-locale flatpak. I'm working on > adding support for multi-locale now. This is great news! I have some feedback for the Flatpak build process - I found the .json and stuff in https://hg.mozilla.org/projects/maple/file/tip/taskcluster/docker/firefox-flatpak but I am unfamiliar with Mozilla contribution processes. Is there a pull request or something that I can leave review comments against? Or should I just list out lines/comments here? One thing which we will need to solve in order to publish things normally, especially if we split the different locales out using a .Locale extension, is that a Flatpak bundle can only contain one ref - but a properly split-out Locale extension would add a 2nd ref. Typically to do normal/idiomatic Flatpak installs and upgrades, Flatpak needs to have a ostree repo set up as a remote to pull from - downloading and installing a bundle is kind of a "one off" installation, but to get notified of updates in your app center the newer commits and other refs need to be published in a repo. Does Mozilla plan to also publish a Flatpak repo, or would we set up a process for exporting the bundles from Mozilla into Flathub so that we would publish/mirror the latest through a regular repo? In Flathub we actually don't use bundles - the implementation is kind of lousy (they take loads of RAM during generation) - so we just tar up the whole result of the flatpak-builder repo dir and move that from the build worker to the repo master, and import things to the main repo that way. (I've also joined irc.mozilla.org, nick ramcq, if there is a good channel for us to chat about Flatpak?)
Assignee | ||
Comment 20•6 years ago
|
||
(In reply to Robert McQueen from comment #19) > (In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #18) > > Status update: > > * I managed to get them working on my local environments on Macbook too. > > * I've added tasks to get them in-tree and I'm currently iterating on maple > > within staging releases > > * Docker-image-generation task is green[1], creates the correct image here[2] > > * flatpak bundle-generation task is here[3]. Looks promising so far but > > hasn't run as we have some unrelated issues in the staging environment > > > > Task should go green and generate a single-locale flatpak. I'm working on > > adding support for multi-locale now. > > This is great news! Thanks for quick turnaround. Let me reply inline. 1. Maple is a project branch used by RelEng for testing so I'm merely iterating in staging conditions how this would look lile. 2. What I have so far is https://hg.mozilla.org/projects/maple/rev/0cf2dd5ccd6f but the patch is far away from being production-ready. 3. My immediate goal is to get some multi-locale flatpaks generated in-tree via staging releases and then dogfood the app in iterations to fix all the glitches. Right now there's still plenty of hardcodes and stuff, I agree. I'll try to address them one by one. I've got more stuff locally and a list of fixes/hardcodes I need to address and the list is growing :) 4. Thanks for your help and suggestions with respect to localization and flatpak publishing. > I have some feedback for the Flatpak build process - I found the .json and > stuff in > https://hg.mozilla.org/projects/maple/file/tip/taskcluster/docker/firefox- > flatpak but I am unfamiliar with Mozilla contribution processes. Is there a > pull request or something that I can leave review comments against? Or > should I just list out lines/comments here? If patches are in-tree, using this bug and sending patches here are good ways to start I suppose. https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Introduction contains more verbose information about this. > One thing which we will need to solve in order to publish things normally, > especially if we split the different locales out using a .Locale extension, > is that a Flatpak bundle can only contain one ref - but a properly split-out > Locale extension would add a 2nd ref. > > Typically to do normal/idiomatic Flatpak installs and upgrades, Flatpak > needs to have a ostree repo set up as a remote to pull from - downloading > and installing a bundle is kind of a "one off" installation, but to get > notified of updates in your app center the newer commits and other refs need > to be published in a repo. > > Does Mozilla plan to also publish a Flatpak repo, or would we set up a > process for exporting the bundles from Mozilla into Flathub so that we would > publish/mirror the latest through a regular repo? In Flathub we actually > don't use bundles - the implementation is kind of lousy (they take loads of > RAM during generation) - so we just tar up the whole result of the > flatpak-builder repo dir and move that from the build worker to the repo > master, and import things to the main repo that way. However, for more detailed questions regarding the process of publishing this, I'm not sure I'm the best person to ask. As a first step, my thoughts were to simply publish this within our S3 buckets, or something. Publishing it properly is a separate job as far as I can tell(not sure if that involves signing as well), most likely outside of this bug scope. Let me redirect that question to :Sylvestre, he might have more insight. (he's currently PTO until the 12th of August) > (I've also joined irc.mozilla.org, nick ramcq, if there is a good channel > for us to chat about Flatpak?) I'm mtabara on IRC and can be found in various channels. The one I'm using for debugging flatpaks is #tcmigration.
Comment 21•6 years ago
|
||
(In reply to Robert McQueen from comment #19) > (I've also joined irc.mozilla.org, nick ramcq, if there is a good channel > for us to chat about Flatpak?) We opened #flatpaks if anyone wants to follow along/join in.
Assignee | ||
Comment 22•6 years ago
|
||
Status update: * I'm still WIP to add langpacks to the single-locale flatpak and land that on maple to iterate * more conversations on flatpaks now happen in #flatpaks on IRC * The flatpak generation looks very very promising in the staging release but it's currently failing for some priviledge issues within Docker. Task is here[1]. The rest of things worked smoothly so if I manage to unblock that[2] we will he having single-locale flatpaks for now. I'll continue to polish patches and get them to a more review-able state. Stay tuned for more. [1]: https://tools.taskcluster.net/groups/XAgbo8fTRnKfWDyKUA72rQ/tasks/JuBubPToRhCp9YLqFmn5_w/details [2]: Note: I'm getting `bwrap permissions errors`. On local machine I managed to overcome that by running docker run in privideleged mode but that's very dangerous as it exposes all devices of host. Most likely impossible to reproduce on docker-worker machines for security reasons. Am working with Taskcluster team to unblock on that and find a workaround.
Assignee | ||
Comment 23•6 years ago
|
||
I spoke with some TC folks and dropped a message to their mailing list to ask for some help. Apparently, it's not trivial to solve this (playing with the container privileges in Taskcluster). We have a backup solution if worse comes to worst (to have dedicated workers purely for this that can have dedicated rights) but I'd like to clear other better solutions (if any) before we default to that. In the meantime, I'm trying to get the langpacks bundled within the en-US to reach the milestone I had promised initially. Will pause and re-evaluate timeline after that.
Comment 24•6 years ago
|
||
So, inside flatpak-builder / flatpak, the privileged operations for setting up a container are handled by bubblewrap which is setuid on most systems. The main reason for this is that unpriveleged users are not allowed to open new namespaces on most distro kernels. However, this is a kernel setting, and if you switch to a recent enough bubblewrap, you might not need it to be priveleged - the last 0.3.0 release of bubblewrap (https://github.com/projectatomic/bubblewrap/releases/tag/v0.3.0) allows the namespaces and filesystems to be set up without any priveleged operations. So, if the seccomp filters can be relaxed somewhat in the Docker configuration, and the machines permit user namespaces to be created, you might be OK!
Comment 25•6 years ago
|
||
ramcq | is there a "recipe" on how to run bubblewrap inside of docker, based off | https://github.com/projectatomic/bubblewrap/releases/tag/v0.3.0 ? ramcq | Mozilla are looking at integrating flatpak builds into their docker-based task runner infra ramcq | would be good if I could say, sysctl yada yada, echo foo >Dockerfile ramcq | https://bugzilla.mozilla.org/show_bug.cgi?id=1441922#c22 walters | ramcq: bwrap's CI does it, see: https://github.com/projectatomic/bubblewrap/blob/master/.papr.yml#L22 walters | then follow it into https://github.com/projectatomic/bubblewrap/blob/master/ci/papr.sh walters | currently it passes `--privileged` but that is probably a bug; it's sufficient to do `--security-opt seccomp=unconfined`
Assignee | ||
Comment 26•6 years ago
|
||
Status update: * I have single-locale bundled with langpacks ready working on my local Docker * still having trouble reproducing that in the Taskcluster Docker containers for permissions errors. I moved a bit forward with some help from TC team but am still few hops from victory Once I manage to get unblocked there and have a successful run, I'll hand this over to testing and pause for further discussions with respect to: * publishing * reviewing * optimizing
Assignee | ||
Comment 27•6 years ago
|
||
Status update: * just to clarify - things are working properly on my Docker environment - multi-locale flatpaks (en-US bundled with langpacks) ready for testing. * I'm still having difficulties translating that into TC. I've tried different provisioners/workerTypes and various other solutions. Am in touch with them to find an alternative way, most likely running them in a dedicated workerType As soon as I'm unblocked on TC side, jobs should just be green on maple, code is already landed.
Assignee | ||
Comment 28•6 years ago
|
||
Green task in Taskcluster! Task is here[1], flatpak bundle is here[2]. More artifacts here[3] (signing manifest, checksums and others). The bundle is bundled with en-US + langpacks. Can be installed with `flatpak install target.flatpak`. I'll pause here for a bit to test this for now and suggest future areas of improvement. Leftover from my side: * get this polished in-tree and potentially landed if we get green go * still figure it out how can I bake the version/build number for the firefox I'm building. Currently I don't think I am doing that * chain this task to the generate-checksums-beetmover one to make sure it runs after the SHA512 sums is done. [1]: https://tools.taskcluster.net/groups/Id3xHl2KTbGkxMqblk8ZNg/tasks/GhbsyyHqS1OaCEXuOyhlmA/details [2]: https://queue.taskcluster.net/v1/task/GhbsyyHqS1OaCEXuOyhlmA/runs/0/artifacts/public/build/target.flatpak [3]: https://tools.taskcluster.net/groups/Id3xHl2KTbGkxMqblk8ZNg/tasks/GhbsyyHqS1OaCEXuOyhlmA/runs/0/artifacts
Assignee | ||
Comment 29•6 years ago
|
||
:robert@mcqueen.me.uk Out of curiosity, would you know if/how to bake the version/build number in flatpaks? Right now I don't think that's baked anywhere in the configs. The en-US tarball that we use and download is correctly pulled from S3 (via env var BUILD_NUMBER and VERSION in https://tools.taskcluster.net/groups/Id3xHl2KTbGkxMqblk8ZNg/tasks/GhbsyyHqS1OaCEXuOyhlmA/details) but I'm wondering if there's any other place we need to tweak that. Or does that suffice?
Comment 30•6 years ago
|
||
(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #29) > :robert@mcqueen.me.uk > > Out of curiosity, would you know if/how to bake the version/build number in > flatpaks? Right now I don't think that's baked anywhere in the configs. > > The en-US tarball that we use and download is correctly pulled from S3 (via > env var BUILD_NUMBER and VERSION in > https://tools.taskcluster.net/groups/Id3xHl2KTbGkxMqblk8ZNg/tasks/ > GhbsyyHqS1OaCEXuOyhlmA/details) but I'm wondering if there's any other place > we need to tweak that. Or does that suffice? If you add <releases> with a <release> entry, the version number and sometimes the release notes will be shown in GNOME Software, Flathub etc. https://www.freedesktop.org/software/appstream/docs/chap-Metadata.html#tag-releases
Assignee | ||
Comment 31•6 years ago
|
||
Assignee | ||
Updated•6 years ago
|
Assignee | ||
Comment 32•6 years ago
|
||
Comment on attachment 9002794 [details] [diff] [review] [in-tree] Generate multilocale flatpaks As we discussed, adding you as a reviewer here. I guess the interesting bits that really matter are the: * taskcluster/docker/firefox-flatpak/Dockerfile - where the initial docker image is being created with all the neccesary bits for building flatpaks * taskcluster/docker/firefox-flatpak/Makefile to customize and massage the Firefox installation to fit some of the preferences and customationzations needed * taskcluster/docker/firefox-flatpak/default-preferences.js * taskcluster/docker/firefox-flatpak/distribution.ini * taskcluster/docker/firefox-flatpak/flatpak.json.in - manifest for flatpak * taskcluster/docker/firefox-flatpak/runme.sh - where all the magic happens Let me know if I can help with more information. Thank you again for this!
Comment 33•6 years ago
|
||
Comment on attachment 9002794 [details] [diff] [review] [in-tree] Generate multilocale flatpaks Review of attachment 9002794 [details] [diff] [review]: ----------------------------------------------------------------- I've given a quick review to the 'flatpak' and desktop integration parts. Rob is still best for the build parts. I'll try and build it myself and dig a bit deeper into the d-bus sandboxing and see if we can improve the confinement there. ::: taskcluster/docker/firefox-flatpak/Makefile @@ +6,5 @@ > + mkdir -p $(prefix)/share/appdata > + cp -p org.mozilla.Firefox.appdata.xml $(prefix)/share/appdata > + desktop-file-install --dir $(prefix)/share/applications org.mozilla.Firefox.desktop > + mkdir -p $(prefix)/share/icons/hicolor/128x128/apps/ $(prefix)/share/appdata > + cp -p firefox/browser/chrome/icons/default/default128.png $(prefix)/share/icons/hicolor/128x128/apps/org.mozilla.Firefox.png Export at least the 64px version as well, that's used by some software stores and otherwise you'll get an automatically resized version which I'm sure the designers wouldn't like. Generally I'd recommend exporting all the icons you have (16, 32, 48, 64, 128). ::: taskcluster/docker/firefox-flatpak/flatpak.json.in @@ +1,4 @@ > +{ > + "app-id": "org.mozilla.Firefox", > + "runtime": "org.freedesktop.Platform", > + "runtime-version": "1.6", There's a new FDo release - 18.08 which should be used now, the older version is out of security support. 18.08 will have two years of security support and be stable within that period (see https://gitlab.com/freedesktop-sdk/freedesktop-sdk/wikis/release for details) @@ +3,5 @@ > + "runtime": "org.freedesktop.Platform", > + "runtime-version": "1.6", > + "sdk": "org.freedesktop.Sdk", > + "command": "firefox", > + "rename-appdata-file" : "org.mozilla.Firefox.appdata.xml", Why do you need to rename something that already has the correct naming? @@ +4,5 @@ > + "runtime-version": "1.6", > + "sdk": "org.freedesktop.Sdk", > + "command": "firefox", > + "rename-appdata-file" : "org.mozilla.Firefox.appdata.xml", > + "rename-icon" : "org.mozilla.Firefox", Same as above, not sure why you need to rename something that already has the correct naming. @@ +9,5 @@ > + "finish-args": [ > + /* X11 + XShm access */ > + "--share=ipc", "--socket=x11", > + /* Wayland access */ > + "--socket=wayland", Do you want Firefox to be wayland native (if so, awesome!)? If this is not a supported configuration then you can just remove this and it'll use XWayland by default. @@ +19,5 @@ > + "--socket=pulseaudio", > + /* Enable access to dri */ > + "--device=dri", > + /* Enable themes support */ > + "--filesystem=~/.config/dconf:ro", This hardcoding can be wrong on some systems, you need xdg-run/dconf as noted below. @@ +20,5 @@ > + /* Enable access to dri */ > + "--device=dri", > + /* Enable themes support */ > + "--filesystem=~/.config/dconf:ro", > + "--env=DCONF_USER_CONFIG_DIR=.config/dconf", For dconf support you need all of: "--filesystem=xdg-run/dconf", "--filesystem=xdg-config/dconf:ro", "--talk-name=ca.desrt.dconf", "--env=DCONF_USER_CONFIG_DIR=.config/dconf" @@ +22,5 @@ > + /* Enable themes support */ > + "--filesystem=~/.config/dconf:ro", > + "--env=DCONF_USER_CONFIG_DIR=.config/dconf", > + /* Required for file reveal in file manager */ > + "--socket=session-bus" These permissions are super broad. You should run the flatpak under --log-session-bus and whitelist the specific D-Bus endpoints. (I will try and build this locally myself and do exactly that but didn't want to hold up the review. @@ +28,5 @@ > + "modules": [ > + { > + "name": "firefox", > + "no-autogen": true, > + "make-install-args": ["prefix=/app"], given the way you use a Makefile here it'd be more idiomatic flatpak to use "buildsystem": "simple" and putting each line inside a "build-commands" block. (see http://docs.flatpak.org/en/latest/flatpak-builder-command-reference.html for more on that). It's just a recommendation though, if your team is more comfortable with Make, then carry on. ::: taskcluster/docker/firefox-flatpak/org.mozilla.Firefox.appdata.xml @@ +1,1 @@ > +<?xml version="1.0" encoding="UTF-8"?> High quality appdata files require screenshots, release tags and OARS https://hughsie.github.io/oars/ @@ +1,3 @@ > +<?xml version="1.0" encoding="UTF-8"?> > +<application> > + <id type="desktop">org.mozilla.Firefox.desktop</id> you shouldn't need the .desktop suffix anymore. Just org.mozilla.Firefox is better. @@ +9,5 @@ > + <p> > + Firefox is a Web browser that is smaller, faster, and in some ways more secure than other competitors. Firefox gives users a cleaner interface and faster download speeds. > + </p> > + <p> > + Firefox is repack of upstream binaries to the flatpak. This text is a bit ungainly! @@ +17,5 @@ > + <keyword>mozilla</keyword> > + <keyword>internet</keyword> > + <keyword>web</keyword> > + </keywords> > + <url type="homepage">https://www.mozilla.org/en-US/firefox/new/</url> You should also include url type=donation (likely pointing to https://donate.mozilla.org/ ) and url type=bugtracker pointing to the obvious locations
Comment 34•6 years ago
|
||
This comment is coming at you from Flatpak-ed firefox! Exciting. The bundle linked above doesn't actually work out of the box, which is a bit of a shame: > libdbus-glib-1.so.2: cannot open shared object file: No such file or directory dbug-glib isn't part of the base Freedesktop Platform. There are two possible ways to fix this. You can either do what LibreOffice does and build your own copy of dbus-glib and bundle it into your flatpak (in which case, copy and paste the code here into the manifest: https://github.com/flathub/org.libreoffice.LibreOffice/blob/master/org.libreoffice.LibreOffice.json#L46 ) Or you could switch the manifest to use the GNOME Platform instead, which is a superset of the Freedesktop one and does have that installed. I tested both the stable 3.28 release and the master release (that will be 3.30) and they both work fine. 3.28 uses Freedesktop 1.6 as it's base, the master release has switched over the the new 18.08 base. If you want to test that yourself then you need to run: > flatpak run --runtime=org.gnome.Platform//3.28 org.mozilla.Firefox I also tested the open folder implementation using the following command: > flatpak run --runtime=org.gnome.Platform//3.28 --log-session-bus --unshare=session-bus org.mozilla.Firefox You should absolutely remove the --share=session-bus permission and replace it with: "--talk-name=org.freedesktop.FileManager1" "--talk-name=org.gtk.vfs.*" "--talk-name=org.freedesktop.Notifications" in the finish-args. I'll keep testing and see if there's anything else that needs to talk dbus to the system, if you know of it then it's really easy to hunt down with the logger, although it's somewhat chatty on the command line.
Comment 35•6 years ago
|
||
In order to take advantage of Flatpak's sandboxing, Firefox will need to use the `xdg-desktop-portal` D-Bus API for file selectors. This will have the added advantage of making Firefox use native file selectors in KDE environments. I'm actually interested in implementing this functionality, but I'm new to Firefox contributions. Should I open another issue for that particular piece?
Reporter | ||
Comment 36•6 years ago
|
||
(In reply to Nathan Moos from comment #35) > I'm actually interested in implementing this functionality Great, thanks! > Should I open another issue for that particular piece? Yes, bugs are cheap.
Comment 37•5 years ago
|
||
General Flatpak question. What's the easiest way to know we are in a Flatpak? Snap sets an environment variable: https://searchfox.org/mozilla-central/source/browser/components/shell/nsGNOMEShellService.cpp#90 Is there something similar for Flatpak?
Comment 38•5 years ago
|
||
There's a /.flatpak-info file in the root of every sandbox. This contains a bunch of information about the flatpak environment you're in. (see more here: https://blogs.gnome.org/mclasen/2018/07/02/flatpak-in-detail-part-3/) There's also a /app/.updated file that tells you that an update is ready to be installed. It'd be wonderful if Firefox could hook its 'restart to update' stuff into that. (more on using that here https://blogs.gnome.org/mclasen/2018/11/26/an-update-on-flatpak-updates/ )
Comment 39•5 years ago
|
||
Bug 1411579 already has code for detecting a Flatpak sandbox environment, so you should probably share that function. It is based on the way GTK itself detects such an environment: https://github.com/GNOME/gtk/blob/e0ce028c88858b96aeda9e41734a39a3a04f705d/gtk/gtkprivate.c#L272
Comment 40•5 years ago
|
||
Note, if this turns out to be easier to run in generic-worker with multiuser engine (separate OS users per task) rather than docker-worker, the work to add multiuser engine to linux is in bug 1499054. That bug isn't a huge amount of work, but note we may have some competing priorities (notably bug 1499055). A priority call should probably be made about whether we should tackle that now.
We would also need to set up a linux worker with generic-worker, which for this we could do manually in EC2, and take a snapshot, as we currently don't have automation around building EC2 AMIs for generic-worker linux workers.
Comment 41•5 years ago
|
||
(In reply to Pete Moore [:pmoore][:pete] from comment #40)
Note, if this turns out to be easier to run in generic-worker with multiuser engine (separate OS users per task) rather than docker-worker, the work to add multiuser engine to linux is in bug 1499054. That bug isn't a huge amount of work, but note we may have some competing priorities (notably bug 1499055). A priority call should probably be made about whether we should tackle that now.
Based on my understanding of the situation, if this isn't a huge amount of work, I think it's worth the detour. Essentially, the flatpak-builder workload brings its own containerization and it's the nesting of these within docker which causes the problem. Because the actual build workload is in any case sandboxed by flatpak, this trouble is avoidable.
We would also need to set up a linux worker with generic-worker, which for this we could do manually in EC2, and take a snapshot, as we currently don't have automation around building EC2 AMIs for generic-worker linux workers.
Is it possible to arrange that these images can have certain dependencies installed (ie flatpak and flatpak-builder) or is something else (a buildroot?) required to bring additional binaries/dependencies into the multiuser engine to run the actual build jobs? As well as the competing containerisation, in this workload having our own Docker image is also allowing us to install our needed dependencies into the sandbox.
Assignee | ||
Comment 42•5 years ago
|
||
Status update on this: workflow is ready on our testing trees but some of the underlying tooling prevents us to ship to production for security reasons. We're working on this with RelOps and Taskcluster teams to find a solution to sort this out in early Q3.
Comment 43•5 years ago
|
||
(In reply to Robert McQueen from comment #41)
(In reply to Pete Moore [:pmoore][:pete] from comment #40)
Note, if this turns out to be easier to run in generic-worker with multiuser engine (separate OS users per task) rather than docker-worker, the work to add multiuser engine to linux is in bug 1499054. That bug isn't a huge amount of work, but note we may have some competing priorities (notably bug 1499055). A priority call should probably be made about whether we should tackle that now.
Based on my understanding of the situation, if this isn't a huge amount of work, I think it's worth the detour. Essentially, the flatpak-builder workload brings its own containerization and it's the nesting of these within docker which causes the problem. Because the actual build workload is in any case sandboxed by flatpak, this trouble is avoidable.
We would also need to set up a linux worker with generic-worker, which for this we could do manually in EC2, and take a snapshot, as we currently don't have automation around building EC2 AMIs for generic-worker linux workers.
Is it possible to arrange that these images can have certain dependencies installed (ie flatpak and flatpak-builder) or is something else (a buildroot?) required to bring additional binaries/dependencies into the multiuser engine to run the actual build jobs? As well as the competing containerisation, in this workload having our own Docker image is also allowing us to install our needed dependencies into the sandbox.
I think for initial rollout, we can prepare an image with everything required, and we can make the installation steps transparent by linking to e.g. a puppet config or setup script from the task log header. Long term, if the system dependencies change regularly, we may want to look into a simpler way to apply changes, but for the meantime, I suspect adding system dependencies that require root to install will need a PR to the repo that manages the puppet configs (or some other system configuration tool that Release Operations decides to use for these workers).
Assignee | ||
Comment 44•5 years ago
|
||
(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #42)
Status update on this: workflow is ready on our testing trees but some of the underlying tooling prevents us to ship to production for security reasons. We're working on this with RelOps and Taskcluster teams to find a solution to sort this out in early Q3.
Status update: attempt at fixing ^ is going to happen late in July in bug 1499054.
Assignee | ||
Comment 45•5 years ago
|
||
Bug 1499054 got fixed, we can now resurrect the work in here to enable flatpaks via generic-workers.
Assignee | ||
Comment 46•5 years ago
•
|
||
Met with @walac @pmoore and @fubar earlier today to talk about next steps. I was worried that there wasn't a clear way of provisioning the generic-workers for Linux in order to build the pre-requirements for building flatpaks. But sounds like there is \o/
So next steps for me:
- reuse one of the existing generic-workers (e.g. https://github.com/taskcluster/generic-worker/tree/master/worker_types/gwci-linux) in order to create a flatpak workerType
- make sure I can deploy on my own so that I can be self-serve in deploying new changes
- adjust the https://github.com/taskcluster/generic-worker/blob/master/worker_types/gwci-linux/userdata to encompass whatever we used to build a successful Docker container - https://gist.github.com/MihaiTabara/8fc247e477683aa4e71d30a4b7f3a470
- once we have a good workerType, trigger staging releases in order to test the building of the flatpaks
- once that works, add task to ship-it to the Flathub
Talked to @tomprince earlier in the week. I initially thought we can get away with some docker-worker + TC secrets to push but we'll need to use a scriptworker. So that means we'll need to build something similar to https://github.com/mozilla-releng/pushsnapscript but more leightweight since we can use cli-ed commands. The task itself should only pass as arguments the channel
.
Other useful resources:
- https://github.com/taskcluster/generic-worker#set-up-your-env
- https://github.com/taskcluster/mozilla-history/blob/master/AWSWorkerTypes/gwci-linux#L63
- https://tools.taskcluster.net/aws-provisioner/gwci-linux/view
- https://github.com/taskcluster/generic-worker/blob/master/worker_types/gwci-linux/userdata
Comment 47•5 years ago
|
||
(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #46)
I'm little concerned that you mention only infrastructure changes but there is no update about actual flatpak manifest. Could you post updated manifest in attachment so people could review it? It would be nice to base it on top of recently released freedesktop 19.08 runtime with two years of support.
Assignee | ||
Comment 48•5 years ago
|
||
(In reply to Emerson Bernier from comment #47)
(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #46)
I'm little concerned that you mention only infrastructure changes but there is no update about actual flatpak manifest. Could you post updated manifest in attachment so people could review it? It would be nice to base it on top of recently released freedesktop 19.08 runtime with two years of support.
Thanks for raising this! Just to confirm, you mean upgrading from 18.08 which is what we were using to 19.08? If so, that makes a lot of sense. My main goal is to make sure the generic-worker infrastructure can built us what the docker-worker infrastructure did. Then, it should be a matter of adjusting those requirements and get up-to-date with most recent stable releases.
Comment 49•5 years ago
|
||
(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #48)
Thanks for raising this! Just to confirm, you mean upgrading from 18.08 which is what we were using to 19.08?
Yes. Also I think some permissions listed in "finish-args" in manifest (https://bug1441922.bmoattachments.org/attachment.cgi?id=9002794) may be dropped:
"--filesystem=~/.config/dconf:ro" and "--env=DCONF_USER_CONFIG_DIR=.config/dconf" (see https://blogs.gnome.org/mclasen/2019/07/12/settings-in-a-sandbox-world/)
"--filesystem=home:rw" can be replaced by "--env=GTK_USE_PORTAL=1" which will also make possible opening file like pdf in external apps. In that case you may add "--filesystem=~/.mozilla" to keep access for existing profiles.
"--socket=session-bus" can be replaced by "--talk-name=org.a11y.*" and "--talk-name=org.freedesktop.Notifications".
Comment 50•5 years ago
|
||
@Emerson Bernier: Some notes on your proposal:
"--filesystem=home:rw" can be replaced by "--env=GTK_USE_PORTAL=1" which will also make possible opening file like pdf in external apps.
Setting "--env=GTK_USE_PORTAL=1" should not be necessary as GTK has a built-in check for detecting sandboxed environments. Afaik, Firefox calls the corresponding GTK API to detect sandboxed environments?
You should probably also keep "--filesystem=home:ro" (read-only, but not writable anymore) so that file:///-URIs continue to work. (Unless you actually don't want to support that anymore…)
"--socket=session-bus" can be replaced by "--talk-name=org.a11y.*" and "--talk-name=org.freedesktop.Notifications".
libnotify/glib on 19.08 may be using the org.freedesktop.portal.Notification interface already. In this case you'll probably not need the second permission either.
Comment 51•5 years ago
|
||
(In reply to ntninja from comment #50)
Setting "--env=GTK_USE_PORTAL=1" should not be necessary as GTK has a built-in check for detecting sandboxed environments. Afaik, Firefox calls the corresponding GTK API to detect sandboxed environments?
You are right, in flatpak, portals seems working automatically in FF.
You should probably also keep "--filesystem=home:ro" (read-only, but not writable anymore) so that file:///-URIs continue to work. (Unless you actually don't want to support that anymore…)
I would say that home/host permissions were a stop-gap solution until apps gain portal support. They're quite an abuse from security/privacy POV. Also leaking configs from host may cause various issues, for example some special .asoundrc can break alsa in sandbox. It's Mozilla call.
"--socket=session-bus" can be replaced by "--talk-name=org.a11y.*" and "--talk-name=org.freedesktop.Notifications".
libnotify/glib on 19.08 may be using the org.freedesktop.portal.Notification interface already. In this case you'll probably not need the second permission either.
I can't confirm this. Without "--talk-name=org.freedesktop.Notifications" I get only non-native notifications. I tested this on Thunderbird 68.1 in KDE 5.16 session. BTW: libnotify doesn't exists in runtime so Mozilla has to bundle it in manifest themselves in order to use it.
Anyway, this is kind of discussion I hoped for before it will be set & forget :)
Comment 52•5 years ago
|
||
You should probably also keep "--filesystem=home:ro" (read-only, but not writable anymore) so that file:///-URIs continue to work. (Unless you actually don't want to support that anymore…)
I would say that home/host permissions were a stop-gap solution until apps gain portal support. They're quite an abuse from security/privacy POV. Also leaking configs from host may cause various issues, for example some special .asoundrc can break alsa in sandbox. It's Mozilla call.
I agree about it being Mozilla's call. Just wanted point out that without "--filesystem=home:ro" hardly any local HTML file will be able to be opened correctly in the browser anymore, as the document portal will only mirror the HTML file itself but none of the auxiliary files (think the typical {page_name}_files
directory created by offline-safes, …) it may use. NOT having this option set will effectively disable the ability to view any local HTML file correctly.
(I just wanted to have this particular fact pointed out in writing before anybody makes this decision lightly.)
Additional read-only exceptions for portable storage ("--filesystem=/media:ro") and system docs ("--filesystem=/usr/share/docs:ro --filesystem=/usr/local/share/docs:ro --filesystem=/opt:ro") should probably also be considered in this context as well as there are many (non-sandboxed) applications were “View help” will simply open the given page using the default browser/handler.
BTW: I'm not really a fan of adding all these exceptions, but if this shouldn't break existing usage I fear these exceptions may be necessary to keep around for a very very very long time.
"--socket=session-bus" can be replaced by "--talk-name=org.a11y.*" and "--talk-name=org.freedesktop.Notifications".
libnotify/glib on 19.08 may be using the org.freedesktop.portal.Notification interface already. In this case you'll probably not need the second permission either.
I can't confirm this. Without "--talk-name=org.freedesktop.Notifications" I get only non-native notifications. I tested this on Thunderbird 68.1 in KDE 5.16 session. BTW: libnotify doesn't exists in runtime so Mozilla has to bundle it in manifest themselves in order to use it.
Your right, I just checked the libnotify source-code and it pretty plain obvious that nobody tried to address this (or pretty much any other) issue yet.
GIO/GNotification has support for this, but as you have stated you are not using this.
Since libnotify is used by many applications, I guess it's best to just keep using the code and sandbox exception for now and hope that someone (maybe sponsored?) will create a patched version of this library that will “just work” as a drop-in replacement.
Comment 53•5 years ago
|
||
(In reply to ntninja from comment #52)
Additional read-only exceptions for portable storage ("--filesystem=/media:ro") and system docs ("--filesystem=/usr/share/docs:ro --filesystem=/usr/local/share/docs:ro --filesystem=/opt:ro") should probably also be considered in this context as well as there are many (non-sandboxed) applications were “View help” will simply open the given page using the default browser/handler.
BTW: I'm not really a fan of adding all these exceptions, but if this shouldn't break existing usage I fear these exceptions may be necessary to keep around for a very very very long time.
Users could still add more permissions to firefox flatpak locally (opt-in) however unlike snap flatpak doesn't have UX for adding more permissions easily so maybe this should be opt-out by default. In that case I would prefer single "--filesystem=host:ro" which will be easier to override that multiple of specific dirs.
There is also problem with /tmp - while opening files in external apps firefox download them to /tmp or TMPDIR at first. As /tmp from sandbox is inaccessible outside it, external app won't be able to find and open the file. This can be solved either by:
- "--filesystem=/tmp" - easiest but least secure solution
- Providing /app/bin/firefox[1] wrapper which sets TMPDIR to something readable. For example fedora sets[2] TMPDIR="$XDG_CACHE_HOME/tmp
[1] It can't be done with --env=TMPDIR= permissions because it won't work on systems where bubblewrap is installed as setuid binary, see https://github.com/flatpak/flatpak/issues/2641
[2] https://src.fedoraproject.org/rpms/firefox/blob/0adc4aecec45e195b0f6cbcdb1625839846482bc/f/firefox.spec#_710
Comment 54•5 years ago
|
||
(In reply to Emerson Bernier from comment #53)
Users could still add more permissions to firefox flatpak locally (opt-in) however unlike snap flatpak doesn't have UX for adding more permissions easily so maybe this should be opt-out by default. In that case I would prefer single "--filesystem=host:ro" which will be easier to override that multiple of specific dirs.
I agree with this conclusion: One broad permission makes it more obvious what is happening and is easier to revoke.
There is also problem with /tmp - while opening files in external apps firefox download them to /tmp or TMPDIR at first. As /tmp from sandbox is inaccessible outside it, external app won't be able to find and open the file. This can be solved either by:
- "--filesystem=/tmp" - easiest but least secure solution
- Providing /app/bin/firefox[1] wrapper which sets TMPDIR to something readable. For example fedora sets[2] TMPDIR="$XDG_CACHE_HOME/tmp
[1] It can't be done with --env=TMPDIR= permissions because it won't work on systems where bubblewrap is installed as setuid binary, see https://github.com/flatpak/flatpak/issues/2641
[2] https://src.fedoraproject.org/rpms/firefox/blob/0adc4aecec45e195b0f6cbcdb1625839846482bc/f/firefox.spec#_710
Can't we just patch Firefox to first try using $XDG_RUNTIME_DIR
, before falling back to $TMPDIR
, $TEMP
and finally /tmp
? The runtime is guaranteed to be per-user and the path of that variable will be set to /run/user/$(id -u)/app/org.mozilla.Firefox
which is readable by any classic desktop app of the same user.
Comment 55•5 years ago
|
||
Create fp-gecko-1-gw-linux worker type
Comment 56•5 years ago
|
||
(In reply to ntninja from comment #54)
Can't we just patch Firefox to first try using
$XDG_RUNTIME_DIR
, before falling back to$TMPDIR
,$TEMP
and finally/tmp
? The runtime is guaranteed to be per-user and the path of that variable will be set to/run/user/$(id -u)/app/org.mozilla.Firefox
which is readable by any classic desktop app of the same user.
Yes, that's another solution. This one will have impact beyond flatpak though as unlike TMPDIR, XDG_RUNTIME_DIR is always set on modern systems so it will become de facto new firefox default.
Assignee | ||
Comment 57•5 years ago
|
||
Assignee | ||
Comment 58•4 years ago
|
||
Quick status update here - some of the things changed under the hood in the flatpak world that now enable the building of flatpaks without the need of --privileged
flag. That puts back on the radar the possiblity of using docker-worker (and eventually generic-worker once the TC migration will be completed, but that's not blocking fow now) to build these.
In order to avoid polluting/spamming this bug (with many CCs), I'm tracking the work to make the staging releases of Firefox build and ship the Flatpaks in bug 1591387.
Big shout out to Valentin David and Robert McQueen who have helped streamline this process in the past few weeks.
More to come soon, in the next days.
Assignee | ||
Updated•4 years ago
|
Assignee | ||
Comment 59•4 years ago
•
|
||
Status update:
- we were able to succesfully build flatpaks and publish them to the
beta
channel - we will continue to do so as part of the beta train of 75.0
- for any conversations regarding flatpak manifest improvement that were happening in bug 1591387, let's use bug 1621083 instead
In order to test the currently-shipped Firefox Flatpaks on the Flathub's beta channel:
- depending on the distribution, opening https://flathub.org/beta-repo/appstream/org.mozilla.firefox.flatpakref from browser should cause GNOME Software/KDE discover to ask if user wants to install it
- alternatively
flatpak install --user https://flathub.org/beta-repo/appstream/org.mozilla.firefox.flatpakref
We still have a couple of issues to solve, like l10n and better building configuration, before we're ready to publish to stable
channel (and be available via the Flathub UI).
Last but not least, there is a dedicated Release Automation: Flatpak
component now, within RelEng product, feel free to file bugs against it.
Assignee | ||
Updated•4 years ago
|
Assignee | ||
Comment 60•4 years ago
|
||
Status update:
- we've started publishing Flatpak to the
beta
channel as instructed in comment 59 - in the past few days, a couple of fixes landed in order to fix the
branch
we were using in building the flatpak, opening more ports and fixing l10n; latest Flathub's Flatpak (frombeta
channel) is up to date
Since this bug has a lot of folks CC'ed, I'm taking the opportunity to ask you all to give this a test should you like (instructions in comment above). Feel free to report any bugs.
Shout-out to @barthalion who's been super helpful in the last few weeks, providing lots of valuable help and insight.
Assignee | ||
Comment 61•4 years ago
|
||
Status update:
- courtesy to @alexl (Alexander Larsson) who explained l10n configuration in bug 1621074 and @barthalion (Bartłomiej Piotrowski) who implemented this we now have functional l10n in Flatpaks
The last flatpak we have on Flathub's beta channel should contain most of the blocking fixes that we had, so please give that a test whenever you have some time.
Assignee | ||
Updated•4 years ago
|
Assignee | ||
Comment 62•4 years ago
|
||
Status update: we've officially shipped Firefox 75.0 Flatpak into the Flathub store on the stable
channel - https://flathub.org/apps/details/org.mozilla.firefox
There are still some fixes/improvements to be addressed but those are tracked separately.
I think we can finally close this bug. If there are more concerns and other follow-ups, feel free to file a bug in this component.
Last but not least, I'd like to emphasize that this was a great team effort and coordination with the community folks for which I'm extremely thankful.
Assignee | ||
Comment 63•4 years ago
|
||
To sum-up, for posterity:
For now:
- we're shipping https://hg.mozilla.org/releases/mozilla-beta/ based Firefox Flatpaks to the Flathub
beta
channel three times a week, as part of our automation, the same way we do with Snaps - we're shipping https://hg.mozilla.org/releases/mozilla-release/ based Firefox Flatpaks to the Flathub
stable
channel with each release of Firefox, as part of our automation, the same way we do with Snaps
In the future, we might want to extend the support so that Firefox Nightly, Firefox Beta and even Devedition become standalone apps in the Flathub (stable) store, each with its own appid. But for now, we're going to let this bake for a while.
Comment 64•4 years ago
|
||
First of all, big thanks for your excellent work on bringing Firefox to flatpak with nice sandboxing and Portal support.
For whatever reason, the org.mozilla.firefox repository under Flathub github is missing and the publisher link on the Flathub page for Firefox (https://github.com/flathub/org.mozilla.firefox/graphs/contributors/) gives a 404 error. I think for such a prominent app it would be important to have it working. Also, it is not clear where people should submit bugs in the Flatpak release – flathub github or here in Bugzilla?
Everything else seem to be be working fine so far, but antialiasing for at least some fonts is broken on Debian 10.
Comment 65•4 years ago
|
||
The publisher page link issue is a bug in the flathub website frontend - Firefox is the first app that uses the direct publish system (previously Flathub has only added runtimes that way) so this needs an extra fix. There's a bug there for that issue: https://github.com/flathub/linux-store-frontend/issues/213
As for issues with the flatpak, please open them here and fix in the Firefox codebase - that's the point of publishing the bits from Mozilla directly!
Assignee | ||
Comment 66•4 years ago
|
||
(In reply to Pablo from comment #64)
For whatever reason, the org.mozilla.firefox repository under Flathub github is missing and the publisher link on the Flathub page for Firefox (https://github.com/flathub/org.mozilla.firefox/graphs/contributors/) gives a 404 error. I think for such a prominent app it would be important to have it working. Also, it is not clear where people should submit bugs in the Flatpak release – flathub github or here in Bugzilla?
Thanks a lot for raising these. Taking them one by one:
-
re: https://github.com/flathub/org.mozilla.firefox 404 - this is somewhat expected. Submitting flatpaks is done via PR under normal circumstances and then each commit there regenerates the CI on Fltahub side as docs depict it. We didn't go down that path but instead used the API to submit our flatpak directly from our automation. The account was created manually by Flathub admins. We're aware of that and we're hoping to fix it server-side.
-
re: publisher link - AFAIK, it's a fallout from previous point - it's currently tracked in bug 1627831 and hopefully will be fixed soon on server side.
-
re: filing bugs - that's a very good question. I'll raise that with Flathub folks to see how to best divide the jurisdiction. The Flathub folks have all the know-how to deal with the under-the-hood mechanics in building flatpaks, while we're the owners of the release pipeline. However, in the past weeks, lots of good issues and feedback came via #flatpaks (both from Mozilla's https://chat.mozilla.org/#/room/#flatpaks:mozilla.org and from IRC freenode) that ended up as bugs here. While we, as RelEng, don't necessarily have the bandwidth/expertise to fix all the filed bugs, the community has been really great and helped us a lot with this. So I'd encourage you to file it here.
Everything else seem to be be working fine so far, but antialiasing for at least some fonts is broken on Debian 10.
There's also a related bug 1621915 on funky fonts.
Comment hidden (obsolete) |
Comment 68•4 years ago
|
||
(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #66)
- re: filing bugs - that's a very good question. I'll raise that with Flathub folks to see how to best divide the jurisdiction. The Flathub folks have all the know-how to deal with the under-the-hood mechanics in building flatpaks, while we're the owners of the release pipeline. However, in the past weeks, lots of good issues and feedback came via #flatpaks (both from Mozilla's https://chat.mozilla.org/#/room/#flatpaks:mozilla.org and from IRC freenode) that ended up as bugs here. While we, as RelEng, don't necessarily have the bandwidth/expertise to fix all the filed bugs, the community has been really great and helped us a lot with this. So I'd encourage you to file it here.
Hi, there are bug [1] reported against Red Hat Firefox flatpak [2]. Main goal of the RH flatpak repo was to provide FF flatpak until it's available from Mozilla which is done now so we (RH) should continue to support Mozilla FF flatpak only.
I'll discuss that with RH folks how we can help here. Meanwhile Firefox flatpak linux integration bugs can be reported as Core/Widget:GTK where will be visible to me and can be fixed.
[1] https://github.com/xhorak/firefox-devedition-flatpak/issues
[2] https://firefox-flatpak.mojefedora.cz/
Comment 69•4 years ago
|
||
btw. Bug 1278719 also tracks some Firefox specific flatpak issues although they may be related to flatpacks from https://firefox-flatpak.mojefedora.cz/. Mihai, do you mind to use the Bug 1278719 to also track Mozilla flatpak issues which are on Firefox side or is there any other tracker?
Thanks.
Assignee | ||
Comment 70•4 years ago
•
|
||
(In reply to Martin Stránský [:stransky] from comment #68)
Hi, there are bug [1] reported against Red Hat Firefox flatpak [2]. Main goal of the RH flatpak repo was to provide FF flatpak until it's available from Mozilla which is done now so we (RH) should continue to support Mozilla FF flatpak only.
I'll discuss that with RH folks how we can help here. Meanwhile Firefox flatpak linux integration bugs can be reported as Core/Widget:GTK where will be visible to me and can be fixed.
Hi Martin, thanks for reaching out.
- I see a very good list of issues there in the Redhat's flatpak. I wonder how many of those are still ongoing in the official Flatpak. Should we run through those at some point to see if they are still ocurring in the existing flatpak and transfer them in bugzilla?
- I'm happy to use the
Core/Widget:GTK
component for stuff that's related to Linux integration. And maybe keep this component for issues related to release/shipping the flatpak? Since I'm actively triaging this component, I will try to exercise good judgment to triage correctly between them. Would that work for you? - with respect to https://firefox-flatpak.mojefedora.cz/ - I'm wondering whether there is a way to migrate those users to the official flatpak?
(In reply to Martin Stránský [:stransky] from comment #69)
btw. Bug 1278719 also tracks some Firefox specific flatpak issues although they may be related to flatpacks from https://firefox-flatpak.mojefedora.cz/. Mihai, do you mind to use the Bug 1278719 to also track Mozilla flatpak issues which are on Firefox side or is there any other tracker?
Absolutely. To be honest, I've been using this bug for a long time as a "meta" tracker one and had some related children bugs in which I worked. I'm happy to use that bug from now one since this is resolved anyway.
Assignee | ||
Comment 71•4 years ago
|
||
Moving bug 1621915, bug 1623591 and bug 1627831 as deps to bug 1278719 and we'll keep the latter as main meta/tracking bug for any further issues.
Updated•4 years ago
|
Updated•4 years ago
|
Assignee | ||
Comment 72•4 years ago
|
||
Comment on attachment 9002794 [details] [diff] [review]
[in-tree] Generate multilocale flatpaks
Found in triaging, cleaning up my queues of requests.
This landed a good while back, nothing to do further more.
Description
•