Closed Bug 1441922 Opened 7 years ago Closed 5 years ago

Generate multilocale Firefox flatpak

Categories

(Release Engineering :: Release Automation, enhancement)

Product:
Release Engineering
Component:
Release Automation
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: Sylvestre, Assigned: mtabara)

References

(Blocks 1 open bug)

Details

Attachments

(2 files, 2 obsolete files)

[in-tree] Generate multilocale flatpaks
17.49 KB, patch
Details | Diff | Splinter Review
Bug 1441922 - create fp-gecko-1-gw-linux worker type
47 bytes, text/x-phabricator-request
Details | Review
Flatpak, like Snap, should include all shipped locales. See https://bugzilla.mozilla.org/show_bug.cgi?id=1297519 for the original implementation
Blocks: flatpak
No longer blocks: snappy
Started working on this, with ETA to get the packaging done before the All-hands.
Assignee: nobody → mtabara
(In reply to Mike Kaply [:mkaply] from comment #2) > See: > > https://bugzilla.mozilla.org/show_bug.cgi?id=1278719#c31 Thanks for that heads-up. I started working on this but didn't manage to finish in time before the All-hands. It's on my list to complete before the end of Q2. I'll keep you updated.
Some other things took priority at the end of Q2, I'm sorry I have to push this by another week. ETA is now 10th of July.
Cleared up my queue in the meantime, I'm actively working on this bug effective yesterday. My testing will be on https://hg.mozilla.org/projects/maple. Some staging releases will come up there soon.
Status update: trying to make things work on my local machine (macbook) with Docker. I'm using stuff from https://bugzilla.mozilla.org/show_bug.cgi?id=1278719#c31. Once I get that working, I'll be adding tasks similarly to snaps, and have them be created within staging releases on maple.
Status update: * I've got Docker running the flatpacks from here[1] on my local machine * I've extracted some of the installation deps within a local Dockerfile which is later going to be used in-tree * am still hitting some issues when trying to run the flatpak-builder as I don't have the right SDK. Installing it chases the rabbit hole in various GPG, repo, and other related issues. Trying to debug all to figure things out. Once I get it working on my local machine, I'll try to break it down into pieces to add it in-tree and have a staging release re-do the steps. [1]: https://bugzilla.mozilla.org/show_bug.cgi?id=1278719#c31
Status update: after a bunch of hiccups with installation deps, docker privileges, fuse device not found, build & installation issues, I finally managed to reproduce the behavior from here[1] within a Docker container on my local macbook machine. One small step for me, giant leap for flatpaks :) I kept notes of the steps I've done so far and I've committed my containers so I can debug more if I need to. My next short-goal-small-victory is to obtain the Dockerfile in which we need to run the main script to bundle flatpaks. I'll reuse the same logic and approach from snaps. [1]: https://github.com/xhorak/firefox-devedition-flatpak/tree/master/org.mozilla.FirefoxUpstreamBinary
Attached patch Dockerfile (obsolete) — — Splinter Review
Okay, I managed to get a first version of Dockerfile which sets the groundwork and environment for `flatpak` and `flatpak-builder` commands and solves all the issues I've had earlier with installing the pre-req. Note to self - two quick issues: * the image is huge, 2.7Gb vs ~225 Mb for snaps counterpart. I need to find a way to install the bare minimum to make this work * make sure the flatpak-builder doesn't throw the sdk/platform * still need to find a way to tell Docker via Taskcluster somehow, to run the commands with `docker run --privileged`. This is going to be tricky but I'll poke #taskcluster for some help if I don't find a smart way.
Status update: I've managed to recreate via Dockerfile the environment I used the other day to obtain the first flatpaks and ... I get a nasty segmentation fault. No stacktrace, very difficult to trace. I'm out of ideas for now, I've tried various different ways to get a glimpse of the error, unsuccessfully. My last guess is it could be related to the SDK, potentially. .... Downloading sources Initializing build dir FB: Running 'flatpak build-init --arch=x86_64 /root/app org.mozilla.FirefoxUpstreamBinary org.gnome.Sdk org.gnome.Platform 3.26' Committing stage init to cache Segmentation fault ...
(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #9) > Created attachment 8993187 [details] [diff] [review] > Dockerfile > > Okay, I managed to get a first version of Dockerfile which sets the > groundwork and environment for `flatpak` and `flatpak-builder` commands and > solves all the issues I've had earlier with installing the pre-req. > > Note to self - two quick issues: > * the image is huge, 2.7Gb vs ~225 Mb for snaps counterpart. I need to find > a way to install the bare minimum to make this work This is likely due to the full GNOME SDK and Platform which are included. The base system for simply running flatpak-builder is tiny. Normally when building a manifest, flatpak-builder pulls all of the required SDK/platform/extension build dependencies. Is it necessary that the SDK is pulled during the Dockerfile build for some reason? If you want to cache the SDK inside your Docker image, you can make this trade, but it has a space cost as you can see. Relatedly: depending how much GNOME stuff you need, you might reduce the build size (and runtime dependency size) by using the freedesktop.org platform, rather than the GNOME platform. The gnome.org platform really does contain every last GNOME library you can imagine - freedesktop.org has the basics like Gtk, GIO, GSettings, Gvfs, GStreamer/ffmpeg, etc. For instance, LibreOffice is able to build/run OK against the freedesktop SDK. Finally, the current correct remote for building/publishing the main (Freedesktop, GNOME and KDE) runtimes is Flathub. You can replace the https://sdk.gnome.org/gnome.flatpakrepo with https://dl.flathub.org/repo/flathub.flatpakrepo (and conventionally this remote is named as flathub). > * make sure the flatpak-builder doesn't throw the sdk/platform What do you mean by this? > * still need to find a way to tell Docker via Taskcluster somehow, to run > the commands with `docker run --privileged`. This is going to be tricky but > I'll poke #taskcluster for some help if I don't find a smart way.
(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #10) > Status update: I've managed to recreate via Dockerfile the environment I > used the other day to obtain the first flatpaks and ... I get a nasty > segmentation fault. No stacktrace, very difficult to trace. I'm out of ideas > for now, I've tried various different ways to get a glimpse of the error, > unsuccessfully. My last guess is it could be related to the SDK, potentially. > > .... > Downloading sources > Initializing build dir > FB: Running 'flatpak build-init --arch=x86_64 /root/app > org.mozilla.FirefoxUpstreamBinary org.gnome.Sdk org.gnome.Platform 3.26' > Committing stage init to cache > Segmentation fault > ... Nasty! I've not seen this before. At this stage the SDK might not even have done anything - to me this feels more like a problem with the platform/tools than what is being built - like ostree could be doing some FUSE stuff here. Does this problem reproduce on a "normal" Linux box Docker? I've not used the Mac OS X Docker stuff. If you add "main/dbgsym" to the apt sources.list line for the PPA, you should be able to find the -dbgsym packages for ostree and flatpak, which might help getting a backtrace from gdb. (https://wiki.ubuntu.com/DebuggingProgramCrash#Installing_dbgsym_packages_from_a_PPA) The other thing that might help with some "contextual clues" is G_MESSAGES_DEBUG="all" which should give any debug output from ostree, flatpak, etc. Otherwise my normal debug ideas here would be `strace -f -e file` the whole thing to see roughly where it might be failing.
(In reply to Sylvestre Ledru [:sylvestre] from comment #0) > Flatpak, like Snap, should include all shipped locales. > See https://bugzilla.mozilla.org/show_bug.cgi?id=1297519 for the original > implementation For completeness, it's relatively trivial to do the right thing for splitting up Locales in Flatpaks. The way that Locales would be split out in a Flatpak is to relocate everything into /app/share/runtime/locale/<base_lang_code> folders which was language dependent (where the base lang code is en, pt, etc without any _variants or @dialects) and then symlinking back from wherever Firefox expected it into those paths. You can then turn the .Locale extension on in the Flatpak manifest (separate-locales: true) and it will create an extension which will be subsetted at install time to those languages in use by the user/system. A reasonable example of this is in LibreOffice, where two different trees of language-dependent stuff are moved into subdirectories of the /app/share/runtime/locale folder and symlinked back. On typical 1-2 language systems 100s of MB of download/install space was saved. https://cgit.freedesktop.org/libreoffice/core/tree/solenv/bin/assemble-flatpak.sh#n46 As you can see - one set of locales (C/en) should be kept built-in because the entire Locale extension (ie /app/share/runtime/locale folder and all children) is optional, and the only other caveat is that Firefox must be OK with dangling symlinks where locale stuff is normally found. LibreOffice needed a small patch to skip a dangling symlink in the locale data, for this situation.
@robert: thanks a lot for all your replies, this is very helpful! I'll give it a try using your suggestions. I was traveling on Thursday/Friday, hence the delays in answering, mea culpa.
(In reply to robert from comment #11) > (In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #9) > > Created attachment 8993187 [details] [diff] [review] > > Dockerfile > > > > Okay, I managed to get a first version of Dockerfile which sets the > > groundwork and environment for `flatpak` and `flatpak-builder` commands and > > solves all the issues I've had earlier with installing the pre-req. > > > > Note to self - two quick issues: > > * the image is huge, 2.7Gb vs ~225 Mb for snaps counterpart. I need to find > > a way to install the bare minimum to make this work > > This is likely due to the full GNOME SDK and Platform which are included. > The base system for simply running flatpak-builder is tiny. Normally when > building a manifest, flatpak-builder pulls all of the required > SDK/platform/extension build dependencies. Is it necessary that the SDK is > pulled during the Dockerfile build for some reason? If you want to cache the > SDK inside your Docker image, you can make this trade, but it has a space > cost as you can see. > > Relatedly: depending how much GNOME stuff you need, you might reduce the > build size (and runtime dependency size) by using the freedesktop.org > platform, rather than the GNOME platform. The gnome.org platform really does > contain every last GNOME library you can imagine - freedesktop.org has the > basics like Gtk, GIO, GSettings, Gvfs, GStreamer/ffmpeg, etc. For instance, > LibreOffice is able to build/run OK against the freedesktop SDK. > > Finally, the current correct remote for building/publishing the main > (Freedesktop, GNOME and KDE) runtimes is Flathub. You can replace the > https://sdk.gnome.org/gnome.flatpakrepo with > https://dl.flathub.org/repo/flathub.flatpakrepo (and conventionally this > remote is named as flathub). This worked like a charm, thanks a lot for suggesting this! I wasn't sure how to tell flatpak-builder to do that but sounds like https://github.com/flatpak/flatpak/issues/1445#issuecomment-373347884 was what I was missing. I tweaked the template manifest to point to the freedesktop sdk/platform installations in conjungtion to telling flatpak-builder to use `--install-deps-from=flathub`, where flathub is defined in Dockerfile via `flatpak remote-add --from flathub https://dl.flathub.org/repo/flathub.flatpakrepo` flatpak-builder --install-deps-from=flathub --assumeyes --disable-rofiles-fuse --disable-cache --force-clean --repo=repo app org.mozilla.FirefoxUpstreamBinary.json Image contains the bare minimum at is down to 474 Mb! Platform/SDK installation now happen at runtime which is exactly what I wanted. This is no longer an issue, moving on to the next one. Still end up with segmentation fault so I'll proceed with your suggestions in improving the stacktrack debugging. > > * still need to find a way to tell Docker via Taskcluster somehow, to run > > the commands with `docker run --privileged`. This is going to be tricky but > > I'll poke #taskcluster for some help if I don't find a smart way. Right now, I start the docker containers with `--privileged` which spares me of some nasty fuse errors. With Taskcluster, I won't be able to run that given the mechanics under the hood so I'll likely end up with those fuse errors. But so far I have the segmentation fault issue to deal with first.
Note to self: I could try using a different version too. Maybe there's something wrong with the existing installer builders.
Could be something wrong with my Macbook-based Docker containers. Johan helped me test this on his native Linux box and sounds like he's getting the following error: Building module firefox in /work/firefox-devedition-flatpak/org.mozilla.FirefoxUpstreamBinary/.flatpak-builder/build/firefox-1 ======================================================================== tar (child): bzip2: Cannot exec: No such file or directory tar (child): Error is not recoverable: exiting now tar: Child returned status 2 tar: Error is not recoverable: exiting now Error: module firefox: Child process exited with code 2 + flatpak build-bundle repo org.mozilla.FirefoxUpstreamBinary.flatpak org.mozilla.FirefoxUpstreamBinary error: 'repo' is not a valid repository * which still fails but moves past the seg fault I was getting. * he's also using Docker version 18.05.0-ce, build f150324782 while I have Docker version 18.03.1-ce, build 9ee9f40 so there could be something related to that as well, not sure. * he ran all without the privileged flag which is great news! Next steps: * move stuff in-tree and rely on docker-workers for remaining of the testing * iterate there to make these operational I'll be back with updates, work-in-progress.
Status update: * I managed to get them working on my local environments on Macbook too. * I've added tasks to get them in-tree and I'm currently iterating on maple within staging releases * Docker-image-generation task is green[1], creates the correct image here[2] * flatpak bundle-generation task is here[3]. Looks promising so far but hasn't run as we have some unrelated issues in the staging environment Task should go green and generate a single-locale flatpak. I'm working on adding support for multi-locale now. [1]: https://tools.taskcluster.net/groups/RYL82EquQDyilRvUdALHSg/tasks/KaTBqnEtTrqVf5LbNpATVg/details [2]: https://queue.taskcluster.net/v1/task/KaTBqnEtTrqVf5LbNpATVg/runs/0/artifacts/public/image.tar.zst [3]: https://tools.taskcluster.net/groups/Svy8QGRdSLigc1ON0kyI7w/tasks/DpbURMhdSiGg8cBcIy7Lwg/details
(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #18) > Status update: > * I managed to get them working on my local environments on Macbook too. > * I've added tasks to get them in-tree and I'm currently iterating on maple > within staging releases > * Docker-image-generation task is green[1], creates the correct image here[2] > * flatpak bundle-generation task is here[3]. Looks promising so far but > hasn't run as we have some unrelated issues in the staging environment > > Task should go green and generate a single-locale flatpak. I'm working on > adding support for multi-locale now. This is great news! I have some feedback for the Flatpak build process - I found the .json and stuff in https://hg.mozilla.org/projects/maple/file/tip/taskcluster/docker/firefox-flatpak but I am unfamiliar with Mozilla contribution processes. Is there a pull request or something that I can leave review comments against? Or should I just list out lines/comments here? One thing which we will need to solve in order to publish things normally, especially if we split the different locales out using a .Locale extension, is that a Flatpak bundle can only contain one ref - but a properly split-out Locale extension would add a 2nd ref. Typically to do normal/idiomatic Flatpak installs and upgrades, Flatpak needs to have a ostree repo set up as a remote to pull from - downloading and installing a bundle is kind of a "one off" installation, but to get notified of updates in your app center the newer commits and other refs need to be published in a repo. Does Mozilla plan to also publish a Flatpak repo, or would we set up a process for exporting the bundles from Mozilla into Flathub so that we would publish/mirror the latest through a regular repo? In Flathub we actually don't use bundles - the implementation is kind of lousy (they take loads of RAM during generation) - so we just tar up the whole result of the flatpak-builder repo dir and move that from the build worker to the repo master, and import things to the main repo that way. (I've also joined irc.mozilla.org, nick ramcq, if there is a good channel for us to chat about Flatpak?)
(In reply to Robert McQueen from comment #19) > (In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #18) > > Status update: > > * I managed to get them working on my local environments on Macbook too. > > * I've added tasks to get them in-tree and I'm currently iterating on maple > > within staging releases > > * Docker-image-generation task is green[1], creates the correct image here[2] > > * flatpak bundle-generation task is here[3]. Looks promising so far but > > hasn't run as we have some unrelated issues in the staging environment > > > > Task should go green and generate a single-locale flatpak. I'm working on > > adding support for multi-locale now. > > This is great news! Thanks for quick turnaround. Let me reply inline. 1. Maple is a project branch used by RelEng for testing so I'm merely iterating in staging conditions how this would look lile. 2. What I have so far is https://hg.mozilla.org/projects/maple/rev/0cf2dd5ccd6f but the patch is far away from being production-ready. 3. My immediate goal is to get some multi-locale flatpaks generated in-tree via staging releases and then dogfood the app in iterations to fix all the glitches. Right now there's still plenty of hardcodes and stuff, I agree. I'll try to address them one by one. I've got more stuff locally and a list of fixes/hardcodes I need to address and the list is growing :) 4. Thanks for your help and suggestions with respect to localization and flatpak publishing. > I have some feedback for the Flatpak build process - I found the .json and > stuff in > https://hg.mozilla.org/projects/maple/file/tip/taskcluster/docker/firefox- > flatpak but I am unfamiliar with Mozilla contribution processes. Is there a > pull request or something that I can leave review comments against? Or > should I just list out lines/comments here? If patches are in-tree, using this bug and sending patches here are good ways to start I suppose. https://developer.mozilla.org/en-US/docs/Mozilla/Developer_guide/Introduction contains more verbose information about this. > One thing which we will need to solve in order to publish things normally, > especially if we split the different locales out using a .Locale extension, > is that a Flatpak bundle can only contain one ref - but a properly split-out > Locale extension would add a 2nd ref. > > Typically to do normal/idiomatic Flatpak installs and upgrades, Flatpak > needs to have a ostree repo set up as a remote to pull from - downloading > and installing a bundle is kind of a "one off" installation, but to get > notified of updates in your app center the newer commits and other refs need > to be published in a repo. > > Does Mozilla plan to also publish a Flatpak repo, or would we set up a > process for exporting the bundles from Mozilla into Flathub so that we would > publish/mirror the latest through a regular repo? In Flathub we actually > don't use bundles - the implementation is kind of lousy (they take loads of > RAM during generation) - so we just tar up the whole result of the > flatpak-builder repo dir and move that from the build worker to the repo > master, and import things to the main repo that way. However, for more detailed questions regarding the process of publishing this, I'm not sure I'm the best person to ask. As a first step, my thoughts were to simply publish this within our S3 buckets, or something. Publishing it properly is a separate job as far as I can tell(not sure if that involves signing as well), most likely outside of this bug scope. Let me redirect that question to :Sylvestre, he might have more insight. (he's currently PTO until the 12th of August) > (I've also joined irc.mozilla.org, nick ramcq, if there is a good channel > for us to chat about Flatpak?) I'm mtabara on IRC and can be found in various channels. The one I'm using for debugging flatpaks is #tcmigration.
(In reply to Robert McQueen from comment #19) > (I've also joined irc.mozilla.org, nick ramcq, if there is a good channel > for us to chat about Flatpak?) We opened #flatpaks if anyone wants to follow along/join in.
Status update: * I'm still WIP to add langpacks to the single-locale flatpak and land that on maple to iterate * more conversations on flatpaks now happen in #flatpaks on IRC * The flatpak generation looks very very promising in the staging release but it's currently failing for some priviledge issues within Docker. Task is here[1]. The rest of things worked smoothly so if I manage to unblock that[2] we will he having single-locale flatpaks for now. I'll continue to polish patches and get them to a more review-able state. Stay tuned for more. [1]: https://tools.taskcluster.net/groups/XAgbo8fTRnKfWDyKUA72rQ/tasks/JuBubPToRhCp9YLqFmn5_w/details [2]: Note: I'm getting `bwrap permissions errors`. On local machine I managed to overcome that by running docker run in privideleged mode but that's very dangerous as it exposes all devices of host. Most likely impossible to reproduce on docker-worker machines for security reasons. Am working with Taskcluster team to unblock on that and find a workaround.
I spoke with some TC folks and dropped a message to their mailing list to ask for some help. Apparently, it's not trivial to solve this (playing with the container privileges in Taskcluster). We have a backup solution if worse comes to worst (to have dedicated workers purely for this that can have dedicated rights) but I'd like to clear other better solutions (if any) before we default to that. In the meantime, I'm trying to get the langpacks bundled within the en-US to reach the milestone I had promised initially. Will pause and re-evaluate timeline after that.
So, inside flatpak-builder / flatpak, the privileged operations for setting up a container are handled by bubblewrap which is setuid on most systems. The main reason for this is that unpriveleged users are not allowed to open new namespaces on most distro kernels. However, this is a kernel setting, and if you switch to a recent enough bubblewrap, you might not need it to be priveleged - the last 0.3.0 release of bubblewrap (https://github.com/projectatomic/bubblewrap/releases/tag/v0.3.0) allows the namespaces and filesystems to be set up without any priveleged operations. So, if the seccomp filters can be relaxed somewhat in the Docker configuration, and the machines permit user namespaces to be created, you might be OK!
ramcq | is there a "recipe" on how to run bubblewrap inside of docker, based off | https://github.com/projectatomic/bubblewrap/releases/tag/v0.3.0 ? ramcq | Mozilla are looking at integrating flatpak builds into their docker-based task runner infra ramcq | would be good if I could say, sysctl yada yada, echo foo >Dockerfile ramcq | https://bugzilla.mozilla.org/show_bug.cgi?id=1441922#c22 walters | ramcq: bwrap's CI does it, see: https://github.com/projectatomic/bubblewrap/blob/master/.papr.yml#L22 walters | then follow it into https://github.com/projectatomic/bubblewrap/blob/master/ci/papr.sh walters | currently it passes `--privileged` but that is probably a bug; it's sufficient to do `--security-opt seccomp=unconfined`
Status update: * I have single-locale bundled with langpacks ready working on my local Docker * still having trouble reproducing that in the Taskcluster Docker containers for permissions errors. I moved a bit forward with some help from TC team but am still few hops from victory Once I manage to get unblocked there and have a successful run, I'll hand this over to testing and pause for further discussions with respect to: * publishing * reviewing * optimizing
Status update: * just to clarify - things are working properly on my Docker environment - multi-locale flatpaks (en-US bundled with langpacks) ready for testing. * I'm still having difficulties translating that into TC. I've tried different provisioners/workerTypes and various other solutions. Am in touch with them to find an alternative way, most likely running them in a dedicated workerType As soon as I'm unblocked on TC side, jobs should just be green on maple, code is already landed.
Green task in Taskcluster! Task is here[1], flatpak bundle is here[2]. More artifacts here[3] (signing manifest, checksums and others). The bundle is bundled with en-US + langpacks. Can be installed with `flatpak install target.flatpak`. I'll pause here for a bit to test this for now and suggest future areas of improvement. Leftover from my side: * get this polished in-tree and potentially landed if we get green go * still figure it out how can I bake the version/build number for the firefox I'm building. Currently I don't think I am doing that * chain this task to the generate-checksums-beetmover one to make sure it runs after the SHA512 sums is done. [1]: https://tools.taskcluster.net/groups/Id3xHl2KTbGkxMqblk8ZNg/tasks/GhbsyyHqS1OaCEXuOyhlmA/details [2]: https://queue.taskcluster.net/v1/task/GhbsyyHqS1OaCEXuOyhlmA/runs/0/artifacts/public/build/target.flatpak [3]: https://tools.taskcluster.net/groups/Id3xHl2KTbGkxMqblk8ZNg/tasks/GhbsyyHqS1OaCEXuOyhlmA/runs/0/artifacts
:robert@mcqueen.me.uk Out of curiosity, would you know if/how to bake the version/build number in flatpaks? Right now I don't think that's baked anywhere in the configs. The en-US tarball that we use and download is correctly pulled from S3 (via env var BUILD_NUMBER and VERSION in https://tools.taskcluster.net/groups/Id3xHl2KTbGkxMqblk8ZNg/tasks/GhbsyyHqS1OaCEXuOyhlmA/details) but I'm wondering if there's any other place we need to tweak that. Or does that suffice?
Flags: needinfo?(robert)
(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #29) > :robert@mcqueen.me.uk > > Out of curiosity, would you know if/how to bake the version/build number in > flatpaks? Right now I don't think that's baked anywhere in the configs. > > The en-US tarball that we use and download is correctly pulled from S3 (via > env var BUILD_NUMBER and VERSION in > https://tools.taskcluster.net/groups/Id3xHl2KTbGkxMqblk8ZNg/tasks/ > GhbsyyHqS1OaCEXuOyhlmA/details) but I'm wondering if there's any other place > we need to tweak that. Or does that suffice? If you add <releases> with a <release> entry, the version number and sometimes the release notes will be shown in GNOME Software, Flathub etc. https://www.freedesktop.org/software/appstream/docs/chap-Metadata.html#tag-releases
Flags: needinfo?(robert)
Attachment #8993187 - Attachment is obsolete: true
Comment on attachment 9002794 [details] [diff] [review] [in-tree] Generate multilocale flatpaks As we discussed, adding you as a reviewer here. I guess the interesting bits that really matter are the: * taskcluster/docker/firefox-flatpak/Dockerfile - where the initial docker image is being created with all the neccesary bits for building flatpaks * taskcluster/docker/firefox-flatpak/Makefile to customize and massage the Firefox installation to fit some of the preferences and customationzations needed * taskcluster/docker/firefox-flatpak/default-preferences.js * taskcluster/docker/firefox-flatpak/distribution.ini * taskcluster/docker/firefox-flatpak/flatpak.json.in - manifest for flatpak * taskcluster/docker/firefox-flatpak/runme.sh - where all the magic happens Let me know if I can help with more information. Thank you again for this!
Attachment #9002794 - Flags: review?(robert)
Comment on attachment 9002794 [details] [diff] [review] [in-tree] Generate multilocale flatpaks Review of attachment 9002794 [details] [diff] [review]: ----------------------------------------------------------------- I've given a quick review to the 'flatpak' and desktop integration parts. Rob is still best for the build parts. I'll try and build it myself and dig a bit deeper into the d-bus sandboxing and see if we can improve the confinement there. ::: taskcluster/docker/firefox-flatpak/Makefile @@ +6,5 @@ > + mkdir -p $(prefix)/share/appdata > + cp -p org.mozilla.Firefox.appdata.xml $(prefix)/share/appdata > + desktop-file-install --dir $(prefix)/share/applications org.mozilla.Firefox.desktop > + mkdir -p $(prefix)/share/icons/hicolor/128x128/apps/ $(prefix)/share/appdata > + cp -p firefox/browser/chrome/icons/default/default128.png $(prefix)/share/icons/hicolor/128x128/apps/org.mozilla.Firefox.png Export at least the 64px version as well, that's used by some software stores and otherwise you'll get an automatically resized version which I'm sure the designers wouldn't like. Generally I'd recommend exporting all the icons you have (16, 32, 48, 64, 128). ::: taskcluster/docker/firefox-flatpak/flatpak.json.in @@ +1,4 @@ > +{ > + "app-id": "org.mozilla.Firefox", > + "runtime": "org.freedesktop.Platform", > + "runtime-version": "1.6", There's a new FDo release - 18.08 which should be used now, the older version is out of security support. 18.08 will have two years of security support and be stable within that period (see https://gitlab.com/freedesktop-sdk/freedesktop-sdk/wikis/release for details) @@ +3,5 @@ > + "runtime": "org.freedesktop.Platform", > + "runtime-version": "1.6", > + "sdk": "org.freedesktop.Sdk", > + "command": "firefox", > + "rename-appdata-file" : "org.mozilla.Firefox.appdata.xml", Why do you need to rename something that already has the correct naming? @@ +4,5 @@ > + "runtime-version": "1.6", > + "sdk": "org.freedesktop.Sdk", > + "command": "firefox", > + "rename-appdata-file" : "org.mozilla.Firefox.appdata.xml", > + "rename-icon" : "org.mozilla.Firefox", Same as above, not sure why you need to rename something that already has the correct naming. @@ +9,5 @@ > + "finish-args": [ > + /* X11 + XShm access */ > + "--share=ipc", "--socket=x11", > + /* Wayland access */ > + "--socket=wayland", Do you want Firefox to be wayland native (if so, awesome!)? If this is not a supported configuration then you can just remove this and it'll use XWayland by default. @@ +19,5 @@ > + "--socket=pulseaudio", > + /* Enable access to dri */ > + "--device=dri", > + /* Enable themes support */ > + "--filesystem=~/.config/dconf:ro", This hardcoding can be wrong on some systems, you need xdg-run/dconf as noted below. @@ +20,5 @@ > + /* Enable access to dri */ > + "--device=dri", > + /* Enable themes support */ > + "--filesystem=~/.config/dconf:ro", > + "--env=DCONF_USER_CONFIG_DIR=.config/dconf", For dconf support you need all of: "--filesystem=xdg-run/dconf", "--filesystem=xdg-config/dconf:ro", "--talk-name=ca.desrt.dconf", "--env=DCONF_USER_CONFIG_DIR=.config/dconf" @@ +22,5 @@ > + /* Enable themes support */ > + "--filesystem=~/.config/dconf:ro", > + "--env=DCONF_USER_CONFIG_DIR=.config/dconf", > + /* Required for file reveal in file manager */ > + "--socket=session-bus" These permissions are super broad. You should run the flatpak under --log-session-bus and whitelist the specific D-Bus endpoints. (I will try and build this locally myself and do exactly that but didn't want to hold up the review. @@ +28,5 @@ > + "modules": [ > + { > + "name": "firefox", > + "no-autogen": true, > + "make-install-args": ["prefix=/app"], given the way you use a Makefile here it'd be more idiomatic flatpak to use "buildsystem": "simple" and putting each line inside a "build-commands" block. (see http://docs.flatpak.org/en/latest/flatpak-builder-command-reference.html for more on that). It's just a recommendation though, if your team is more comfortable with Make, then carry on. ::: taskcluster/docker/firefox-flatpak/org.mozilla.Firefox.appdata.xml @@ +1,1 @@ > +<?xml version="1.0" encoding="UTF-8"?> High quality appdata files require screenshots, release tags and OARS https://hughsie.github.io/oars/ @@ +1,3 @@ > +<?xml version="1.0" encoding="UTF-8"?> > +<application> > + <id type="desktop">org.mozilla.Firefox.desktop</id> you shouldn't need the .desktop suffix anymore. Just org.mozilla.Firefox is better. @@ +9,5 @@ > + <p> > + Firefox is a Web browser that is smaller, faster, and in some ways more secure than other competitors. Firefox gives users a cleaner interface and faster download speeds. > + </p> > + <p> > + Firefox is repack of upstream binaries to the flatpak. This text is a bit ungainly! @@ +17,5 @@ > + <keyword>mozilla</keyword> > + <keyword>internet</keyword> > + <keyword>web</keyword> > + </keywords> > + <url type="homepage">https://www.mozilla.org/en-US/firefox/new/</url> You should also include url type=donation (likely pointing to https://donate.mozilla.org/ ) and url type=bugtracker pointing to the obvious locations
This comment is coming at you from Flatpak-ed firefox! Exciting. The bundle linked above doesn't actually work out of the box, which is a bit of a shame: > libdbus-glib-1.so.2: cannot open shared object file: No such file or directory dbug-glib isn't part of the base Freedesktop Platform. There are two possible ways to fix this. You can either do what LibreOffice does and build your own copy of dbus-glib and bundle it into your flatpak (in which case, copy and paste the code here into the manifest: https://github.com/flathub/org.libreoffice.LibreOffice/blob/master/org.libreoffice.LibreOffice.json#L46 ) Or you could switch the manifest to use the GNOME Platform instead, which is a superset of the Freedesktop one and does have that installed. I tested both the stable 3.28 release and the master release (that will be 3.30) and they both work fine. 3.28 uses Freedesktop 1.6 as it's base, the master release has switched over the the new 18.08 base. If you want to test that yourself then you need to run: > flatpak run --runtime=org.gnome.Platform//3.28 org.mozilla.Firefox I also tested the open folder implementation using the following command: > flatpak run --runtime=org.gnome.Platform//3.28 --log-session-bus --unshare=session-bus org.mozilla.Firefox You should absolutely remove the --share=session-bus permission and replace it with: "--talk-name=org.freedesktop.FileManager1" "--talk-name=org.gtk.vfs.*" "--talk-name=org.freedesktop.Notifications" in the finish-args. I'll keep testing and see if there's anything else that needs to talk dbus to the system, if you know of it then it's really easy to hunt down with the logger, although it's somewhat chatty on the command line.
In order to take advantage of Flatpak's sandboxing, Firefox will need to use the `xdg-desktop-portal` D-Bus API for file selectors. This will have the added advantage of making Firefox use native file selectors in KDE environments. I'm actually interested in implementing this functionality, but I'm new to Firefox contributions. Should I open another issue for that particular piece?
(In reply to Nathan Moos from comment #35) > I'm actually interested in implementing this functionality Great, thanks! > Should I open another issue for that particular piece? Yes, bugs are cheap.
General Flatpak question. What's the easiest way to know we are in a Flatpak? Snap sets an environment variable: https://searchfox.org/mozilla-central/source/browser/components/shell/nsGNOMEShellService.cpp#90 Is there something similar for Flatpak?
There's a /.flatpak-info file in the root of every sandbox. This contains a bunch of information about the flatpak environment you're in. (see more here: https://blogs.gnome.org/mclasen/2018/07/02/flatpak-in-detail-part-3/) There's also a /app/.updated file that tells you that an update is ready to be installed. It'd be wonderful if Firefox could hook its 'restart to update' stuff into that. (more on using that here https://blogs.gnome.org/mclasen/2018/11/26/an-update-on-flatpak-updates/ )

Bug 1411579 already has code for detecting a Flatpak sandbox environment, so you should probably share that function. It is based on the way GTK itself detects such an environment: https://github.com/GNOME/gtk/blob/e0ce028c88858b96aeda9e41734a39a3a04f705d/gtk/gtkprivate.c#L272

Note, if this turns out to be easier to run in generic-worker with multiuser engine (separate OS users per task) rather than docker-worker, the work to add multiuser engine to linux is in bug 1499054. That bug isn't a huge amount of work, but note we may have some competing priorities (notably bug 1499055). A priority call should probably be made about whether we should tackle that now.

We would also need to set up a linux worker with generic-worker, which for this we could do manually in EC2, and take a snapshot, as we currently don't have automation around building EC2 AMIs for generic-worker linux workers.

(In reply to Pete Moore [:pmoore][:pete] from comment #40)

Note, if this turns out to be easier to run in generic-worker with multiuser engine (separate OS users per task) rather than docker-worker, the work to add multiuser engine to linux is in bug 1499054. That bug isn't a huge amount of work, but note we may have some competing priorities (notably bug 1499055). A priority call should probably be made about whether we should tackle that now.

Based on my understanding of the situation, if this isn't a huge amount of work, I think it's worth the detour. Essentially, the flatpak-builder workload brings its own containerization and it's the nesting of these within docker which causes the problem. Because the actual build workload is in any case sandboxed by flatpak, this trouble is avoidable.

We would also need to set up a linux worker with generic-worker, which for this we could do manually in EC2, and take a snapshot, as we currently don't have automation around building EC2 AMIs for generic-worker linux workers.

Is it possible to arrange that these images can have certain dependencies installed (ie flatpak and flatpak-builder) or is something else (a buildroot?) required to bring additional binaries/dependencies into the multiuser engine to run the actual build jobs? As well as the competing containerisation, in this workload having our own Docker image is also allowing us to install our needed dependencies into the sandbox.

Status update on this: workflow is ready on our testing trees but some of the underlying tooling prevents us to ship to production for security reasons. We're working on this with RelOps and Taskcluster teams to find a solution to sort this out in early Q3.

(In reply to Robert McQueen from comment #41)

(In reply to Pete Moore [:pmoore][:pete] from comment #40)

Note, if this turns out to be easier to run in generic-worker with multiuser engine (separate OS users per task) rather than docker-worker, the work to add multiuser engine to linux is in bug 1499054. That bug isn't a huge amount of work, but note we may have some competing priorities (notably bug 1499055). A priority call should probably be made about whether we should tackle that now.

Based on my understanding of the situation, if this isn't a huge amount of work, I think it's worth the detour. Essentially, the flatpak-builder workload brings its own containerization and it's the nesting of these within docker which causes the problem. Because the actual build workload is in any case sandboxed by flatpak, this trouble is avoidable.

We would also need to set up a linux worker with generic-worker, which for this we could do manually in EC2, and take a snapshot, as we currently don't have automation around building EC2 AMIs for generic-worker linux workers.

Is it possible to arrange that these images can have certain dependencies installed (ie flatpak and flatpak-builder) or is something else (a buildroot?) required to bring additional binaries/dependencies into the multiuser engine to run the actual build jobs? As well as the competing containerisation, in this workload having our own Docker image is also allowing us to install our needed dependencies into the sandbox.

I think for initial rollout, we can prepare an image with everything required, and we can make the installation steps transparent by linking to e.g. a puppet config or setup script from the task log header. Long term, if the system dependencies change regularly, we may want to look into a simpler way to apply changes, but for the meantime, I suspect adding system dependencies that require root to install will need a PR to the repo that manages the puppet configs (or some other system configuration tool that Release Operations decides to use for these workers).

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #42)

Status update on this: workflow is ready on our testing trees but some of the underlying tooling prevents us to ship to production for security reasons. We're working on this with RelOps and Taskcluster teams to find a solution to sort this out in early Q3.

Status update: attempt at fixing ^ is going to happen late in July in bug 1499054.

Depends on: 1499054

Bug 1499054 got fixed, we can now resurrect the work in here to enable flatpaks via generic-workers.

Met with @walac @pmoore and @fubar earlier today to talk about next steps. I was worried that there wasn't a clear way of provisioning the generic-workers for Linux in order to build the pre-requirements for building flatpaks. But sounds like there is \o/

So next steps for me:

Talked to @tomprince earlier in the week. I initially thought we can get away with some docker-worker + TC secrets to push but we'll need to use a scriptworker. So that means we'll need to build something similar to https://github.com/mozilla-releng/pushsnapscript but more leightweight since we can use cli-ed commands. The task itself should only pass as arguments the channel.

Other useful resources:

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #46)
I'm little concerned that you mention only infrastructure changes but there is no update about actual flatpak manifest. Could you post updated manifest in attachment so people could review it? It would be nice to base it on top of recently released freedesktop 19.08 runtime with two years of support.

(In reply to Emerson Bernier from comment #47)

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #46)
I'm little concerned that you mention only infrastructure changes but there is no update about actual flatpak manifest. Could you post updated manifest in attachment so people could review it? It would be nice to base it on top of recently released freedesktop 19.08 runtime with two years of support.

Thanks for raising this! Just to confirm, you mean upgrading from 18.08 which is what we were using to 19.08? If so, that makes a lot of sense. My main goal is to make sure the generic-worker infrastructure can built us what the docker-worker infrastructure did. Then, it should be a matter of adjusting those requirements and get up-to-date with most recent stable releases.

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #48)

Thanks for raising this! Just to confirm, you mean upgrading from 18.08 which is what we were using to 19.08?

Yes. Also I think some permissions listed in "finish-args" in manifest (https://bug1441922.bmoattachments.org/attachment.cgi?id=9002794) may be dropped:

"--filesystem=~/.config/dconf:ro" and "--env=DCONF_USER_CONFIG_DIR=.config/dconf" (see https://blogs.gnome.org/mclasen/2019/07/12/settings-in-a-sandbox-world/)

"--filesystem=home:rw" can be replaced by "--env=GTK_USE_PORTAL=1" which will also make possible opening file like pdf in external apps. In that case you may add "--filesystem=~/.mozilla" to keep access for existing profiles.

"--socket=session-bus" can be replaced by "--talk-name=org.a11y.*" and "--talk-name=org.freedesktop.Notifications".

@Emerson Bernier: Some notes on your proposal:

"--filesystem=home:rw" can be replaced by "--env=GTK_USE_PORTAL=1" which will also make possible opening file like pdf in external apps.

Setting "--env=GTK_USE_PORTAL=1" should not be necessary as GTK has a built-in check for detecting sandboxed environments. Afaik, Firefox calls the corresponding GTK API to detect sandboxed environments?
You should probably also keep "--filesystem=home:ro" (read-only, but not writable anymore) so that file:///-URIs continue to work. (Unless you actually don't want to support that anymore…)

"--socket=session-bus" can be replaced by "--talk-name=org.a11y.*" and "--talk-name=org.freedesktop.Notifications".

libnotify/glib on 19.08 may be using the org.freedesktop.portal.Notification interface already. In this case you'll probably not need the second permission either.

(In reply to ntninja from comment #50)

Setting "--env=GTK_USE_PORTAL=1" should not be necessary as GTK has a built-in check for detecting sandboxed environments. Afaik, Firefox calls the corresponding GTK API to detect sandboxed environments?

You are right, in flatpak, portals seems working automatically in FF.

You should probably also keep "--filesystem=home:ro" (read-only, but not writable anymore) so that file:///-URIs continue to work. (Unless you actually don't want to support that anymore…)

I would say that home/host permissions were a stop-gap solution until apps gain portal support. They're quite an abuse from security/privacy POV. Also leaking configs from host may cause various issues, for example some special .asoundrc can break alsa in sandbox. It's Mozilla call.

"--socket=session-bus" can be replaced by "--talk-name=org.a11y.*" and "--talk-name=org.freedesktop.Notifications".

libnotify/glib on 19.08 may be using the org.freedesktop.portal.Notification interface already. In this case you'll probably not need the second permission either.

I can't confirm this. Without "--talk-name=org.freedesktop.Notifications" I get only non-native notifications. I tested this on Thunderbird 68.1 in KDE 5.16 session. BTW: libnotify doesn't exists in runtime so Mozilla has to bundle it in manifest themselves in order to use it.

Anyway, this is kind of discussion I hoped for before it will be set & forget :)

You should probably also keep "--filesystem=home:ro" (read-only, but not writable anymore) so that file:///-URIs continue to work. (Unless you actually don't want to support that anymore…)

I would say that home/host permissions were a stop-gap solution until apps gain portal support. They're quite an abuse from security/privacy POV. Also leaking configs from host may cause various issues, for example some special .asoundrc can break alsa in sandbox. It's Mozilla call.

I agree about it being Mozilla's call. Just wanted point out that without "--filesystem=home:ro" hardly any local HTML file will be able to be opened correctly in the browser anymore, as the document portal will only mirror the HTML file itself but none of the auxiliary files (think the typical {page_name}_files directory created by offline-safes, …) it may use. NOT having this option set will effectively disable the ability to view any local HTML file correctly.
(I just wanted to have this particular fact pointed out in writing before anybody makes this decision lightly.)

Additional read-only exceptions for portable storage ("--filesystem=/media:ro") and system docs ("--filesystem=/usr/share/docs:ro --filesystem=/usr/local/share/docs:ro --filesystem=/opt:ro") should probably also be considered in this context as well as there are many (non-sandboxed) applications were “View help” will simply open the given page using the default browser/handler.

BTW: I'm not really a fan of adding all these exceptions, but if this shouldn't break existing usage I fear these exceptions may be necessary to keep around for a very very very long time.

"--socket=session-bus" can be replaced by "--talk-name=org.a11y.*" and "--talk-name=org.freedesktop.Notifications".

libnotify/glib on 19.08 may be using the org.freedesktop.portal.Notification interface already. In this case you'll probably not need the second permission either.

I can't confirm this. Without "--talk-name=org.freedesktop.Notifications" I get only non-native notifications. I tested this on Thunderbird 68.1 in KDE 5.16 session. BTW: libnotify doesn't exists in runtime so Mozilla has to bundle it in manifest themselves in order to use it.

Your right, I just checked the libnotify source-code and it pretty plain obvious that nobody tried to address this (or pretty much any other) issue yet.

GIO/GNotification has support for this, but as you have stated you are not using this.

Since libnotify is used by many applications, I guess it's best to just keep using the code and sandbox exception for now and hope that someone (maybe sponsored?) will create a patched version of this library that will “just work” as a drop-in replacement.

(In reply to ntninja from comment #52)

Additional read-only exceptions for portable storage ("--filesystem=/media:ro") and system docs ("--filesystem=/usr/share/docs:ro --filesystem=/usr/local/share/docs:ro --filesystem=/opt:ro") should probably also be considered in this context as well as there are many (non-sandboxed) applications were “View help” will simply open the given page using the default browser/handler.

BTW: I'm not really a fan of adding all these exceptions, but if this shouldn't break existing usage I fear these exceptions may be necessary to keep around for a very very very long time.

Users could still add more permissions to firefox flatpak locally (opt-in) however unlike snap flatpak doesn't have UX for adding more permissions easily so maybe this should be opt-out by default. In that case I would prefer single "--filesystem=host:ro" which will be easier to override that multiple of specific dirs.

There is also problem with /tmp - while opening files in external apps firefox download them to /tmp or TMPDIR at first. As /tmp from sandbox is inaccessible outside it, external app won't be able to find and open the file. This can be solved either by:

  1. "--filesystem=/tmp" - easiest but least secure solution
  2. Providing /app/bin/firefox[1] wrapper which sets TMPDIR to something readable. For example fedora sets[2] TMPDIR="$XDG_CACHE_HOME/tmp

[1] It can't be done with --env=TMPDIR= permissions because it won't work on systems where bubblewrap is installed as setuid binary, see https://github.com/flatpak/flatpak/issues/2641
[2] https://src.fedoraproject.org/rpms/firefox/blob/0adc4aecec45e195b0f6cbcdb1625839846482bc/f/firefox.spec#_710

(In reply to Emerson Bernier from comment #53)

Users could still add more permissions to firefox flatpak locally (opt-in) however unlike snap flatpak doesn't have UX for adding more permissions easily so maybe this should be opt-out by default. In that case I would prefer single "--filesystem=host:ro" which will be easier to override that multiple of specific dirs.

I agree with this conclusion: One broad permission makes it more obvious what is happening and is easier to revoke.

There is also problem with /tmp - while opening files in external apps firefox download them to /tmp or TMPDIR at first. As /tmp from sandbox is inaccessible outside it, external app won't be able to find and open the file. This can be solved either by:

  1. "--filesystem=/tmp" - easiest but least secure solution
  2. Providing /app/bin/firefox[1] wrapper which sets TMPDIR to something readable. For example fedora sets[2] TMPDIR="$XDG_CACHE_HOME/tmp

[1] It can't be done with --env=TMPDIR= permissions because it won't work on systems where bubblewrap is installed as setuid binary, see https://github.com/flatpak/flatpak/issues/2641
[2] https://src.fedoraproject.org/rpms/firefox/blob/0adc4aecec45e195b0f6cbcdb1625839846482bc/f/firefox.spec#_710

Can't we just patch Firefox to first try using $XDG_RUNTIME_DIR, before falling back to $TMPDIR, $TEMP and finally /tmp? The runtime is guaranteed to be per-user and the path of that variable will be set to /run/user/$(id -u)/app/org.mozilla.Firefox which is readable by any classic desktop app of the same user.

Create fp-gecko-1-gw-linux worker type

(In reply to ntninja from comment #54)

Can't we just patch Firefox to first try using $XDG_RUNTIME_DIR, before falling back to $TMPDIR, $TEMP and finally /tmp? The runtime is guaranteed to be per-user and the path of that variable will be set to /run/user/$(id -u)/app/org.mozilla.Firefox which is readable by any classic desktop app of the same user.

Yes, that's another solution. This one will have impact beyond flatpak though as unlike TMPDIR, XDG_RUNTIME_DIR is always set on modern systems so it will become de facto new firefox default.

Depends on: 1591387

Quick status update here - some of the things changed under the hood in the flatpak world that now enable the building of flatpaks without the need of --privileged flag. That puts back on the radar the possiblity of using docker-worker (and eventually generic-worker once the TC migration will be completed, but that's not blocking fow now) to build these.

In order to avoid polluting/spamming this bug (with many CCs), I'm tracking the work to make the staging releases of Firefox build and ship the Flatpaks in bug 1591387.

Big shout out to Valentin David and Robert McQueen who have helped streamline this process in the past few weeks.
More to come soon, in the next days.

Depends on: 1617612
Component: Release Automation: Other → Release Automation: Flatpak
Depends on: 1621083

Status update:

  • we were able to succesfully build flatpaks and publish them to the beta channel
  • we will continue to do so as part of the beta train of 75.0
  • for any conversations regarding flatpak manifest improvement that were happening in bug 1591387, let's use bug 1621083 instead

In order to test the currently-shipped Firefox Flatpaks on the Flathub's beta channel:

We still have a couple of issues to solve, like l10n and better building configuration, before we're ready to publish to stable channel (and be available via the Flathub UI).

Last but not least, there is a dedicated Release Automation: Flatpak component now, within RelEng product, feel free to file bugs against it.

Depends on: 1621053, 1621074

Status update:

  • we've started publishing Flatpak to the beta channel as instructed in comment 59
  • in the past few days, a couple of fixes landed in order to fix the branch we were using in building the flatpak, opening more ports and fixing l10n; latest Flathub's Flatpak (from beta channel) is up to date

Since this bug has a lot of folks CC'ed, I'm taking the opportunity to ask you all to give this a test should you like (instructions in comment above). Feel free to report any bugs.

Shout-out to @barthalion who's been super helpful in the last few weeks, providing lots of valuable help and insight.

Status update:

  • courtesy to @alexl (Alexander Larsson) who explained l10n configuration in bug 1621074 and @barthalion (Bartłomiej Piotrowski) who implemented this we now have functional l10n in Flatpaks

The last flatpak we have on Flathub's beta channel should contain most of the blocking fixes that we had, so please give that a test whenever you have some time.

Status update: we've officially shipped Firefox 75.0 Flatpak into the Flathub store on the stable channel - https://flathub.org/apps/details/org.mozilla.firefox

There are still some fixes/improvements to be addressed but those are tracked separately.
I think we can finally close this bug. If there are more concerns and other follow-ups, feel free to file a bug in this component.

Last but not least, I'd like to emphasize that this was a great team effort and coordination with the community folks for which I'm extremely thankful.

Status: NEW → RESOLVED
Closed: 5 years ago
Resolution: --- → FIXED

To sum-up, for posterity:

For now:

In the future, we might want to extend the support so that Firefox Nightly, Firefox Beta and even Devedition become standalone apps in the Flathub (stable) store, each with its own appid. But for now, we're going to let this bake for a while.

Blocks: 1623591

First of all, big thanks for your excellent work on bringing Firefox to flatpak with nice sandboxing and Portal support.

For whatever reason, the org.mozilla.firefox repository under Flathub github is missing and the publisher link on the Flathub page for Firefox (https://github.com/flathub/org.mozilla.firefox/graphs/contributors/) gives a 404 error. I think for such a prominent app it would be important to have it working. Also, it is not clear where people should submit bugs in the Flatpak release – flathub github or here in Bugzilla?

Everything else seem to be be working fine so far, but antialiasing for at least some fonts is broken on Debian 10.

See: https://github.com/flatpak/flatpak/issues/2861

The publisher page link issue is a bug in the flathub website frontend - Firefox is the first app that uses the direct publish system (previously Flathub has only added runtimes that way) so this needs an extra fix. There's a bug there for that issue: https://github.com/flathub/linux-store-frontend/issues/213

As for issues with the flatpak, please open them here and fix in the Firefox codebase - that's the point of publishing the bits from Mozilla directly!

(In reply to Pablo from comment #64)

For whatever reason, the org.mozilla.firefox repository under Flathub github is missing and the publisher link on the Flathub page for Firefox (https://github.com/flathub/org.mozilla.firefox/graphs/contributors/) gives a 404 error. I think for such a prominent app it would be important to have it working. Also, it is not clear where people should submit bugs in the Flatpak release – flathub github or here in Bugzilla?

Thanks a lot for raising these. Taking them one by one:

  1. re: https://github.com/flathub/org.mozilla.firefox 404 - this is somewhat expected. Submitting flatpaks is done via PR under normal circumstances and then each commit there regenerates the CI on Fltahub side as docs depict it. We didn't go down that path but instead used the API to submit our flatpak directly from our automation. The account was created manually by Flathub admins. We're aware of that and we're hoping to fix it server-side.

  2. re: publisher link - AFAIK, it's a fallout from previous point - it's currently tracked in bug 1627831 and hopefully will be fixed soon on server side.

  3. re: filing bugs - that's a very good question. I'll raise that with Flathub folks to see how to best divide the jurisdiction. The Flathub folks have all the know-how to deal with the under-the-hood mechanics in building flatpaks, while we're the owners of the release pipeline. However, in the past weeks, lots of good issues and feedback came via #flatpaks (both from Mozilla's https://chat.mozilla.org/#/room/#flatpaks:mozilla.org and from IRC freenode) that ended up as bugs here. While we, as RelEng, don't necessarily have the bandwidth/expertise to fix all the filed bugs, the community has been really great and helped us a lot with this. So I'd encourage you to file it here.

Everything else seem to be be working fine so far, but antialiasing for at least some fonts is broken on Debian 10.

There's also a related bug 1621915 on funky fonts.

See: https://github.com/flatpak/flatpak/issues/2861

Blocks: 1627831, 1621915

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #66)

  1. re: filing bugs - that's a very good question. I'll raise that with Flathub folks to see how to best divide the jurisdiction. The Flathub folks have all the know-how to deal with the under-the-hood mechanics in building flatpaks, while we're the owners of the release pipeline. However, in the past weeks, lots of good issues and feedback came via #flatpaks (both from Mozilla's https://chat.mozilla.org/#/room/#flatpaks:mozilla.org and from IRC freenode) that ended up as bugs here. While we, as RelEng, don't necessarily have the bandwidth/expertise to fix all the filed bugs, the community has been really great and helped us a lot with this. So I'd encourage you to file it here.

Hi, there are bug [1] reported against Red Hat Firefox flatpak [2]. Main goal of the RH flatpak repo was to provide FF flatpak until it's available from Mozilla which is done now so we (RH) should continue to support Mozilla FF flatpak only.

I'll discuss that with RH folks how we can help here. Meanwhile Firefox flatpak linux integration bugs can be reported as Core/Widget:GTK where will be visible to me and can be fixed.

[1] https://github.com/xhorak/firefox-devedition-flatpak/issues
[2] https://firefox-flatpak.mojefedora.cz/

btw. Bug 1278719 also tracks some Firefox specific flatpak issues although they may be related to flatpacks from https://firefox-flatpak.mojefedora.cz/. Mihai, do you mind to use the Bug 1278719 to also track Mozilla flatpak issues which are on Firefox side or is there any other tracker?
Thanks.

Flags: needinfo?(mtabara)

(In reply to Martin Stránský [:stransky] from comment #68)

Hi, there are bug [1] reported against Red Hat Firefox flatpak [2]. Main goal of the RH flatpak repo was to provide FF flatpak until it's available from Mozilla which is done now so we (RH) should continue to support Mozilla FF flatpak only.

I'll discuss that with RH folks how we can help here. Meanwhile Firefox flatpak linux integration bugs can be reported as Core/Widget:GTK where will be visible to me and can be fixed.

Hi Martin, thanks for reaching out.

  • I see a very good list of issues there in the Redhat's flatpak. I wonder how many of those are still ongoing in the official Flatpak. Should we run through those at some point to see if they are still ocurring in the existing flatpak and transfer them in bugzilla?
  • I'm happy to use the Core/Widget:GTK component for stuff that's related to Linux integration. And maybe keep this component for issues related to release/shipping the flatpak? Since I'm actively triaging this component, I will try to exercise good judgment to triage correctly between them. Would that work for you?
  • with respect to https://firefox-flatpak.mojefedora.cz/ - I'm wondering whether there is a way to migrate those users to the official flatpak?

(In reply to Martin Stránský [:stransky] from comment #69)

btw. Bug 1278719 also tracks some Firefox specific flatpak issues although they may be related to flatpacks from https://firefox-flatpak.mojefedora.cz/. Mihai, do you mind to use the Bug 1278719 to also track Mozilla flatpak issues which are on Firefox side or is there any other tracker?

Absolutely. To be honest, I've been using this bug for a long time as a "meta" tracker one and had some related children bugs in which I worked. I'm happy to use that bug from now one since this is resolved anyway.

Flags: needinfo?(mtabara)

Moving bug 1621915, bug 1623591 and bug 1627831 as deps to bug 1278719 and we'll keep the latter as main meta/tracking bug for any further issues.

No longer blocks: 1621915, 1623591, 1627831
Component: Release Automation: Flatpak → Release Automation: Other
Attachment #9098246 - Attachment is obsolete: true

Comment on attachment 9002794 [details] [diff] [review]
[in-tree] Generate multilocale flatpaks

Found in triaging, cleaning up my queues of requests.
This landed a good while back, nothing to do further more.

Attachment #9002794 - Flags: review?(robert)
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: