Closed Bug 1591387 Opened 6 years ago Closed 5 years ago

build docker-worker flatpaks in Firefox staging releases

Categories

(Release Engineering :: Release Automation, task)

Product:
Release Engineering
Component:
Release Automation
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(firefox75 fixed, firefox76 fixed)

Status:
RESOLVED FIXED
Tracking Flags:
Tracking Status
firefox75 --- fixed
firefox76 --- fixed

People

(Reporter: mtabara, Assigned: mtabara)

References

Details

Attachments

(1 file)

Bug 1591387 - add flatpak release automation. r=rail
47 bytes, text/x-phabricator-request
jcristau
: approval-mozilla-beta+
Details | Review

We're now off Docker-worker in building flatpaks and instead relying on generic-workers. That means we need to build the same requirements that we used to build with Docker-worker, but within the host machine itself.

There may be some adjustments we'll need to make.
This bug is tracking the work that I started a while ago.

Blocks: 1441922
Summary: provision flatpaks generic-worker with software pre-requisites needed for building flatpaks → build docker-worker flatpaks in Firefox staging releases

Some of the things changed under the hood in the flatpak world that now enable us to build the flatpaks without the need of --privileged argument. That puts back on the map the possiblity of using docker-worker (and eventually generic-worker once the TC migration will be completed, but that's not blocking now) like we obtain it earlier in 2019.

I'm switching this bug to track that work.
Shout out to Valentin David and Robert McQueen who have helped streamline this process in the past few weeks.

I'm tracking this progress for now on https://hg.mozilla.org/projects/maple

Status update - with Valentin's help I managed to finally build the flatpaks locally. I moved them in the tree on the maple project where things are looking promising so far. We have green image built in Taskcluster - https://firefox-ci-tc.services.mozilla.com/tasks/KNDZ5ALvSEaHnm2NVhhosg.

I pushed the rest of the code which should build the flatpaks as part of the promotion phase of Firefox. Waiting to see how/if the flatpak is successfully generated.

Okay, so I moved out of maple because I've hit some automation issues and we're actually trying to get off of maple :)

I moved on our try and submitted a try staging release containing all the required changes.

  • the try staging release is here
  • the in-tree Taskcluster docker image is here
  • the repackaging job that builds the flatpak is here

While the flatpak seems to have been built successfully - (full raw log is here I couldn't help but notice that there's some:

Installing 8/8… ████████████████████ 100%  594.2 kB/sespace failed: Operation not permitted
bwrap: Creating new namespace failed: Operation not permitted
bwrap: Creating new namespace failed: Operation not permitted

warning/errors somewhere in the logs, after installing some of the bits, namely here

I'll give this a test in a virtual box to see if it actually built a valid flatpak. The tar.gz is here.

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #4)

While the flatpak seems to have been built successfully - (full raw log is here I couldn't help but notice that there's some:

Installing 8/8… ████████████████████ 100%  594.2 kB/sespace failed: Operation not permitted
bwrap: Creating new namespace failed: Operation not permitted
bwrap: Creating new namespace failed: Operation not permitted

warning/errors somewhere in the logs, after installing some of the bits, namely here

Maybe Valentin can comment but I guess it's one of two things - either the triggers, which are updating icon caches and mime type registries. We don't need them for the Base App (it's not exporting any icons and stuff, and we don't need them for a build) but because Flatpak considers it to be an app, it probably adds the triggers to the transaction. I don't see a way to disable from the cmdline but the library does support it. Maybe we need to ask Alex for a --no-triggers cmdline option.

One of them could also be the openh264 extension which is an "extra data" flatpak - this means the binary is fetched directly from Cisco and then a script is run (sandboxed) to unpack and install it. We don't need this at build time, so we can add "flatpak mask org.freedesktop.Platform.openh264" command before the install, and it should remove at least one of these errors. Failures installing the extensions like this codec are not considered fatal as Flatpaks are required to degrade gracefully (they are not dependencies).

I'll give this a test in a virtual box to see if it actually built a valid flatpak. The tar.gz is here.

The flatpak works! The main "bug" is that we need to add "--command=firefox" to the "flatpak build-finish" command, otherwise when you run the flatpak it defaults to the notify-send command. :D

Are we also somehow missing the site policy which we wanted to add, which would disable the built in updater? I get an error saying Flatpak is not able to update.

I've addressed the following in a follow-up try staging release:

  • remove sdk/pltf installation. Instead meta file manually - much faster task
  • add policy to disable updates - should improve the UI when hovering over the Flatpak UI

I have also unsuccessfully added the --comand=firefox but it barfs out with:

Not exporting share/icons/hicolor/48x48/apps/org.mozilla.firefox.png, non-allowed export filename
More than one executable found
Using notify-send as command
Please review the exported files and the metadata
+ --command=firefox
runme_latest.sh: line 115: --command=firefox: command not found

Instead, I've temporarily hacked around this by removing the notify-send binary. Once we fix BaseApp, we should be good to go and I'll revert the temp hack.

The new resulting flatpak is here for testing.

The package job was much faster since it skips all the SDK/platform installation and also manually configures the metadata for flatpak so it's cleaner in the code as well.

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #7)

I've addressed the following in a follow-up try staging release:

  • remove sdk/pltf installation. Instead meta file manually - much faster task

Heh, smart!

  • add policy to disable updates - should improve the UI when hovering over the Flatpak UI

Cool.

I have also unsuccessfully added the --comand=firefox but it barfs out with:

Not exporting share/icons/hicolor/48x48/apps/org.mozilla.firefox.png, non-allowed export filename
More than one executable found
Using notify-send as command
Please review the exported files and the metadata
+ --command=firefox
runme_latest.sh: line 115: --command=firefox: command not found

This looks like you forgot a \ on the line before, so it tried to execute it as a new command rather than appended to the flatpak build-finish command?

Instead, I've temporarily hacked around this by removing the notify-send binary. Once we fix BaseApp, we should be good to go and I'll revert the temp hack.

The new resulting flatpak is here for testing.

The package job was much faster since it skips all the SDK/platform installation and also manually configures the metadata for flatpak so it's cleaner in the code as well.

Neat!

(In reply to Robert McQueen from comment #8)

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #7)

I've addressed the following in a follow-up try staging release:

  • remove sdk/pltf installation. Instead meta file manually - much faster task

Heh, smart!

302 to Valentin, he had the idea :)

  • add policy to disable updates - should improve the UI when hovering over the Flatpak UI

Cool

Does it work well for you this time in terms of updates?

I have also unsuccessfully added the --comand=firefox but it barfs out with:

Not exporting share/icons/hicolor/48x48/apps/org.mozilla.firefox.png, non-allowed export filename
More than one executable found
Using notify-send as command
Please review the exported files and the metadata
+ --command=firefox
runme_latest.sh: line 115: --command=firefox: command not found

This looks like you forgot a \ on the line before, so it tried to execute it as a new command rather than appended to the flatpak build-finish command?

Doh, that must be it! I'll give it a try again!

I'm on to finalizing publishing scriptworker then if nothing's left to be done here.
Could you remind me again:
a) does Flathub have multiple channels? We ought to start with beta and then ride the trains to stable?
b) how hard would it be to ramp-up that Flathub staging environment? If it's too hard, I'm thinking we could also start publishing fake releases until we streamline the process or so. For now, I've hacked automation to download actual production releases in order to ship staging flatpaks just to make sure we're as close as possible to a production Firefox tar.gz

:ramcq: what do you think?

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #9)

a) does Flathub have multiple channels? We ought to start with beta and then ride the trains to stable?

IMO if possible we should start with nightly and go from there. We didn't do that for snap and it's sad (means that any breakage flies under the radar until it hits beta, for one thing).

(In reply to Julien Cristau [:jcristau] from comment #10)

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #9)

a) does Flathub have multiple channels? We ought to start with beta and then ride the trains to stable?

IMO if possible we should start with nightly and go from there. We didn't do that for snap and it's sad (means that any breakage flies under the radar until it hits beta, for one thing).

Ideally yes, but IIRC fromearly 2019, Flathub doesn't support nightly. I may be wrong.
If not, the Flathub staging environment worked pretty smooth last Spring, I was able to push there a Firefox version and it wasn't usable from the store. Maybe they can resurrect that and we streamline there the process?

I wonder what other apps have done though (Slack, Spotify, etc). I'll look around...

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #11)

(In reply to Julien Cristau [:jcristau] from comment #10)

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #9)

a) does Flathub have multiple channels? We ought to start with beta and then ride the trains to stable?

IMO if possible we should start with nightly and go from there. We didn't do that for snap and it's sad (means that any breakage flies under the radar until it hits beta, for one thing).

Ideally yes, but IIRC fromearly 2019, Flathub doesn't support nightly. I may be wrong.
If not, the Flathub staging environment worked pretty smooth last Spring, I was able to push there a Firefox version and it wasn't usable from the store. Maybe they can resurrect that and we streamline there the process?

I wonder what other apps have done though (Slack, Spotify, etc). I'll look around...

Spoke with folks on IRC's #flatpaks. They could add a nightly channel but that complicates our side as nightly and beta are not exactly the same. I'd have to change the entire logic in the runme script to handle them separately as we'd need different locations for xpi + tar.gz between the both. Not impossible though.

I'm focusing now on finishing the pushflatpakscript scriptworker by mid-next-week and then we can iterate against Flathub.

Adjust runme to be executable

Rm sdk/pltf install.Manually add meta

Enforce firefox run instead of notify

Add policy to disable updates

Temp hack to default to firefox instead of notify-send

Fix mach linters

Remove firefox command hack. Proper fix

Remove duplicate cmd in runme

Fix indentantion in kind

Fix more linters

Note to self:

  • latest staging release is here
  • its most recent flatpak can be downloaded and tested here

This contains some fixes & improvements. I'll push the patch in a bit too to keep track of what I'm doing.

Few notes about flatpak manifest:

You may consider removing:

--filesystem=xdg-run/dconf
--filesystem=xdg-config/dconf:ro
--talk-name=ca.desrt.dconf
--env=DCONF_USER_CONFIG_DIR=.config/dconf

They aren't needed with xdg-desktop-portal 1.1+ (however ubuntu bionic and xenial are on 1.0.3 without ppa)
https://blogs.gnome.org/mclasen/2019/07/12/settings-in-a-sandbox-world/

Remove:
--talk-name=org.freedesktop.FileManager1 (It's used by Endless but it was added there without explanation. Fedora doesn't set it for their ff flatpak. I don't know why it would be needed.)

Add:
--socket=wayland (For users who want to run with MOZ_ENABLE_WAYLAND=1)
--talk-name=org.freedesktop.Notifications (I think it's required for native notifications to work)

To verify why they're needed:

--talk-name="org.gtk.vfs.*"
--system-talk-name=org.freedesktop.NetworkManager 
--talk-name=org.gnome.SessionManager

Additional things to consider (need more info):
--talk-name=org.a11y.Bus vs --talk-name=org.a11y.* (?)
--filesystem=xdg-public-share:ro (multiple files upload?)
--device=all (u2f support?)

(In reply to Emerson Bernier from comment #15)

Remove:
--talk-name=org.freedesktop.FileManager1 (It's used by Endless but it was added there without explanation. Fedora doesn't set it for their ff flatpak. I don't know why it would be needed.)

https://dxr.mozilla.org/mozilla-central/source/xpcom/io/nsLocalFileUnix.cpp#1922

Add:
--socket=wayland (For users who want to run with MOZ_ENABLE_WAYLAND=1)
--talk-name=org.freedesktop.Notifications (I think it's required for native notifications to work)

To verify why they're needed:

--talk-name="org.gtk.vfs.*"

This one I am not so sure -

--system-talk-name=org.freedesktop.NetworkManager

https://dxr.mozilla.org/mozilla-central/source/netwerk/wifi/nsWifiScannerDBus.cpp#40

--talk-name=org.gnome.SessionManager

https://dxr.mozilla.org/mozilla-central/source/widget/gtk/WakeLockListener.cpp#29


Additional things to consider (need more info):
`--talk-name=org.a11y.Bus vs --talk-name=org.a11y.*` (?)

a11y is handled by Flatpak. (This wouldn't be sufficient without also proxying the a11y bus socket anyway.)

--filesystem=xdg-public-share:ro (multiple files upload?)

Wat?

--device=all (u2f support?)

I'm not sure if this is sufficient without also permitting udev or similar to work. This might need a device portal to enumerate.

--device=all (u2f support?)

U2F does not work at all inside Flatpak at the moment, so I don't believe this would be useful.

--filesystem=xdg-public-share:ro (multiple files upload?)

Is this not already covered by --filesystem=home?

--talk-name="org.gtk.vfs.*"

This was based off of the upstream manifest at the time (not sure if it's changed since then). I believe it was likely added for non-portal file choosers to be able to see mounted GVFS paths (such as connected Google Drive accounts). Although Firefox does now use the portals, there are still some APIs (e.g. allowdirs or the experimental Native File System API) that would require directories to be opened, which Flatpak does not currently support use of portals for.

(In reply to Robert McQueen from comment #16)

(In reply to Emerson Bernier from comment #15)

Remove:
--talk-name=org.freedesktop.FileManager1 (It's used by Endless but it was added there without explanation. Fedora doesn't set it for their ff flatpak. I don't know why it would be needed.)

https://dxr.mozilla.org/mozilla-central/source/xpcom/io/nsLocalFileUnix.cpp#1922

Does it still needed when firefox doesn't have direct fileystem access (like in proposed manifest) and handle things through portals?

Additional things to consider (need more info):
--talk-name=org.a11y.Bus vs --talk-name=org.a11y.* (?)

a11y is handled by Flatpak. (This wouldn't be sufficient without also proxying the a11y bus socket anyway.)

Do you mean this permission won't work and can be dropped?

--filesystem=xdg-public-share:ro (multiple files upload?)

Wat?

I imagined case when user want to select multiple files to upload somewhere (AFAIK portal supports only one file at time) but nevermind.

(In reply to Ryan Gonzalez from comment #17)

--filesystem=xdg-public-share:ro (multiple files upload?)

Is this not already covered by --filesystem=home?

There is no --filesystem=home in proposed manifest: https://phabricator.services.mozilla.com/D59561

Martin:

Can you offer any thoughts on the last few comments?

Flags: needinfo?(stransky)

Mike, thanks a lot. Adding Jan who is our flatpak expert.

Flags: needinfo?(stransky)
Flags: needinfo?(jhorak)

(In reply to Emerson Bernier from comment #18)

(In reply to Robert McQueen from comment #16)

(In reply to Emerson Bernier from comment #15)

Remove:
--talk-name=org.freedesktop.FileManager1 (It's used by Endless but it was added there without explanation. Fedora doesn't set it for their ff flatpak. I don't know why it would be needed.)

https://dxr.mozilla.org/mozilla-central/source/xpcom/io/nsLocalFileUnix.cpp#1922

Does it still needed when firefox doesn't have direct fileystem access (like in proposed manifest) and handle things through portals?

Fedora use --socket=session-bus for revealing files, having just --talk-name=org.freedesktop.FileManager1 is probably better. What works is revealing files in the given --filesystem=* locations (most notably is xdg-download). If you download the file using the portal (like choosing download location on your own outside of the --filesystem) it does not work ATM. The user is not even notified that the saving failed, so having disabled --filesystem=xdg-home is probably bad idea. We need to fix it, but that's most likely non-trivial - as long as the Firefox is receiving a filedescriptor of the file for writing, which Firefox is not exactly expecting. Even after that the revealing file/opening the downloaded file won't work because of how the flatpak/portals are designed :(.

Additional things to consider (need more info):
--talk-name=org.a11y.Bus vs --talk-name=org.a11y.* (?)

a11y is handled by Flatpak. (This wouldn't be sufficient without also proxying the a11y bus socket anyway.)

In my setup I need to add --talk-name=org.a11y.Bus to make screen reader working.

Flags: needinfo?(jhorak)

(In reply to Jan Horak [:jhorak] from comment #21)

Fedora use --socket=session-bus for revealing files, having just --talk-name=org.freedesktop.FileManager1 is probably better. What works is revealing files in the given --filesystem=* locations (most notably is xdg-download). If you download the file using the portal (like choosing download location on your own outside of the --filesystem) it does not work ATM. The user is not even notified that the saving failed, so having disabled --filesystem=xdg-home is probably bad idea. We need to fix it, but that's most likely non-trivial - as long as the Firefox is receiving a filedescriptor of the file for writing, which Firefox is not exactly expecting. Even after that the revealing file/opening the downloaded file won't work because of how the flatpak/portals are designed :(.

Do you mean saving files to arbitrary home dirs through portal without "--filesystem=home" doesn't work? I just tested this on fedora ff flatpak and it worked.

Flags: needinfo?(jhorak)

(In reply to Emerson Bernier from comment #22)

Do you mean saving files to arbitrary home dirs through portal without "--filesystem=home" doesn't work? I just tested this on fedora ff flatpak and it worked.

I'm not him, but here are some things that definitely don't work without that permission:

  1. If you set a default downloads directory in about:preferences#general, Firefox will automatically attempt to create files in that directory without asking for the download location first. In this case the document portal will not be consulted, so whatever location you have set will have to be writable.
  2. Also, Firefox will remember your list of recent downloads across restarts and gives you the option open those items on the next start, which requires read access. (xdg-desktop-portal explicitly requires you to pass an open FD to the file you which to have opened to prove that you actually have that access.)

(In reply to ntninja from comment #23)

  1. If you set a default downloads directory in about:preferences#general, Firefox will automatically attempt to create files in that directory without asking for the download location first. In this case the document portal will not be consulted, so whatever location you have set will have to be writable.

This isn't possible even on fedora which gives read-only home access. Users who change default download dir may add write access there accordingly themselves. In general allowing to read/write in arbitrary paths make whole sandboxing rather pointless.

(In reply to Emerson Bernier from comment #24)

(In reply to ntninja from comment #23)

  1. If you set a default downloads directory in about:preferences#general, Firefox will automatically attempt to create files in that directory without asking for the download location first. In this case the document portal will not be consulted, so whatever location you have set will have to be writable.

This isn't possible even on fedora which gives read-only home access. Users who change default download dir may add write access there accordingly themselves. In general allowing to read/write in arbitrary paths make whole sandboxing rather pointless.

Then the option is broken on Fedora, even if for very understandable reasons. Mid-term this will need https://github.com/flatpak/xdg-desktop-portal/issues/200 to be implemented for a proper fix, short term the button to change the location should probably just be disabled if the home directory is read-only.

Depends on: 1617612
Attachment #9120162 - Attachment description: Bug 1591387 - add flatpak stub for try testing → Bug 1591387 - add flatpak release automation. r=rail
Pushed by mtabara@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/bcabd259e2a4 add flatpak release automation. r=rail

Comment on attachment 9120162 [details]
Bug 1591387 - add flatpak release automation. r=rail

Beta/Release Uplift Approval Request

  • User impact if declined: None. This adds the automation bits to build & package Firefox into flatpaks to be published to beta channel on Flathub. Plan is to iterate for the duration of 75.0 cycle to iron out all the other remaining issues and target 7th of April as the first official Firefox Flatpak.
  • Is this code covered by automated tests?: No
  • Has the fix been verified in Nightly?: No
  • Needs manual test from QE?: No
  • If yes, steps to reproduce:
  • List of other uplifts needed: None
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky):
  • String changes made/needed:
Attachment #9120162 - Flags: approval-mozilla-beta?

We've wrapped up the work in adding the automated publisher in bug 1617612. Those are now up and running for both staging/production environment. Once the aforementioned patch lands to beta, we will start shipping Firefox beta releases to Flathub's beta channel (no showing on the UI as Flathub doesn't support it).

Plan going forward is to iron out all leftovers issues with respect to building the Flatpaks, during 75.0 cycle so that we make it in time for 7th of April when we have planned the 75.0 release. If all goes well, we'd like to target that as our first official Flatpak on the stable channel in Flathub.

I might file another bug/bugs to iron the remaining issues.

Before I forget, in order to test the currently-shipped Firefox Flatpaks on the beta channel:

Component: Release Automation: Other → Release Automation: Flatpak
QA Contact: sfraser
Blocks: 1621083

@all: would you mind if we continue the conversation on cleaning/improving the Flatpak manifest in bug 1621083? I'd say we leave this bug serve its original purpose and let it be closed, now that we have a functional flatpak building story in try staging releases.

Feel free to CC there and let's continue the conversation there.

Status: NEW → RESOLVED
Closed: 5 years ago
Flags: needinfo?(jhorak)
Resolution: --- → FIXED

Comment on attachment 9120162 [details]
Bug 1591387 - add flatpak release automation. r=rail

publish flatpak betas, approved for 75.0b3

Attachment #9120162 - Flags: approval-mozilla-beta? → approval-mozilla-beta+

It is great that a flatpak is now available. As a matter of design, it seems to be lacking native messaging support just as the snap package is. Is there any way to fix this? Because without native message, KeePassxc Browser and other add-ons that need it don't work.

(In reply to Johannes Rohr from comment #33)

It is great that a flatpak is now available. As a matter of design, it seems to be lacking native messaging support just as the snap package is. Is there any way to fix this? Because without native message, KeePassxc Browser and other add-ons that need it don't work.

To be honest, I don't have knowledge on this, I'd need to find out someone at Mozilla who does.

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #34)

(In reply to Johannes Rohr from comment #33)

It is great that a flatpak is now available. As a matter of design, it seems to be lacking native messaging support just as the snap package is. Is there any way to fix this? Because without native message, KeePassxc Browser and other add-ons that need it don't work.

To be honest, I don't have knowledge on this, I'd need to find out someone at Mozilla who does.

Thanks for your response. Sounds like I best submit a separate bug for that, I just don't know against which package. Again, this seems to affect both snap and flatpak alike.

(In reply to Johannes Rohr from comment #33)

It is great that a flatpak is now available. As a matter of design, it seems to be lacking native messaging support just as the snap package is. Is there any way to fix this? Because without native message, KeePassxc Browser and other add-ons that need it don't work.

Unfortunately the way native messaging hosts are currently designed is conceptually incompatible with the way (Flatpak's) sandboxes work:

  • Discovery: Firefox expects foreign applications to place manifest files its own data directory (~/.mozilla/native-messaging-hosts to be exact). This is incompatible with what the sandbox restrictions are supposed to do, but an easy workaround would be designate a new shared directory that all interested applications would request access to and write their manifests there. Only downside would be that different applications could then mess up each others native-messaging files, but as a quick measure it would do.
  • Interaction: This is were things get really bad. The native messaging manifests contain a path to a program that Firefox should execute, with the implicit assumption that all applications share the same process and filesystem environments. Since each application runs in its own sandbox however, that assumption is no longer true. Apps could start placing binaries into shared folders, but then those binaries would have to be run inside Firefox's sandbox (thereby completely subverting all sandbox security guarantees) and they would have to have complete access to everything they provide to the Firefox extension from the Firefox sandbox (thereby requiring them to basically be dummy scripts that just forward whatever they receive to some other API on their parent application that does proper permissioning, duplicating that work between every application offering services)

In short: To properly support this we need a discovery/broker mechanism inside xdg-desktop-portal so that apps can advertise support for native-messaging and a D-Bus protocol to replace the current JSON/stdio based communication interface. Support for this needs be part of Firefox, xdg-desktop-portal and every interested app unfortunately. (Using D-Bus service activation and bus enumeration, support xdg-desktop-portal could in theory be skipped, but its probably better to solve this properly once, rather than cooking up a half-baked solution.)

(In reply to ntninja from comment #36)

Thanks for your informative comments! I undderstand that there isn't going to be a quick fix, but given that Flatpak and Snap are getting ever more relevant and provide a way of keeping your firefox up-to-date without breaking the system, I submitted a feature request: #1621763

Hm, markdown doesn't work as expected.... This is my feature request: bug #1621763
Hope this gets rendered as an URL now...

Component: Release Automation: Flatpak → Release Automation: Other
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: