1297531 - Run tests against Firefox snaps

Status: RESOLVED WONTFIX

(Release Engineering Graveyard :: Release Automation: Snap, defect, P3)

Description

[Rail Aliiev [:rail]]

Filing just a place holder, not quite sure what we want here.

Comment 1

[Joel Maher ( :jmaher ) (UTC -8)]

snap builds are run on mozilla-release:
https://treeherder.mozilla.org/#/jobs?repo=mozilla-release&searchStr=snap&selectedJob=211866394

There is a push and a repackage job, which job needs tests run against it?  I don't see a build released, maybe I don't understand what this is.  I would really like more information before weighing in on what we would need to do for running some automated tests.

:jld, can you help clarify what snap is and what artifacts are generated and how to use them?

Comment 2

[Johan Lorenzo [:jlorenzo]]

I can answer, I'm the one who set these jobs up.

The repackage job is the one you're looking for. More precisely, the relevant artifact is target.snap[1]. target.snap is what's published to the Snap store[2] thanks to the push job[3].

One issue that may hold testing is: the repackage job is only run during release promotion. We should enable snaps on push with [4]. publication should need no change, as far as I know.

In case you're wondering, Snaps are not signed (bug 1417960). 

[1] For instance: https://queue.taskcluster.net/v1/task/XVLJhlQzSoelNqy_ScTPTg/runs/0/artifacts/public/build/target.snap
[2] https://snapcraft.io/firefox
[3] For instance: https://tools.taskcluster.net/groups/NZF0P14CRziO4gby6xrhRQ/tasks/UeXzL35VQa2uLZqrJGcXog/runs/1/logs/public%2Flogs%2Flive_backing.log#L1455
[4] https://searchfox.org/mozilla-central/source/taskcluster/ci/release-snap-repackage/kind.yml#18-19

Comment 3

[Johan Lorenzo [:jlorenzo]]

Please let me know if you have any questions :)

Comment 4

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

The snap-repackage job doesn't work on Try currently: it tries to upload langpacks to addons.allizom.org, but it fails because the version number is in the wrong format (it's concatenated with a build ID, like "65.0a1buildid20181114175259"), plus some errors about “[w]ords … that violate the Mozilla conditions of use”; see the logs at [1] for an example.

I considered trying to write a variant of the job that would just take a regular build and skip all the release-pipeline stuff, but I ran into a problem with Snap itself: Firefox apparently can't be installed from just a .snap file, because even with the --dangerous option to disable signature checking, it's silently denied access to the `browser-support` component and crashes early in startup; it looks like we need a snap-declaration assertion[2] to grant that, and there's a whole PKI system around those.

(It would probably work with --devmode, which disables confinement, but the goal here — as least as I see it — is specifically to test whether Firefox has regressions under the normal confinement policy.)

And then there's the question of actually running tests on these.  I don't know a lot about our test harnesses, but my understanding is that we generally need to add command-line arguments and set environment variables, and might need to allow inbound loopback network connections for some (most?) types.  I think Snap can do that (one of the examples in the docs is a webserver listening on port 80), but some glue would need to be written.


[1] https://treeherder.mozilla.org/#/jobs?repo=try&revision=601e80eb58d187c7c35b8392e2a599d7b92f459b&selectedJob=211819665
[2] https://docs.ubuntu.com/core/en/reference/assertions/snap-declaration

Comment 5

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

…I guess I should also clarify what Snap is, since that was part of the question: it's a package manager for Linux where applications are run in a sandbox with all their dependencies, so the same package can run on any distribution.  (See also Flatpak, which is a roughly similar idea.)  I think technically the package manager part is called “Snappy”, but the command for managing it is named `snap`.

That sandbox also imposes restrictions for security/privacy/permissions purposes, which have caused / are causing some known bugs (bug 1461848, bug 1472589, and bug 1501895 are the ones I've been involved with).  It's separate from our sandboxing, but there's overlap in the technologies used.

And the part of this that makes things nontrivial for testing: as part of this sandboxing, the executable isn't run directly; it's launched by messaging a daemon (snapd) which sets up the actual launch (and there seems to be some more indirection there, as the application isn't a direct descendent of snapd either).

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

in this case we wouldn't be able to launch and monitor the process?  also running firefox with a testing profile and a url to test against seems much more difficult.

How do you install the snap packages?  Is there a way to automate that (currently we do testing via unpacking a .zip file or similar bundle)?

Comment 7

[Olivier Tilloy]

(In reply to Jed Davis [:jld] ⟨⏰|UTC-5⟩ ⟦he/him⟧ from comment #4)
> I considered trying to write a variant of the job that would just take a
> regular build and skip all the release-pipeline stuff, but I ran into a
> problem with Snap itself: Firefox apparently can't be installed from just a
> .snap file, because even with the --dangerous option to disable signature
> checking, it's silently denied access to the `browser-support` component and
> crashes early in startup; it looks like we need a snap-declaration
> assertion[2] to grant that, and there's a whole PKI system around those.

When side-loaded with --dangerous, the browser-support plug must be manually connected: `snap connect firefox:browser-sandbox`.
The assertion is only required for this to happen automatically at install time, when installed from a trusted store.

Comment 8

[Olivier Tilloy]

(In reply to Joel Maher ( :jmaher ) (UTC-4) from comment #6)
> in this case we wouldn't be able to launch and monitor the process?  also
> running firefox with a testing profile and a url to test against seems much
> more difficult.

I am not familiar with the firefox testing infrastructure and tools, but launching firefox installed as a snap with a custom profile and a URL is as simple as:

    snap run firefox -P foobar http://example.org

Comment 9

[Jed Davis [:jld] ⟨⏰|UTC-7⟩ ⟦he/him⟧]

(In reply to Olivier Tilloy from comment #7)
> When side-loaded with --dangerous, the browser-support plug must be manually
> connected: `snap connect firefox:browser-sandbox`.
> The assertion is only required for this to happen automatically at install
> time, when installed from a trusted store.

Thank you; I'm now able to run a Snap-repacked Try build, so it should be relatively simple to automate that in a way that works with non-release builds.  (For Taskcluster, I'm told it's a little difficult to have a job that consumes a build but still acts like a build in terms of having tests based on it, but it should be simpler to have a variant of the build job that produces a .snap instead of a .tar.bz2.)

(In reply to Jed Davis [:jld] ⟨⏰|UTC-5⟩ ⟦he/him⟧ from comment #5)
> the executable isn't run directly; it's launched by
> messaging a daemon (snapd) which sets up the actual launch

This is incorrect; I misunderstood the role of snapd.  The executable is exec'ed by the `snap run` process (by way of snap-confine and snap-exec) so it should be a more or less drop-in replacement as far as process monitoring is concerned.  And I've confirmed that it inherits environment variables normally, so that should also work.

Comment 10

[Julien Cristau [:jcristau]]

Calling this wontfix since firefox snaps are not built by mozilla these days.