1139218 - Expose moz.build metadata on hg.mozilla.org

Status: RESOLVED FIXED

(Developer Services :: Mercurial: hg.mozilla.org, defect)

Description

[Gregory Szorc [:gps]]

The metadata stored in Files in moz.build files is useful both for in-tree and out-of-tree consumers.

While some Files metadata may eventually be sensitive to the commit it is attached to, a lot of consumers only care about the state of the metadata on "tip."

With that use case in mind, it was always my intention to deploy a web service that returned metadata for a set of input files. Essentially an HTTP endpoint for `mach file-info`.

This bug tracks writing that web service. I plan to throw the web service behind `mach file-info`. e.g. `mach file-info serve`.

Comment 1

[Gregory Szorc [:gps]]

Attachment: MozReview Request: Bug 1139218 - Add an HTTP server for exposing files metadata

Bug 1139218 - Add an HTTP server for exposing files metadata

Data from Files sub-contexts in moz.build files is useful for services
running on other machines. These services shouldn't be required to have
a full copy of the source checkout in order to query metadata. To make
it easier to access this data, we'll provide a web service.

This patch implements a WSGI web service for serving up files info
as JSON. The web service is constructed by specifying a mapping of
symbolic names to paths of Mercurial repositories.

A mach command for creating a simple HTTP server out of the WSGI
application has been created.

Tests for the web service have been added. However, they don't run
as part of Firefox's build automation because they require a bunch of
3rd party Python packages, which we have no inclination of checking
into the tree at this time because their utility is marginal.

Comment 2

[Gregory Szorc [:gps]]

https://reviewboard.mozilla.org/r/10087/#review8871

dminor: I'm very interested in your feedback that the HTTP API and response data is satisfactory to your needs.

Comment 3

[Gregory Szorc [:gps]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=8109ada62561

Comment 4

[Mike Hommey [:glandium]]

Comment on attachment 8614853 [details]
MozReview Request: Bug 1139218 - Add an HTTP server for exposing files metadata

https://reviewboard.mozilla.org/r/10087/#review8915

::: python/mozbuild/mozbuild/frontend/files_info_server.py:1
(Diff revision 1)
> +# This Source Code Form is subject to the terms of the Mozilla Public

I don't see the benefit of making this available through a mach command. This could very well be a separate module. The server code also feel weird located in mozbuild.frontend.

::: python/mozbuild/mozbuild/frontend/files_info_server.py:44
(Diff revision 1)
> +    new_mtime = os.path.getmtime(store_path)

Checking directory timestamp is fishy, especially when the .hg/store directory contains things that might have been updated without the repository having changed. I suggest to check .hg/store/00changelog.d. You won't have any relevant changes to the repository without changes to this file.

::: python/mozbuild/mozbuild/frontend/files_info_server.py:118
(Diff revision 1)
> +        a = mozpath.abspath(mozpath.join(repo_path, f))

You might as well abspath-ize the repo paths when you initialize the server. Then use normpath here.

Comment 5

[Dan Minor [:dminor]]

Comment on attachment 8614853 [details]
MozReview Request: Bug 1139218 - Add an HTTP server for exposing files metadata

https://reviewboard.mozilla.org/r/10087/#review8965

::: python/mozbuild/mozbuild/frontend/files_info_server.py:115
(Diff revision 1)
> +            files.add(v)

Maybe just an array here unless you're planning on supporting more than 'f' in the future. Also I think I'd prefer 'file' to 'f'.

::: python/mozbuild/mozbuild/frontend/files_info_server.py:102
(Diff revision 1)
> +def get_file_info(req):

A docstring with example request and reply json would be good here.

::: python/mozbuild/mozbuild/frontend/files_info_server.py:88
(Diff revision 1)
> +def get_repo(req):

A docstring with example request and reply json would be good here.

::: python/mozbuild/mozbuild/frontend/files_info_server.py:77
(Diff revision 1)
> +def get_repos(req):

A docstring with example request and reply json would be good here.

Comment 6

[Gregory Szorc [:gps]]

So, I always wanted to deploy this metadata service directly on hg.mozilla.org (or any Mercurial server for that matter). However, we couldn't do that because hg.mozilla.org was running Python 2.6 and the mozbuild Python package requires Python 2.7. The work around would require standing up a separate service (probably in EC2) and we'd need to establish repo syncing and other foo.

Work is ongoing to switch hg.mozilla.org to run from Python 2.7. So, since that blocker is gone, I think I'll repurpose this bug to integrate the file metadata serving directly from the Mercurial server. This also means that MozReview pushes can get file info in-process without having to speak to a network service. This will make things faster and cut down on points of failure. Wins everywhere.

Comment 7

[Gregory Szorc [:gps]]

Attachment: MozReview Request: hgmo: support querying moz.build file info (bug 1139218); r=smacleod

hgmo: support querying moz.build file info (bug 1139218)

The mozbuild Python package used by the Firefox build system recently
gained the ability to read moz.build files directly from Mercurial
repositories (bug 1168607). This opens the door to all kinds of exciting
possibilities for integrating moz.build info into Mercurial and tools
that are built on top of Mercurial, like code review (MozReview).

In this commit, we implement a generic Python module for extracting
moz.build file info from a Mercurial repository instance. We provide
a CLI command on the hgmo extension that prints obtained moz.build file
info. In future commits, we plan to call this CLI command from the
context of hgweb.

The output from the new command should be going through the Mercurial
templater. However, configuring this is extra work and there are no
users for non-JSON output yet. So, we defer this work until another
time.

Comment 8

[Steven MacLeod [:smacleod]]

Comment on attachment 8621314 [details]
MozReview Request: hgmo: support querying moz.build file info (bug 1139218); r=smacleod

https://reviewboard.mozilla.org/r/10943/#review9857

Looks good to me.

::: test-requirements.txt:24
(Diff revision 1)
> +jsmin==2.1.1

What's with the jsmin requirement? is this something you're just tagging along with this commit?

Comment 9

[Gregory Szorc [:gps]]

Attachment: MozReview Request: hgweb-chroot: generation of chroot environment (bug 1139218); r=smacleod

hgmo: use modern import syntax; r?smacleod

Mercurial has adopted a modern import syntax that is compatible with
Python 3. Switch to it.

Comment 10

[Gregory Szorc [:gps]]

Comment on attachment 8621314 [details]
MozReview Request: hgmo: support querying moz.build file info (bug 1139218); r=smacleod

hgmo: support querying moz.build file info (bug 1139218); r?smacleod

The mozbuild Python package used by the Firefox build system recently
gained the ability to read moz.build files directly from Mercurial
repositories (bug 1168607). This opens the door to all kinds of exciting
possibilities for integrating moz.build info into Mercurial and tools
that are built on top of Mercurial, like code review (MozReview).

In this commit, we implement a generic Python module for extracting
moz.build file info from a Mercurial repository instance. We provide
a CLI command on the hgmo extension that prints obtained moz.build file
info. In future commits, we plan to call this CLI command from the
context of hgweb.

The output from the new command should be going through the Mercurial
templater. However, configuring this is extra work and there are no
users for non-JSON output yet. So, we defer this work until another
time.

Comment 11

[Gregory Szorc [:gps]]

Attachment: MozReview Request: hgmo: web command for retrieving moz.build info; r=smacleod

hgmo: web command for retrieving moz.build info; r?smacleod

We provide a mechanism for exposing moz.build files info via the hgweb
server.

For security reasons that are explained in the added docs, we require a
wrapper command to look up moz.build files info. A subsequent commit
will implement a standalone command for establishing a secure
execution context for hg.mozilla.org servers.

We do not yet handle non-JSON output. Ideally, output should be going
through the templater so we can have HTML output of moz.build info.
But, hg.mozilla.org currently has a quirky relationship with JSON, as
the pushlog-legacy extension hard codes a JSON handler and interferes
with Mercurial's built-in templater. This will all need to be cleaned up
eventually.

Comment 12

[Steven MacLeod [:smacleod]]

https://reviewboard.mozilla.org/r/10943/#review9857

> What's with the jsmin requirement? is this something you're just tagging along with this commit?

Still waiting on a reply to this issue.

Comment 13

[Steven MacLeod [:smacleod]]

Comment on attachment 8634973 [details]
MozReview Request: hgweb-chroot: generation of chroot environment (bug 1139218); r=smacleod

https://reviewboard.mozilla.org/r/12693/#review12157

Ship It!

Comment 14

[Gregory Szorc [:gps]]

https://reviewboard.mozilla.org/r/10943/#review9857

> Still waiting on a reply to this issue.

jsmin is required by mozbuild. The code isn't used, but the import dependency is sneaking in. We could probably fix that in mozbuild if we tried hard enough. But no harm, no foul.

Comment 15

[Gregory Szorc [:gps]]

Better summary :)

kang: heads up, you'll be getting some reviews around chroot/cgroups foo.

Comment 16

[Gregory Szorc [:gps]]

Comment on attachment 8621314 [details]
MozReview Request: hgmo: support querying moz.build file info (bug 1139218); r=smacleod

hgmo: support querying moz.build file info (bug 1139218); r?smacleod

The mozbuild Python package used by the Firefox build system recently
gained the ability to read moz.build files directly from Mercurial
repositories (bug 1168607). This opens the door to all kinds of exciting
possibilities for integrating moz.build info into Mercurial and tools
that are built on top of Mercurial, like code review (MozReview).

In this commit, we implement a generic Python module for extracting
moz.build file info from a Mercurial repository instance. We provide
a CLI command on the hgmo extension that prints obtained moz.build file
info. In future commits, we plan to call this CLI command from the
context of hgweb.

The output from the new command should be going through the Mercurial
templater. However, configuring this is extra work and there are no
users for non-JSON output yet. So, we defer this work until another
time.

Comment 17

[Gregory Szorc [:gps]]

Comment on attachment 8634974 [details]
MozReview Request: hgmo: web command for retrieving moz.build info; r=smacleod

hgmo: web command for retrieving moz.build info; r?smacleod

We provide a mechanism for exposing moz.build files info via the hgweb
server.

For security reasons that are explained in the added docs, we require a
wrapper command to look up moz.build files info. A subsequent commit
will implement a standalone command for establishing a secure
execution context for hg.mozilla.org servers.

We do not yet handle non-JSON output. Ideally, output should be going
through the templater so we can have HTML output of moz.build info.
But, hg.mozilla.org currently has a quirky relationship with JSON, as
the pushlog-legacy extension hard codes a JSON handler and interferes
with Mercurial's built-in templater. This will all need to be cleaned up
eventually.

Comment 18

[Gregory Szorc [:gps]]

Comment on attachment 8634973 [details]
MozReview Request: hgweb-chroot: generation of chroot environment (bug 1139218); r=smacleod

docker: Docker image for hgweb chroot (bug 1139218); r?smacleod, kang

We need to evaluate some possibly untrusted Python code on the hgweb
servers. These servers are running CentOS 6. As much as I'd like to use
LXC, getting it to work on CentOS 6 is more or less an exercise in
futility. CentOS 6 is running Linux 2.6 and many of the nice features of
LXC (such as user namespaces) aren't available until later kernel
versions.

Until we run hg.mozilla.org on a modern OS and kernel, chroots seem to
be the easiest and most practical solution available to us.

This commit introduces a Docker image that is used to build an archive
of a chroot environment suitable for evaluating moz.build files. We
build Python 2.7.10 from source and install our minimum required Python
package dependencies. We manually copy shared libraries needed for
running Python. There are tools like jailkit that make creating
chroot environments very simple. However, the environment is currently
very simple, so it's not too difficult to manage.

Code for fetching the chroot archive has also been added. We'll hook
this up to our deployment infrastructure in a subsequent commit.

Comment 19

[Gregory Szorc [:gps]]

Attachment: MozReview Request: ansible/hg-web: configure cgroups (bug 1139218); r?fubar, kang

ansible/hg-web: configure cgroups (bug 1139218); r?fubar, kang

We will be securing moz.build evaluation on hgweb machines with chroot
and cgroups.

cgroups currently aren't configured on the hgweb machines. This commit
adds the Ansible bits to configure it.

We define a new control group - mozbuild - for evaluating mozbuild
files. We create a new "mozbuild" user and create a control group rule
that all processes for this user should be placed in the "mozbuild"
control group.

Our control group has some basic subsystem adjustments in place. These
are currently incomplete and probably aren't suited for a shared control
group. e.g. I think the memory and CPU limits apply to every process in
the control group. If multiple requests arrive at the same time, it
would be easy to exhaust resources. I'm also not sure 512 MB is enough
memory. Python can be a hog. But, it provides a place to start the
conversation.

Comment 20

[Gregory Szorc [:gps]]

Attachment: MozReview Request: ansible/hg-web: configure moz.build chroot environment (bug 1139218); r=fubar, kang

ansible/hg-web: configure chroot (bug 1139218)

Create a chroot directory on hgweb suitable for executing moz.build
files. We bind mount the repos directory into the chroot. Even though
filesystem permissions are such that the "mozbuild" user has read-only
access, we mount the repositories read-only because defense in depth is
a good practice.

Comment 21

[Gregory Szorc [:gps]]

Attachment: MozReview Request: hgweb-chroot: create a C program to containerize moz.build evaluation (bug 1139218); r=kang

hgweb-chroot: create a C program to enter chroot (bug 1139218); r?kang

The hg.mozilla.org WSGI processes (running as the hg user) will need to
spawn a new, sandboxed process to evaluate moz.build files. This commit
introduces the executable that starts the process.

Implemented is a short C program that essentially sanitizes the
environment and execs a Python process inside a chroot. The program
attempts to ditch references to its executing process/environment,
chroots to our "jail," changes all permissions to the "mozbuild" user,
then executes the appropriate Python moz.build evaluation process
(actually a Mercurial command).

Comment 22

[Gregory Szorc [:gps]]

https://reviewboard.mozilla.org/r/10941/#review12355

This series is very much a work in progress. That being said, I /think/ it is mostly plumbing work that remains. All the basics of using a chroot + cgroups + bind mounted filesystem + wrapper process are there. I "just" need to hook everything up.

Kang: I'm mostly curious for your thoughts on the security side of things. I /think/ this is mostly what was discussed in the email thread. Just want to make sure I'm not missing anything too obvious.

Comment 23

[Kendall Libby [:fubar] (he/him)]

Comment on attachment 8636913 [details]
MozReview Request: ansible/hg-web: configure cgroups (bug 1139218); r?fubar, kang

https://reviewboard.mozilla.org/r/13755/#review12405

looks good to me, though I admit my knowledge of cgroups is limited to this morning's reading. :-) limits look reasonable.

Comment 24

[Gregory Szorc [:gps]]

Comment on attachment 8621314 [details]
MozReview Request: hgmo: support querying moz.build file info (bug 1139218); r=smacleod

hgmo: support querying moz.build file info (bug 1139218); r?smacleod

The mozbuild Python package used by the Firefox build system recently
gained the ability to read moz.build files directly from Mercurial
repositories (bug 1168607). This opens the door to all kinds of exciting
possibilities for integrating moz.build info into Mercurial and tools
that are built on top of Mercurial, like code review (MozReview).

In this commit, we implement a generic Python module for extracting
moz.build file info from a Mercurial repository instance. We provide
a CLI command on the hgmo extension that prints obtained moz.build file
info. In future commits, we plan to call this CLI command from the
context of hgweb.

The output from the new command should be going through the Mercurial
templater. However, configuring this is extra work and there are no
users for non-JSON output yet. So, we defer this work until another
time.

Comment 25

[Gregory Szorc [:gps]]

Comment on attachment 8634974 [details]
MozReview Request: hgmo: web command for retrieving moz.build info; r=smacleod

hgmo: web command for retrieving moz.build info; r?smacleod

We provide a mechanism for exposing moz.build files info via the hgweb
server.

For security reasons that are explained in the added docs, we require a
wrapper command to look up moz.build files info. A subsequent commit
will implement a standalone command for establishing a secure
execution context for hg.mozilla.org servers.

We do not yet handle non-JSON output. Ideally, output should be going
through the templater so we can have HTML output of moz.build info.
But, hg.mozilla.org currently has a quirky relationship with JSON, as
the pushlog-legacy extension hard codes a JSON handler and interferes
with Mercurial's built-in templater. This will all need to be cleaned up
eventually.

Comment 26

[Gregory Szorc [:gps]]

Comment on attachment 8636915 [details]
MozReview Request: hgweb-chroot: create a C program to containerize moz.build evaluation (bug 1139218); r=kang

hgweb-chroot: create a C program to enter chroot (bug 1139218); r?kang

The hg.mozilla.org WSGI processes (running as the hg user) will need to
spawn a new, sandboxed process to evaluate moz.build files. This commit
introduces the executable that starts the process.

Implemented is a short C program that essentially sanitizes the
environment and execs a Python process inside a chroot. The program
attempts to ditch references to its executing process/environment,
switches cgroup groups, chroots to our "jail," changes all permissions
to the "mozbuild" user, then executes the appropriate Python moz.build
evaluation process (actually a Mercurial command).

Comment 27

[Gregory Szorc [:gps]]

Comment on attachment 8634973 [details]
MozReview Request: hgweb-chroot: generation of chroot environment (bug 1139218); r=smacleod

hgweb-chroot: generation of chroot environment (bug 1139218); r?smacleod, kang

We need to evaluate some possibly untrusted Python code on the hgweb
servers. These servers are running CentOS 6. As much as I'd like to use
LXC, getting it to work on CentOS 6 is more or less an exercise in
futility. CentOS 6 is running Linux 2.6 and many of the nice features of
LXC (such as user namespaces) aren't available until later kernel
versions.

Until we run hg.mozilla.org on a modern OS and kernel, chroots seem to
be the easiest and most practical solution available to us.

This commit introduces a Docker image that is used to build files that
will be used for a chroot environment suitable for evaluating moz.build
files. We build Python 2.7.10 from source and install our minimum required
Python package dependencies. We manually copy shared libraries needed for
running Python. There are tools like jailkit that make creating
chroot environments very simple. However, the environment is currently
very simple, so it's not too difficult to manage, so I didn't bother
with the added complexity.

A mach command for obtaining files from the Docker image has been added.
We'll hook up these files to our deployment infrastructure in a
subsequent commit.

Comment 28

[Gregory Szorc [:gps]]

Comment on attachment 8636914 [details]
MozReview Request: ansible/hg-web: configure moz.build chroot environment (bug 1139218); r=fubar, kang

ansible/hg-web: configure moz.build chroot environment (bug 1139218); r?fubar, kang

We will be securing moz.build evaluation on hgweb machines with a chroot
and cgroups.

cgroups currently aren't configured on the hgweb machines. This commit
adds the Ansible bits to configure it.

We define a new control group - mozbuild - for evaluating mozbuild
files. We create a new "mozbuild" user and create a control group rule
that all processes for this user should be placed in the "mozbuild"
control group.

The "mozbuild" control group has some basic subsystem adjustments in
place to limit resource usage and device access.

Ansible has been updated to deploy our chroot files from the local
machine. It generates the current versions of the files and copies
them to the remote server.

We set up a bind mount of the repositories directory inside the chroot.
While the "mozbuild" user doesn't have permissions to modify anything
inside /repo/hg/mozilla (the entire tree is owned by hg:hg with
privileges 4775 on directories and 0664 on files), we add an extra layer
of caution by making the bind mount read-only to the chroot.

Open Issues:

1. How do I verify that mozbuild-eval is actually being placed in the
   "mozbuild" control group?
2. Still need to lock down access to the network.
3. The settings on the control group subsystems should be heavily
   scrutinized.
4. Does it make sense to share limits across processes? e.g. if multiple
   HTTP requests come in at one time, multiple processes could exhaust
   process limits. I'm not sure how else to do this aside from possibly
   having the C program clone control groups and create a one-off
   control group for every process. Seems complex.

Comment 29

[Gregory Szorc [:gps]]

https://reviewboard.mozilla.org/r/10941/#review12473

OK. This is mostly in a happy state. I've deployed everything to hgweb1.dmz.scl3.mozilla.com (it's been out of the load balancer for a few weeks since we were planning to upgrade it to CentOS 7). I can confirm HTTP requests are spawning /usr/local/bin/mozbuild-eval and JSON is being generated. Pretty awesome to see working!

What I don't know is whether the C program and cgroups foo is as secure as it needs to be. I'm not an expert here and have been learning this stuff from documentation online. I'd *really* appreciate a thorough security review here.

I feel really bad that we don't have tests. But running them inside Docker will be a lot of work. Still, I think having automated tests that verify at least the cgroup policies are working would be a good idea. e.g. if the devices limitation isn't in place, an attacker could almost certainly escape from the chroot.

Comment 30

[Gregory Szorc [:gps]]

https://reviewboard.mozilla.org/r/13759/#review12475

I also wonder if we should be using clone() with some explicit flags like CLONE_NEWPID, CLONE_NEWIPC, and CLONE_NEWNS to create better process isolation.

::: testing/docker/builder-hgweb-chroot/mozbuild-eval.c:97
(Diff revision 2)
> +

Do we need a setgroups() here?

Comment 31

[Gregory Szorc [:gps]]

https://reviewboard.mozilla.org/r/13757/#review12479

::: ansible/roles/hg-web/tasks/main.yml:145
(Diff revision 2)
> +- name: create mozbuild user
> +  user: name=mozbuild

puppet seems to be undoing this :/

Comment 32

[Gregory Szorc [:gps]]

https://reviewboard.mozilla.org/r/13759/#review12483

::: testing/docker/builder-hgweb-chroot/mozbuild-eval.c:54
(Diff revision 2)
> +    cg = cgroup_new_cgroup("mozbuild");
> +    if (!cg) {
> +        printf("unable to obtain mozbuild cgroup\n");
> +        return 1;
> +    }
> +    if (cgroup_attach_task(cg)) {

Reading the libcgroup source and looking at strace output, I'm pretty sure we need to call cgroup_get_cgroup() here to populate the list of controllers for the cgroup. Otherwise, the internal list is empty and cgroup_attach_task() effectively no-ops. Classic poor documentation right there.

Comment 33

[Gregory Szorc [:gps]]

Comment on attachment 8636915 [details]
MozReview Request: hgweb-chroot: create a C program to containerize moz.build evaluation (bug 1139218); r=kang

hgweb-chroot: create a C program to enter chroot (bug 1139218); r?kang

The hg.mozilla.org WSGI processes (running as the hg user) will need to
spawn a new, sandboxed process to evaluate moz.build files. This commit
introduces the executable that starts the process.

Implemented is a short C program that essentially sanitizes the
environment and execs a Python process inside a chroot. The program
attempts to ditch references to its executing process/environment,
switches cgroup groups, chroots to our "jail," changes all permissions
to the "mozbuild" user, then executes the appropriate Python moz.build
evaluation process (actually a Mercurial command).

Comment 34

[Gregory Szorc [:gps]]

Comment on attachment 8634973 [details]
MozReview Request: hgweb-chroot: generation of chroot environment (bug 1139218); r=smacleod

hgweb-chroot: generation of chroot environment (bug 1139218); r?smacleod, kang

We need to evaluate some possibly untrusted Python code on the hgweb
servers. These servers are running CentOS 6. As much as I'd like to use
LXC, getting it to work on CentOS 6 is more or less an exercise in
futility. CentOS 6 is running Linux 2.6 and many of the nice features of
LXC (such as user namespaces) aren't available until later kernel
versions.

Until we run hg.mozilla.org on a modern OS and kernel, chroots seem to
be the easiest and most practical solution available to us.

This commit introduces a Docker image that is used to build files that
will be used for a chroot environment suitable for evaluating moz.build
files. We build Python 2.7.10 from source and install our minimum required
Python package dependencies. We manually copy shared libraries needed for
running Python. There are tools like jailkit that make creating
chroot environments very simple. However, the environment is currently
very simple, so it's not too difficult to manage, so I didn't bother
with the added complexity.

A mach command for obtaining files from the Docker image has been added.
We'll hook up these files to our deployment infrastructure in a
subsequent commit.

Comment 35

[Gregory Szorc [:gps]]

Comment on attachment 8636914 [details]
MozReview Request: ansible/hg-web: configure moz.build chroot environment (bug 1139218); r=fubar, kang

ansible/hg-web: configure moz.build chroot environment (bug 1139218); r?fubar, kang

We will be securing moz.build evaluation on hgweb machines with a chroot
and cgroups.

cgroups currently aren't configured on the hgweb machines. This commit
adds the Ansible bits to configure it.

We define a new control group - mozbuild - for evaluating mozbuild
files. We create a new "mozbuild" user and create a control group rule
that all processes for this user should be placed in the "mozbuild"
control group.

The "mozbuild" control group has some basic subsystem adjustments in
place to limit resource usage and device access.

Ansible has been updated to deploy our chroot files from the local
machine. It generates the current versions of the files and copies
them to the remote server.

We set up a bind mount of the repositories directory inside the chroot.
While the "mozbuild" user doesn't have permissions to modify anything
inside /repo/hg/mozilla (the entire tree is owned by hg:hg with
privileges 4775 on directories and 0664 on files), we add an extra layer
of caution by making the bind mount read-only to the chroot.

Open Issues:

1. How do I verify that mozbuild-eval is actually being placed in the
   "mozbuild" control group?
2. Still need to lock down access to the network.
3. The settings on the control group subsystems should be heavily
   scrutinized.
4. Does it make sense to share limits across processes? e.g. if multiple
   HTTP requests come in at one time, multiple processes could exhaust
   process limits. I'm not sure how else to do this aside from possibly
   having the C program clone control groups and create a one-off
   control group for every process. Seems complex.

Comment 36

[Gregory Szorc [:gps]]

https://reviewboard.mozilla.org/r/13759/#review12483

> Reading the libcgroup source and looking at strace output, I'm pretty sure we need to call cgroup_get_cgroup() here to populate the list of controllers for the cgroup. Otherwise, the internal list is empty and cgroup_attach_task() effectively no-ops. Classic poor documentation right there.

I added a call to cgroup_get_cgroup() and strace reveals we're actually writing out the tasks file for each controller in the mozbuild control group now. It must be tradition for security-sensitive code to have a poor API and/or be under-documented. So sad.

Comment 37

[Kendall Libby [:fubar] (he/him)]

Comment on attachment 8636914 [details]
MozReview Request: ansible/hg-web: configure moz.build chroot environment (bug 1139218); r=fubar, kang

https://reviewboard.mozilla.org/r/13757/#review12515

> 1. How do I verify that mozbuild-eval is actually being placed in the "mozbuild" control group?

In general? Check for PID in /cgroup/*/mozbuild/tasks ?

> 2. Still need to lock down access to the network.

It looks like access can't be prevented, only prioritized or classified. 

> 4. Does it make sense to share limits across processes?

I'm not familiar enough with what moz.build is used for/by to evaluate how big of a risk this is. how long does each process last, and how much of each resource does it actually use? the cpu and mem resources both have a bunch of reporting options; worth gathering data and adjusting/re-evaluating as we go?

::: ansible/roles/hg-web/files/cgconfig-mozbuild.conf:33
(Diff revision 3)
> +    devices {

RHEL--

"The Device Whitelist (devices) subsystem is considered to be a Technology Preview in Red Hat Enterprise Linux 6.
Technology preview features are currently not supported under Red Hat Enterprise Linux 6 subscription services, might not be functionally complete, and are generally not suitable for production use."

If write access MUST be disallowed, we should verify that it actually works as described.

::: ansible/roles/hg-web/files/hgrc:45
(Diff revision 3)
> +mozbuildinfowrapper = /usr/local/bin/mozbuild-eval %repo%

Suggest using cgexec on the wrapper to explicitly and immediately place the process into the cgroup.

From RHEL's documentation:
"Note, however, that the daemon moves the process to the cgroup only after the appropriate condition is fulfilled. Therefore, the [ftp] process might run for a short time in the wrong group. Furthermore, if the process quickly spawns children while in the wrong group, these children might not be moved." (https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/sec-Moving_a_Process_to_a_Control_Group.html#The_cgred_Service)

::: ansible/roles/hg-web/files/cgconfig-mozbuild.conf:35
(Diff revision 3)
> +        devices.deny = "a";

devices.deny uses the same syntax as .allow, so I think this should actually be

devices.deny = "a *:* rwm";

to disallow read/write/create on any major:minor device, of any type (block or char).

No other concerns.

Comment 38

[Steven MacLeod [:smacleod]]

Comment on attachment 8621314 [details]
MozReview Request: hgmo: support querying moz.build file info (bug 1139218); r=smacleod

https://reviewboard.mozilla.org/r/10943/#review12539

::: hgext/hgmo/tests/test-mozbuildinfo.t:32
(Diff revision 4)
> +      "bug_component_counts": [], 

what's with the trailing whitespace on a bunch of the json output?

Comment 39

[Gregory Szorc [:gps]]

https://reviewboard.mozilla.org/r/13757/#review12515

The processes are typically <1s in length. It's pretty difficult to verify the tasks file has the pid contents you are looking for. However, I did strace the C wrapper and verified the PID is being written into the tasks file. So it /should/ be working.

Could we add a firewall rule to drop packets having a certain classification?

Comment 40

[Kendall Libby [:fubar] (he/him)]

looks like you can. some parts of the internet say 'iptables -m cgroup' but a quick man page scan failed. alternatively, use tc (traffic control) and, optionally, iptables.

Comment 41

[Guillaume Destuynder [:kang] [this account is no longer active]]

Comment on attachment 8636915 [details]
MozReview Request: hgweb-chroot: create a C program to containerize moz.build evaluation (bug 1139218); r=kang

https://reviewboard.mozilla.org/r/13759/#review12591

As per bugzilla comment 30 - using CLONE_NEWXX would be a good idea if supported by the host system.
That would be using the unshare() call instead of clone(). (http://man7.org/linux/man-pages/man2/unshare.2.html - see also the unshare and nsenter programs from the utils-linux package for examples)

I would suggest CLONE_NEWNS|CLONE_NEWIPC|CLONE_NEWPID|CLONE_NEWUTS for ex. Keep in mind that CLONE_NEWPID will most likely require you to fork() before exec'ing or new commands will fail to run.
The cgroup code seem ok - albeit this is more about resource usage control than a strong security control (same for chroot, as root could rechroot outside for ex.).

Finally, after exec any fd not marked close-on-exec will remain open in the new process, this can be fixed by forcefully closing them, or verifying they're all close-on-exec (they should be in this case the program is pretty small),  or simply unsharing with CLONE_FILES.

Note that this is effectively creating what people generally call a "container", and you can select what you really want/need to unshare through these CLONE flags (but you most likely want CLONE_FS or/and make sure mounts are unshared when using CLONE_NEWNS since you're already chroot-ing anyway).

Comment 42

[Guillaume Destuynder [:kang] [this account is no longer active]]

Comment on attachment 8634973 [details]
MozReview Request: hgweb-chroot: generation of chroot environment (bug 1139218); r=smacleod

https://reviewboard.mozilla.org/r/12693/#review12593

::: testing/docker/builder-hgweb-chroot/Dockerfile:7
(Diff revision 4)
> +FROM centos:centos6

Is the image download also verified? (in particular I dont know if the version of docker used as the problems hilighted here: https://titanous.com/posts/docker-insecurity fixed/worked around).
Most teams "fix that" by adding an additional checksum around the docker pull (generated from a manually checked/known good image) - basically, doing it like it was done for Python-2.7 here would be fine.

Comment 43

[Guillaume Destuynder [:kang] [this account is no longer active]]

Comment on attachment 8636914 [details]
MozReview Request: ansible/hg-web: configure moz.build chroot environment (bug 1139218); r=fubar, kang

https://reviewboard.mozilla.org/r/13757/#review12597

::: ansible/roles/hg-web/tasks/main.yml:197
(Diff revision 3)
> +        mode=4755

This seems ok but I'll still note that this is generally frowned upon vs "sudo ./chrootprogram" as the later requires authentication, while the former gives you root-rights for the chroot app, on demand.
its therefore important to ensure the app exec'd by the chroot app cannot be modified

Comment 44

[Steven MacLeod [:smacleod]]

Comment on attachment 8634974 [details]
MozReview Request: hgmo: web command for retrieving moz.build info; r=smacleod

https://reviewboard.mozilla.org/r/13491/#review12663

LGTM!

::: hgext/hgmo/tests/test-mozbuildinfo-webcommand.t:116
(Diff revision 3)
> +Errors displayed properly

Please throw in a simple test which uses a bogus wrapper command to test the non-json output returned error handling.

Comment 45

[Steven MacLeod [:smacleod]]

Comment on attachment 8634973 [details]
MozReview Request: hgweb-chroot: generation of chroot environment (bug 1139218); r=smacleod

https://reviewboard.mozilla.org/r/12693/#review12665

Once :kang's concern is addressed this LGTM.

Comment 46

[Gregory Szorc [:gps]]

https://reviewboard.mozilla.org/r/10943/#review12539

> what's with the trailing whitespace on a bunch of the json output?

Blame Python's json serializer :/

Comment 47

[Gregory Szorc [:gps]]

Comment on attachment 8621314 [details]
MozReview Request: hgmo: support querying moz.build file info (bug 1139218); r=smacleod

hgmo: support querying moz.build file info (bug 1139218); r=smacleod

The mozbuild Python package used by the Firefox build system recently
gained the ability to read moz.build files directly from Mercurial
repositories (bug 1168607). This opens the door to all kinds of exciting
possibilities for integrating moz.build info into Mercurial and tools
that are built on top of Mercurial, like code review (MozReview).

In this commit, we implement a generic Python module for extracting
moz.build file info from a Mercurial repository instance. We provide
a CLI command on the hgmo extension that prints obtained moz.build file
info. In future commits, we plan to call this CLI command from the
context of hgweb.

The output from the new command should be going through the Mercurial
templater. However, configuring this is extra work and there are no
users for non-JSON output yet. So, we defer this work until another
time.

Comment 48

[Gregory Szorc [:gps]]

Comment on attachment 8634974 [details]
MozReview Request: hgmo: web command for retrieving moz.build info; r=smacleod

hgmo: web command for retrieving moz.build info; r?smacleod

We provide a mechanism for exposing moz.build files info via the hgweb
server.

For security reasons that are explained in the added docs, we require a
wrapper command to look up moz.build files info. A subsequent commit
will implement a standalone command for establishing a secure
execution context for hg.mozilla.org servers.

We do not yet handle non-JSON output. Ideally, output should be going
through the templater so we can have HTML output of moz.build info.
But, hg.mozilla.org currently has a quirky relationship with JSON, as
the pushlog-legacy extension hard codes a JSON handler and interferes
with Mercurial's built-in templater. This will all need to be cleaned up
eventually.

Comment 49

[Gregory Szorc [:gps]]

Comment on attachment 8636915 [details]
MozReview Request: hgweb-chroot: create a C program to containerize moz.build evaluation (bug 1139218); r=kang

hgweb-chroot: create a C program to enter chroot (bug 1139218); r?kang

The hg.mozilla.org WSGI processes (running as the hg user) will need to
spawn a new, sandboxed process to evaluate moz.build files. This commit
introduces the executable that starts the process.

Implemented is a short C program that essentially sanitizes the
environment and execs a Python process inside a chroot. The program
attempts to ditch references to its executing process/environment,
switches cgroup groups, chroots to our "jail," changes all permissions
to the "mozbuild" user, then executes the appropriate Python moz.build
evaluation process (actually a Mercurial command).

Comment 50

[Gregory Szorc [:gps]]

Comment on attachment 8634973 [details]
MozReview Request: hgweb-chroot: generation of chroot environment (bug 1139218); r=smacleod

hgweb-chroot: generation of chroot environment (bug 1139218); r?smacleod, kang

We need to evaluate some possibly untrusted Python code on the hgweb
servers. These servers are running CentOS 6. As much as I'd like to use
LXC, getting it to work on CentOS 6 is more or less an exercise in
futility. CentOS 6 is running Linux 2.6 and many of the nice features of
LXC (such as user namespaces) aren't available until later kernel
versions.

Until we run hg.mozilla.org on a modern OS and kernel, chroots seem to
be the easiest and most practical solution available to us.

This commit introduces a Docker image that is used to build files that
will be used for a chroot environment suitable for evaluating moz.build
files. We build Python 2.7.10 from source and install our minimum required
Python package dependencies. We manually copy shared libraries needed for
running Python. There are tools like jailkit that make creating
chroot environments very simple. However, the environment is currently
very simple, so it's not too difficult to manage, so I didn't bother
with the added complexity.

A mach command for obtaining files from the Docker image has been added.
We'll hook up these files to our deployment infrastructure in a
subsequent commit.

Comment 51

[Gregory Szorc [:gps]]

Comment on attachment 8636914 [details]
MozReview Request: ansible/hg-web: configure moz.build chroot environment (bug 1139218); r=fubar, kang

ansible/hg-web: configure moz.build chroot environment (bug 1139218); r?fubar, kang

We will be securing moz.build evaluation on hgweb machines with a chroot
and cgroups.

cgroups currently aren't configured on the hgweb machines. This commit
adds the Ansible bits to configure it.

We define a new control group - mozbuild - for evaluating mozbuild
files. We create a new "mozbuild" user and create a control group rule
that all processes for this user should be placed in the "mozbuild"
control group.

The "mozbuild" control group has some basic subsystem adjustments in
place to limit resource usage and device access.

Ansible has been updated to deploy our chroot files from the local
machine. It generates the current versions of the files and copies
them to the remote server.

We set up a bind mount of the repositories directory inside the chroot.
While the "mozbuild" user doesn't have permissions to modify anything
inside /repo/hg/mozilla (the entire tree is owned by hg:hg with
privileges 4775 on directories and 0664 on files), we add an extra layer
of caution by making the bind mount read-only to the chroot.

Open Issues:

1. How do I verify that mozbuild-eval is actually being placed in the
   "mozbuild" control group?
2. Still need to lock down access to the network.
3. The settings on the control group subsystems should be heavily
   scrutinized.
4. Does it make sense to share limits across processes? e.g. if multiple
   HTTP requests come in at one time, multiple processes could exhaust
   process limits. I'm not sure how else to do this aside from possibly
   having the C program clone control groups and create a one-off
   control group for every process. Seems complex.

Comment 52

[Gregory Szorc [:gps]]

Comment on attachment 8636915 [details]
MozReview Request: hgweb-chroot: create a C program to containerize moz.build evaluation (bug 1139218); r=kang

hgweb-chroot: create a C program to enter chroot (bug 1139218); r?kang

The hg.mozilla.org WSGI processes (running as the hg user) will need to
spawn a new, sandboxed process to evaluate moz.build files. This commit
introduces the executable that starts the process.

Implemented is a short C program that essentially sanitizes the
environment and execs a Python process inside a chroot. The program
attempts to ditch references to its executing process/environment,
switches cgroup groups, chroots to our "jail," changes all permissions
to the "mozbuild" user, then executes the appropriate Python moz.build
evaluation process (actually a Mercurial command).

Comment 53

[Gregory Szorc [:gps]]

Comment on attachment 8634973 [details]
MozReview Request: hgweb-chroot: generation of chroot environment (bug 1139218); r=smacleod

hgweb-chroot: generation of chroot environment (bug 1139218); r?smacleod, kang

We need to evaluate some possibly untrusted Python code on the hgweb
servers. These servers are running CentOS 6. As much as I'd like to use
LXC, getting it to work on CentOS 6 is more or less an exercise in
futility. CentOS 6 is running Linux 2.6 and many of the nice features of
LXC (such as user namespaces) aren't available until later kernel
versions.

Until we run hg.mozilla.org on a modern OS and kernel, chroots seem to
be the easiest and most practical solution available to us.

This commit introduces a Docker image that is used to build files that
will be used for a chroot environment suitable for evaluating moz.build
files. We build Python 2.7.10 from source and install our minimum required
Python package dependencies. We manually copy shared libraries needed for
running Python. There are tools like jailkit that make creating
chroot environments very simple. However, the environment is currently
very simple, so it's not too difficult to manage, so I didn't bother
with the added complexity.

A mach command for obtaining files from the Docker image has been added.
We'll hook up these files to our deployment infrastructure in a
subsequent commit.

Comment 54

[Gregory Szorc [:gps]]

Comment on attachment 8636914 [details]
MozReview Request: ansible/hg-web: configure moz.build chroot environment (bug 1139218); r=fubar, kang

ansible/hg-web: configure moz.build chroot environment (bug 1139218); r?fubar, kang

We will be securing moz.build evaluation on hgweb machines with a chroot
and cgroups.

cgroups currently aren't configured on the hgweb machines. This commit
adds the Ansible bits to configure it.

We define a new control group - mozbuild - for evaluating mozbuild
files. We create a new "mozbuild" user and create a control group rule
that all processes for this user should be placed in the "mozbuild"
control group.

The "mozbuild" control group has some basic subsystem adjustments in
place to limit resource usage and device access.

Ansible has been updated to deploy our chroot files from the local
machine. It generates the current versions of the files and copies
them to the remote server.

We set up a bind mount of the repositories directory inside the chroot.
While the "mozbuild" user doesn't have permissions to modify anything
inside /repo/hg/mozilla (the entire tree is owned by hg:hg with
privileges 4775 on directories and 0664 on files), we add an extra layer
of caution by making the bind mount read-only to the chroot.

Open Issues:

1. How do I verify that mozbuild-eval is actually being placed in the
   "mozbuild" control group?
2. Still need to lock down access to the network.
3. The settings on the control group subsystems should be heavily
   scrutinized.
4. Does it make sense to share limits across processes? e.g. if multiple
   HTTP requests come in at one time, multiple processes could exhaust
   process limits. I'm not sure how else to do this aside from possibly
   having the C program clone control groups and create a one-off
   control group for every process. Seems complex.

Comment 55

[Gregory Szorc [:gps]]

https://reviewboard.mozilla.org/r/12693/#review12593

> Is the image download also verified? (in particular I dont know if the version of docker used as the problems hilighted here: https://titanous.com/posts/docker-insecurity fixed/worked around).
> Most teams "fix that" by adding an additional checksum around the docker pull (generated from a manually checked/known good image) - basically, doing it like it was done for Python-2.7 here would be fine.

I don't think Docker ever properly addressed integrity of `docker pull`. AFAIK the signatures they have implemented only verify the image wasn't tampered from what the server thinks it should be. There is no obvious way to import e.g. signing keys into Docker, so there is no end-to-end verification of image contents. I'll add some explicit checking.

Comment 56

[Guillaume Destuynder [:kang] [this account is no longer active]]

Comment on attachment 8636915 [details]
MozReview Request: hgweb-chroot: create a C program to containerize moz.build evaluation (bug 1139218); r=kang

https://reviewboard.mozilla.org/r/13759/#review12837

Ship It!

Comment 57

[Gregory Szorc [:gps]]

Comment on attachment 8621314 [details]
MozReview Request: hgmo: support querying moz.build file info (bug 1139218); r=smacleod

hgmo: support querying moz.build file info (bug 1139218); r=smacleod

The mozbuild Python package used by the Firefox build system recently
gained the ability to read moz.build files directly from Mercurial
repositories (bug 1168607). This opens the door to all kinds of exciting
possibilities for integrating moz.build info into Mercurial and tools
that are built on top of Mercurial, like code review (MozReview).

In this commit, we implement a generic Python module for extracting
moz.build file info from a Mercurial repository instance. We provide
a CLI command on the hgmo extension that prints obtained moz.build file
info. In future commits, we plan to call this CLI command from the
context of hgweb.

The output from the new command should be going through the Mercurial
templater. However, configuring this is extra work and there are no
users for non-JSON output yet. So, we defer this work until another
time.

Comment 58

[Gregory Szorc [:gps]]

Comment on attachment 8634974 [details]
MozReview Request: hgmo: web command for retrieving moz.build info; r=smacleod

hgmo: web command for retrieving moz.build info; r=smacleod

We provide a mechanism for exposing moz.build files info via the hgweb
server.

For security reasons that are explained in the added docs, we require a
wrapper command to look up moz.build files info. A subsequent commit
will implement a standalone command for establishing a secure
execution context for hg.mozilla.org servers.

We do not yet handle non-JSON output. Ideally, output should be going
through the templater so we can have HTML output of moz.build info.
But, hg.mozilla.org currently has a quirky relationship with JSON, as
the pushlog-legacy extension hard codes a JSON handler and interferes
with Mercurial's built-in templater. This will all need to be cleaned up
eventually.

Comment 59

[Gregory Szorc [:gps]]

Comment on attachment 8636915 [details]
MozReview Request: hgweb-chroot: create a C program to containerize moz.build evaluation (bug 1139218); r=kang

hgweb-chroot: create a C program to containerize moz.build evaluation (bug 1139218); r=kang

The hg.mozilla.org WSGI processes (running as the hg user) will need to
spawn a new, sandboxed process to evaluate moz.build files (read
potentially untrusted Python code). This commit introduces the executable
that starts the process.

Implemented is a C program that creates a specialized, ultra low
privileged container for moz.build evaluation. I found
http://crosbymichael.com/creating-containers-part-1.html and
http://www.unixwiz.net/techtips/chroot-practices.html invaluable when
implementing this.

The program is targeted for CentOS 6. CLONE_NEWUSER isn't exposed to our
servers, so we don't need to worry about that and the privilege
escalation bugs that come with it.

Comment 60

[Gregory Szorc [:gps]]

Comment on attachment 8634973 [details]
MozReview Request: hgweb-chroot: generation of chroot environment (bug 1139218); r=smacleod

hgweb-chroot: generation of chroot environment (bug 1139218); r=smacleod

We need to evaluate some possibly untrusted Python code on the hgweb
servers. These servers are running CentOS 6. As much as I'd like to use
LXC, getting it to work on CentOS 6 is more or less an exercise in
futility. CentOS 6 is running Linux 2.6 and many of the nice features of
LXC (such as user namespaces) aren't available until later kernel
versions.

Until we run hg.mozilla.org on a modern OS and kernel, chroots seem to
be the easiest and most practical solution available to us.

This commit introduces a Docker image that is used to build files that
will be used for a chroot environment suitable for evaluating moz.build
files. We build Python 2.7.10 from source and install our minimum required
Python package dependencies. We manually copy shared libraries needed for
running Python. There are tools like jailkit that make creating
chroot environments very simple. However, the environment is currently
very simple, so it's not too difficult to manage, so I didn't bother
with the added complexity.

A mach command for obtaining files from the Docker image has been added.
We'll hook up these files to our deployment infrastructure in a
subsequent commit.

Comment 61

[Gregory Szorc [:gps]]

Comment on attachment 8636914 [details]
MozReview Request: ansible/hg-web: configure moz.build chroot environment (bug 1139218); r=fubar, kang

ansible/hg-web: configure moz.build chroot environment (bug 1139218); r?fubar, kang

We will be securing moz.build evaluation on hgweb machines with a chroot
and cgroups.

cgroups currently aren't configured on the hgweb machines. This commit
adds the Ansible bits to configure it.

We define a new control group - mozbuild - for evaluating mozbuild
files. We create a new "mozbuild" user and create a control group rule
that all processes for this user should be placed in the "mozbuild"
control group.

The "mozbuild" control group has some basic subsystem adjustments in
place to limit resource usage and device access.

Ansible has been updated to deploy our chroot files from the local
machine. It generates the current versions of the files and copies
them to the remote server.

We set up a bind mount of the repositories directory inside the chroot.
While the "mozbuild" user doesn't have permissions to modify anything
inside /repo/hg/mozilla (the entire tree is owned by hg:hg with
privileges 4775 on directories and 0664 on files), we add an extra layer
of caution by making the bind mount read-only to the chroot.

Open Issues:

1. How do I verify that mozbuild-eval is actually being placed in the
   "mozbuild" control group?
2. Still need to lock down access to the network.
3. The settings on the control group subsystems should be heavily
   scrutinized.
4. Does it make sense to share limits across processes? e.g. if multiple
   HTTP requests come in at one time, multiple processes could exhaust
   process limits. I'm not sure how else to do this aside from possibly
   having the C program clone control groups and create a one-off
   control group for every process. Seems complex.

Comment 62

[Gregory Szorc [:gps]]

https://reviewboard.mozilla.org/r/13757/#review12515

> RHEL--
> 
> "The Device Whitelist (devices) subsystem is considered to be a Technology Preview in Red Hat Enterprise Linux 6.
> Technology preview features are currently not supported under Red Hat Enterprise Linux 6 subscription services, might not be functionally complete, and are generally not suitable for production use."
> 
> If write access MUST be disallowed, we should verify that it actually works as described.

I did the following to seemingly verify that this works as expected:

1. copy `bash`, `mknod`, and `cat` into the chroot
2. chroot into a bash process
3. From outside the chroot, add bash's PID to /cgroup/devices/mozbuild/tasks
4. From inside the chroot `mknod dev0 c 1 8` (/dev/random memory device)

This yielded:

  $ ./mknod: `dev0': Operation not permitted
  
When I moved the bash PID out of the mozbuild control group, mknod works again.

I verified that `cat dev0` yielded output from inside the chroot when bash's PID was set to the default control group. When I moved the PID to the mozbuild control group, we get:

  $ /cat dev0
  ./cat: dev0: Operation not permitted

So, while the docs say it is a Technology Preview, it certainly appears to work. At least for the global device blocking that we want. For the record, I did try a hodgepodge of other devices, including some block devices, to make sure c 1 8 wasn't special.

We don't expose any devices to the chroot other than urandom. And with `mknod(2)` apparently not permitted, no references to devices can be obtained, so additional permissions on devices are irrelevant. But if a handle could be obtained, it appears the control group policy is preventing access. Nice.

Comment 63

[Kendall Libby [:fubar] (he/him)]

Comment on attachment 8636914 [details]
MozReview Request: ansible/hg-web: configure moz.build chroot environment (bug 1139218); r=fubar, kang

https://reviewboard.mozilla.org/r/13757/#review12955

Ship It!

Comment 64

[Guillaume Destuynder [:kang] [this account is no longer active]]

Comment on attachment 8636914 [details]
MozReview Request: ansible/hg-web: configure moz.build chroot environment (bug 1139218); r=fubar, kang

https://reviewboard.mozilla.org/r/13757/#review12967

Ship It!

Comment 65

[Gregory Szorc [:gps]]

Comment on attachment 8621314 [details]
MozReview Request: hgmo: support querying moz.build file info (bug 1139218); r=smacleod

hgmo: support querying moz.build file info (bug 1139218); r=smacleod

The mozbuild Python package used by the Firefox build system recently
gained the ability to read moz.build files directly from Mercurial
repositories (bug 1168607). This opens the door to all kinds of exciting
possibilities for integrating moz.build info into Mercurial and tools
that are built on top of Mercurial, like code review (MozReview).

In this commit, we implement a generic Python module for extracting
moz.build file info from a Mercurial repository instance. We provide
a CLI command on the hgmo extension that prints obtained moz.build file
info. In future commits, we plan to call this CLI command from the
context of hgweb.

The output from the new command should be going through the Mercurial
templater. However, configuring this is extra work and there are no
users for non-JSON output yet. So, we defer this work until another
time.

Comment 66

[Gregory Szorc [:gps]]

Comment on attachment 8634974 [details]
MozReview Request: hgmo: web command for retrieving moz.build info; r=smacleod

hgmo: web command for retrieving moz.build info; r=smacleod

We provide a mechanism for exposing moz.build files info via the hgweb
server.

For security reasons that are explained in the added docs, we require a
wrapper command to look up moz.build files info. A subsequent commit
will implement a standalone command for establishing a secure
execution context for hg.mozilla.org servers.

We do not yet handle non-JSON output. Ideally, output should be going
through the templater so we can have HTML output of moz.build info.
But, hg.mozilla.org currently has a quirky relationship with JSON, as
the pushlog-legacy extension hard codes a JSON handler and interferes
with Mercurial's built-in templater. This will all need to be cleaned up
eventually.

Comment 67

[Gregory Szorc [:gps]]

Comment on attachment 8636915 [details]
MozReview Request: hgweb-chroot: create a C program to containerize moz.build evaluation (bug 1139218); r=kang

hgweb-chroot: create a C program to containerize moz.build evaluation (bug 1139218); r=kang

The hg.mozilla.org WSGI processes (running as the hg user) will need to
spawn a new, sandboxed process to evaluate moz.build files (read
potentially untrusted Python code). This commit introduces the executable
that starts the process.

Implemented is a C program that creates a specialized, ultra low
privileged container for moz.build evaluation. I found
http://crosbymichael.com/creating-containers-part-1.html and
http://www.unixwiz.net/techtips/chroot-practices.html invaluable when
implementing this.

The program is targeted for CentOS 6. CLONE_NEWUSER isn't exposed to our
servers, so we don't need to worry about that and the privilege
escalation bugs that come with it.

Comment 68

[Gregory Szorc [:gps]]

Comment on attachment 8634973 [details]
MozReview Request: hgweb-chroot: generation of chroot environment (bug 1139218); r=smacleod

hgweb-chroot: generation of chroot environment (bug 1139218); r=smacleod

We need to evaluate some possibly untrusted Python code on the hgweb
servers. These servers are running CentOS 6. As much as I'd like to use
LXC, getting it to work on CentOS 6 is more or less an exercise in
futility. CentOS 6 is running Linux 2.6 and many of the nice features of
LXC (such as user namespaces) aren't available until later kernel
versions.

Until we run hg.mozilla.org on a modern OS and kernel, chroots seem to
be the easiest and most practical solution available to us.

This commit introduces a Docker image that is used to build files that
will be used for a chroot environment suitable for evaluating moz.build
files. We build Python 2.7.10 from source and install our minimum required
Python package dependencies. We manually copy shared libraries needed for
running Python. There are tools like jailkit that make creating
chroot environments very simple. However, the environment is currently
very simple, so it's not too difficult to manage, so I didn't bother
with the added complexity.

A mach command for obtaining files from the Docker image has been added.
We'll hook up these files to our deployment infrastructure in a
subsequent commit.

Comment 69

[Gregory Szorc [:gps]]

Comment on attachment 8636914 [details]
MozReview Request: ansible/hg-web: configure moz.build chroot environment (bug 1139218); r=fubar, kang

ansible/hg-web: configure moz.build chroot environment (bug 1139218); r=fubar, kang

We will be securing moz.build evaluation on hgweb machines with a chroot
and cgroups.

cgroups currently aren't configured on the hgweb machines. This commit
adds the Ansible bits to configure it.

We define a new control group - mozbuild - for evaluating mozbuild
files. We create a new "mozbuild" user and create a control group rule
that all processes for this user should be placed in the "mozbuild"
control group.

The "mozbuild" control group has some basic subsystem adjustments in
place to limit resource usage and device access.

Ansible has been updated to deploy our chroot files from the local
machine. It generates the current versions of the files and copies
them to the remote server.

We set up a bind mount of the repositories directory inside the chroot.
While the "mozbuild" user doesn't have permissions to modify anything
inside /repo/hg/mozilla (the entire tree is owned by hg:hg with
privileges 4775 on directories and 0664 on files), we add an extra layer
of caution by making the bind mount read-only to the chroot.

A sudoers policy to allow the "hg" user to invoke `mozbuild-eval` has
been added.

Comment 70

[Steven MacLeod [:smacleod]]

Comment on attachment 8634974 [details]
MozReview Request: hgmo: web command for retrieving moz.build info; r=smacleod

https://reviewboard.mozilla.org/r/13491/#review12985

LGTM

::: hgext/hgmo/__init__.py:253
(Diff revision 6)
> +    except Exception:
> +        return json.dumps({'error': 'invalid JSON returned; report this error'},
> +                          indent=2)

We're still missing a test case for this specific path.

::: hgext/hgmo/__init__.py:54
(Diff revisions 4 - 6)
> -message being returned to the user.
> +message being returned to the user. In all cases, stderr is logged if there
> +is some.

I'd prefer "Output to stderr is logged in all cases".

Comment 71

[Gregory Szorc [:gps]]

url:        https://hg.mozilla.org/hgcustom/version-control-tools/rev/142431d76639dae1d570a4a7c4f8131bd5bec86d
changeset:  142431d76639dae1d570a4a7c4f8131bd5bec86d
user:       Gregory Szorc <[e-mail]>
date:       Wed Jul 29 13:44:50 2015 -0700
description:
hgmo: support querying moz.build file info (bug 1139218); r=smacleod

The mozbuild Python package used by the Firefox build system recently
gained the ability to read moz.build files directly from Mercurial
repositories (bug 1168607). This opens the door to all kinds of exciting
possibilities for integrating moz.build info into Mercurial and tools
that are built on top of Mercurial, like code review (MozReview).

In this commit, we implement a generic Python module for extracting
moz.build file info from a Mercurial repository instance. We provide
a CLI command on the hgmo extension that prints obtained moz.build file
info. In future commits, we plan to call this CLI command from the
context of hgweb.

The output from the new command should be going through the Mercurial
templater. However, configuring this is extra work and there are no
users for non-JSON output yet. So, we defer this work until another
time.

url:        https://hg.mozilla.org/hgcustom/version-control-tools/rev/8dacf729ffd4ffad738dd3945b1b386530177f9a
changeset:  8dacf729ffd4ffad738dd3945b1b386530177f9a
user:       Gregory Szorc <[e-mail]>
date:       Tue Jul 28 14:22:48 2015 -0700
description:
hgweb-chroot: create a C program to containerize moz.build evaluation (bug 1139218); r=kang

The hg.mozilla.org WSGI processes (running as the hg user) will need to
spawn a new, sandboxed process to evaluate moz.build files (read
potentially untrusted Python code). This commit introduces the executable
that starts the process.

Implemented is a C program that creates a specialized, ultra low
privileged container for moz.build evaluation. I found
http://crosbymichael.com/creating-containers-part-1.html and
http://www.unixwiz.net/techtips/chroot-practices.html invaluable when
implementing this.

The program is targeted for CentOS 6. CLONE_NEWUSER isn't exposed to our
servers, so we don't need to worry about that and the privilege
escalation bugs that come with it.

url:        https://hg.mozilla.org/hgcustom/version-control-tools/rev/22d810d602d467d9b1023689b430c51936184d0f
changeset:  22d810d602d467d9b1023689b430c51936184d0f
user:       Gregory Szorc <[e-mail]>
date:       Tue Jul 28 18:10:25 2015 -0700
description:
hgweb-chroot: generation of chroot environment (bug 1139218); r=smacleod

We need to evaluate some possibly untrusted Python code on the hgweb
servers. These servers are running CentOS 6. As much as I'd like to use
LXC, getting it to work on CentOS 6 is more or less an exercise in
futility. CentOS 6 is running Linux 2.6 and many of the nice features of
LXC (such as user namespaces) aren't available until later kernel
versions.

Until we run hg.mozilla.org on a modern OS and kernel, chroots seem to
be the easiest and most practical solution available to us.

This commit introduces a Docker image that is used to build files that
will be used for a chroot environment suitable for evaluating moz.build
files. We build Python 2.7.10 from source and install our minimum required
Python package dependencies. We manually copy shared libraries needed for
running Python. There are tools like jailkit that make creating
chroot environments very simple. However, the environment is currently
very simple, so it's not too difficult to manage, so I didn't bother
with the added complexity.

A mach command for obtaining files from the Docker image has been added.
We'll hook up these files to our deployment infrastructure in a
subsequent commit.

url:        https://hg.mozilla.org/hgcustom/version-control-tools/rev/74b3e81cbc7c7723fe92fd7166516d8755014776
changeset:  74b3e81cbc7c7723fe92fd7166516d8755014776
user:       Gregory Szorc <[e-mail]>
date:       Wed Jul 29 13:38:59 2015 -0700
description:
ansible/hg-web: configure moz.build chroot environment (bug 1139218); r=fubar, kang

We will be securing moz.build evaluation on hgweb machines with a chroot
and cgroups.

cgroups currently aren't configured on the hgweb machines. This commit
adds the Ansible bits to configure it.

We define a new control group - mozbuild - for evaluating mozbuild
files. We create a new "mozbuild" user and create a control group rule
that all processes for this user should be placed in the "mozbuild"
control group.

The "mozbuild" control group has some basic subsystem adjustments in
place to limit resource usage and device access.

Ansible has been updated to deploy our chroot files from the local
machine. It generates the current versions of the files and copies
them to the remote server.

We set up a bind mount of the repositories directory inside the chroot.
While the "mozbuild" user doesn't have permissions to modify anything
inside /repo/hg/mozilla (the entire tree is owned by hg:hg with
privileges 4775 on directories and 0664 on files), we add an extra layer
of caution by making the bind mount read-only to the chroot.

A sudoers policy to allow the "hg" user to invoke `mozbuild-eval` has
been added.

url:        https://hg.mozilla.org/hgcustom/version-control-tools/rev/6ea3d500da28553e05925a7241485873f7a5f877
changeset:  6ea3d500da28553e05925a7241485873f7a5f877
user:       Gregory Szorc <[e-mail]>
date:       Wed Jul 29 15:42:38 2015 -0700
description:
docs: document moz.build evaluation (bug 1139218)

Comment 72

[Gregory Szorc [:gps]]

W00t.

I snuck in a change to the web command that won't enable evaluation unless a config option is enabled. So, once deployed, this code will essentially no-op in production. I'll follow up with kang to discuss the overall feature and the security impact before enabling this on any servers.

Comment 73

[Gregory Szorc [:gps]]

https://hg.mozilla.org/mozilla-central/json-mozbuildinfo

And that's doing exactly what it is expected to do.

Comment 74

[Gregory Szorc [:gps]]

After discussing with kang, I enabled moz.build info on mozilla-central. https://hg.mozilla.org/mozilla-central/json-mozbuildinfo should now work. I'll also enable support on hgcustom/version-control-tools.

We talked about further securing the sandbox with seccomp(), SELinux, or grsecurity. But we think there is a good balance of security, convenience, and risk in the current approach. We'll bolt on additional security in the future if warranted and/or when it is easier. e.g. our ability to run hardened kernels currently is somewhat restricted. Perhaps once we move things to AWS we'll have more flexibility.

Anyway, I'm calling this bug done. Yay! Thank you everyone who helped!