1187596 - [mig scheduler/agent] use separate exchange to receive agents results & heartbeats

Status: RESOLVED FIXED

(Enterprise Information Security Graveyard :: MIG, task)

Description

[Julien Vehent]

The scheduler and agents currently publish all messages into a single rabbitmq exchange called "mig". Agents publish hearbeats using a routing key named "mig.agt.heartbeats", and publish results using a routing key named "mig.agt.results". The scheduler consumes these messages.

Unfortunately, rabbitmq does not provide access control at the binding key level, meaning that any rogue agent can bind a queue with key "mig.agt.heartbeats" or "mig.agt.results" and get copies of the messages sent to the scheduler.

To do so, the agent needs a copy of the client certificate and rabbitmq username/password to connect to rabbitmq, which does significantly mitigate the exposure of this risk.

A solution to this problem requires a rewrite of the rabbitmq integration, and use two exchanges instead of one, inside of the "mig" rabbitmq vhost:
* "toagents" is an exchange where agents can bind and receive message from the scheduler, but have no permission to publish.
* "toscheduler" is an exchange where agents can publish heartbeats and results to the scheduler, but have no permission to bind

This architecture will prevent a rogue agents from obtaining copies of the messages sent to the scheduler.

It is a backward incompatible change. We can ease the migration by moving the scheduler to using the new exchanges, and writing a small worker that redirects messages to the old exchange until all agents are migrated.

needinfo alm for sanity checking.

Comment 1

[Aaron Meihm [:alm]]

Makes sense -- sounds like a good approach to me.

So, I'm assuming this also means we will now have 2 persistent AMQP sessions per agent, or would both exchanges be multiplexed over the same socket?

Comment 2

[Julien Vehent]

I think we can do with a single connection. The AMQP connection does not care about the exchange, it only cares about the virtual host.

Comment 3

[Julien Vehent]

Attachment: https://github.com/mozilla/mig/pull/77

The attached PR has a new organization using 3 exchanges to better separate permissions between schedulers, agents and workers.

I verified that a rogue agent is not anymore able to consume messages sent by other agents to the scheduler. This code can be used to test: https://gist.github.com/jvehent/6b440aa02a2c997e7510

When running it on a test system deployed with standalone_install.sh, it outputs:

    ubuntu@ip-10-153-223-98:~$ ./amqp_spy 
    panic: Exception (403) Reason: "ACCESS_REFUSED - access to exchange 'toschedulers' in vhost 'mig' refused for user 'agent'"

This should fix the issue. But deploying it will be a pain, because the code is backward incompatible.

Comment 4

[Julien Vehent]

Merged and deployed to prod.