Closed Bug 896480 Opened 11 years ago Closed 9 years ago

[TRACKER] MIG: Mozilla Investigator Endpoint Security

Categories

(Enterprise Information Security Graveyard :: MIG, task)

Product:
Enterprise Information Security Graveyard
Component:
MIG
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: jvehent, Assigned: jvehent)

References

(
URL
)

Details

MIG is a platform that allows investigators to send actions to pools of agents. What the actions do depend on the modules available on the agent, the MIG platform tries to focus on providing a set of capabilities without getting too specific about actual actions.

For example: an investigator launches an action to search for an apache module that matches a given md5 value. MIG will register the action, find all the relevant targets and send messages to each target with the content of the action. Each agent then individually perform the action using the module locally, and send the result back to the MIG platform. The MIG platform monitors execution, and will rerun actions when necessary.

Agents are designed to be lightweight and secure. Modules are executed with minimum privileges, in sandboxes when possible.


Phase 1	(Q3 2013)	
    - MIG platform components for single actions (API, DB, Scheduler, Action Queue, Relays)
    - Linux Agent
    - Modules:
        * File search (checksum, content)
        * Firewall control (verify rule, push/delete rule)
        * User/password control (compliance, creation/deletion)

Phase 2	(TBD)	
    - MIG WebUI: actions creation/deletion/status, target system view
    - Support for recurring actions (worker + DB)
    - Simple signature workflow (signature on action, signing chain for recurring->single actions)
    - Modules:
        * Memory search
        * Network sniffer (tcpdump & netfilter hook)
        * Processes inspector
        * Apps/Packages control (search, add, remove)

Phase 3	(TBD)	
    - Windows & MacOS Agent
    - Advanced signature workflow (double signature on actions, agent-side signature verification)
    - MIG WebUI improvement: search, reports, agents view, ...
Depends on: 896491
Depends on: 896579
Blocks: secauto
Status update: the code is at ssh://gitolite3@git-internal.mozilla.org/opsec/mig
Basic scheduler + agents messaging is implemented. Filechecker module is integrated. Next step is to write an iptables module, store results in a database backend, and write a simple API to submit actions.

Also in progress: discussion with the minion team on a common terminology to facilitate parsing results.
Depends on: 925778
Q3 is done and Phase 1 was mostly completed. The following features have not been developed and have been pushed to Q4:
 * Firewall control (verify rule, push/delete rule)
 * User/password control (compliance, creation/deletion)
Status update: lots of progress done on the messages format (actions & commands). MongoDB backend storage is operational. Work in progress to report action completion status. The next step are:
* expose the data in mongo via a rest API
* write a logic parser for actions that have multiple tests: (((test1 and test2) or test3) and test4)

Code is at https://github.com/mozilla/mig/commits/master
Status update: still polishing the messenging layer. Action are now stored in MongoDB, and completion ratio are calculated. Agent will kill long-running filechecker commands. A bunch of code cleanup, reorganization and doc has been done.

2013-12-05 a094f3e Check: rename root password check
2013-12-05 d0ff26d README update
2013-12-05 23f3d20 Scheduler: code reorganization
2013-12-05 f1d0938 Agent: change heartbeat freq to 5 minutes
2013-12-05 c7400c3 Agent: add timeout on long running filechecks
2013-12-05 e262eb6 Scheduler: Fix empty json result bug, caused by inotify pulling the file before it is fully written to disk
2013-12-05 25c7fde Scheduler: multi-critera agent search in mongo
2013-12-05 996953b Scheduler: Add timestamp on action and command
2013-12-05 64ab588 Scheduler: store actions in mongodb
2013-12-05 9d36cec Mozilla License added
2013-12-05 d3b338a Auditd compliance check update
2013-12-05 b308975 Filechecker: code cleanup and comments
2014 MIG goals:

Q1 Measurable Results:
    * Add features: heka logging, GPG, Agent auto-update, basic client console
    * Deploy server infra in AWS: https://mana.mozilla.org/wiki/display/~jvehent@mozilla.com/Mozilla+InvestiGator#MozillaInvestiGator-Architecture
    * Deploy agent on 100+ dev/staging servers
    * Cover 50% of system security policy in compliance checks

Q2 Measurable Results
    * Add features: Rest API, firewall Module, advanced client console
    * Deploy agent to 100% puppet managed nodes. 100+ non-managed nodes (AWS foundation, ...)
    * Open Source the code and prepare a brownbag + conference talk

Q3 Measurable Results
    * Add features: memory module, ACLs per functional group, Web interface
    * Give client access outside opsec, with limited ACLs
    * Target 100% agent deployment in staging & production (managed & unmanaged nodes)

Q4 Measurable Results
    * Add features: activity dashboard (for users), unbounded command module (with sandboxing)
    * Evaluate deployment on user's laptops, including concerns around privacy
Status update: major code refactoring was done over the last month. The scheduler and agent have been broken into smaller source code files. Logging is handled in a separate routine, with syslog support. PGP is here, both for signature and verification.

I am currently fixing PGP code to work with yubikeys, and handle more than one signing key.

2013-12-19 4dde8bf agent: path bin change
2014-01-06 749998f Scheduler: Happy New Year, with a massive refactoring commit.
2014-01-06 d08722c Build: better default
2014-01-06 f7459e8 Agent: find yer own path
2014-01-06 5de7bcd Mig-Action-Generator: First shot at the action creator, with GPG signature handling using gpgme
2014-01-08 4631689 Checks update
2014-01-08 d78ca3e PGP: Extract signature and verification into separate package
2014-01-08 6af329b Merge branch 'pgp'
2014-01-08 2481c5e Action: add Validation function that verifies PGP signature
2014-01-08 b5fda86 Agent: validate actions, includes PGP signature verification
2014-01-08 9535bb8 Mig-action-generator: use Validate() instead of doing it manually
2014-01-08 1037678 PGP: split package in two to prevent gpgme from being linked to mig agents
2014-01-08 e74b002 fix building script for pgp
2014-01-22 6882706 MIG PGP: Add convertion from armored pubkey string to keyring
2014-01-22 c330992 MIG PGP verification: take keyring as argument
2014-01-22 9a86824 MIG Agent: extract configuration
2014-01-25 db6c887 MIG Agent: code refactoring for context management and logging
2014-01-25 dfd0ec9 Rename Action.Check into Action.Order. More general.
2014-01-25 0ecb4bc debug logging by default
2014-01-25 82467f5 MIG Agent: make command launching generic
2014-01-25 ffa2b62 agent: verify scheduleddate before running commands
2014-01-26 46b3a16 Report build version in keepalive messages
2014-01-26 5081c64 Agent reports run failures back to Scheduler
2014-01-27 e09f5c9 Agent: use local keyring io.Reader (bugfix)
2014-01-27 5732869 Agent: code cleanup
Personal TODO list for workweek (deadline Feb.16 2014):
   1. Write usage & dev documentations
   2. Scheduler features:
      2.1 terminate running commands that have reached their expiration date
      2.2 search agents by substring in the agent.QueueLoc (contains OS family, fqdn hostname & id)
   3. console/api:
      3.1 receive new actions via public api
      3.2 return real time command results via websocket
   4. maybe: draft on agent auto-update
Group: mozilla-corporation-confidential
++
Depends on: 1011042
Depends on: 1014714
Depends on: 1014719
Depends on: 1014720
Depends on: 1014721
Depends on: 1014724
Depends on: 1014726
Depends on: 1014728
Depends on: 1014731
Depends on: 1014733
Depends on: 1014735
Depends on: 1014739
Depends on: 1014740
Depends on: 1014742
Depends on: 1014744
Depends on: 1014749
Depends on: 1037965
Depends on: 1037966
Depends on: 1037967
Depends on: 1058656
Suggestions from today's opsec presentation:
* save history in console, to make it available across restarts
* sanity check the Target string (pre-process to show agents count and validate)
* rename action counters: s/timeout/module timeout/ and s/cancelled/agent timeout/
* add a netcat feature: nc -vz destinationIP:port to actions (name - testconn or something)
* improve the documentation: console screencasts, flow diagrams, etc...
Depends on: 1068848
Depends on: 1068849
Depends on: 1068851
Depends on: 1069360
Depends on: 1069988
Depends on: 1071288
Depends on: 1104341
Depends on: 1104708
Depends on: 1104992
Summary: [TRACKER] Mozilla Investigator Endpoint Security → [TRACKER] MIG: Mozilla Investigator Endpoint Security
Depends on: 1109874
Depends on: 1109877
No longer blocks: secauto
Depends on: 1116003
Depends on: 1116210
Component: Operations Security (OpSec): General → Operations Security (OpSec): MIG
Depends on: 1116675
Depends on: 1116677
No longer depends on: 1014742
No longer depends on: 1037966
No longer depends on: 1116677
Depends on: 1116678
No longer depends on: 896579
No longer depends on: 1014714
No longer depends on: 1014719
No longer depends on: 1014720
No longer depends on: 1014721
No longer depends on: 1037965
No longer depends on: 1068851
No longer depends on: 1071279
No longer depends on: 1071286
Depends on: 1116682
No longer depends on: 1014735
No longer depends on: 1014740
No longer depends on: 1014744
No longer depends on: 1109877
Depends on: 1116685
No longer depends on: 1116003
No longer depends on: 1116210
No longer depends on: 1116675
No longer depends on: 1117359
Depends on: 1125172
Depends on: 1134390
Depends on: 1141900
No longer depends on: 1141900
Group: mozilla-employee-confidential
Component: Operations Security (OpSec): MIG → MIG
Product: mozilla.org → Enterprise Information Security
Version: other → unspecified
Group: mozilla-employee-confidential
Depends on: 1201659
Depends on: 1207709
We've had a good run on Bugzilla, but eventually tracking issues on github is more convenient, and hopefully will increase contributions.

So long, and thanks for all the fish.
Status: NEW → RESOLVED
Closed: 9 years ago
Resolution: --- → FIXED
Product: Enterprise Information Security → Enterprise Information Security Graveyard
You need to log in before you can comment on or make changes to this bug.