Sample Demo Output

What is the Assimilation Software?

The Assimilation Software is an extensible and extremely scalable discovery-driven automation engine that uses the discovery data to drive other operations, including continuous best practice compliance audits.  Because we do excruciatingly detailed discovery, we have all the information needed to do these other operations with little or no human input.

Mini architectural overview

We have a central system which we call the Collective Management Authority (CMA), which stores its data in the Neo4j graph database.  Every system we manage has an agent on it which we call a nanoprobe.  Nanoprobes are policy free.  The only thing they do on their own is announce when they start and when they stop.  Everything else they do is because the CMA has told them to do something.  In this architecture, there is one central CMA for everything, and there is a nanoprobe process (agent) on every machine in the data center – including on the machine the CMA is running on.  For this simple demo, we have one nanoprobe which happens to be running on the same machine as the CMA.

The Demo

The key things to note about this demo are:

  • We wipe out the database when we start
  • Nothing was given any configuration information, everything was discovered – including where the CMA is running.

Output from a Trial run : September 2015

Below is some output from a recent test run of the Assimilation software on an Ubuntu desktop. Note that the security rules in this demo were designed for Red Hat 6 but adapted to do the same for Ubuntu as they did for Red Hat. Note that this is a newer version of a demo similar to the 90 second demo. I’ve made a few of the lines bold to better help clarify the output. Like the other demo, everything here was discovered – nothing was configured by hand.

[1]servidor:~/monitor/src $ sudo cma/cma.py --foreground --erasedb 2>&1 | tee cma.out &
servidor:~/monitor/src $ WARNING: Incorrect number of Private CMA keys. Expecting 1, but got 2.
WARNING: YOU MUST SECURELY HIDE all but one private CMA key.
WARNING: SECURELY HIDE *private* key /usr/share/assimilation/crypto.d/#CMA#00001.secret
Monitoring rules loaded from /usr/share/assimilation/monrules
Fork/Event observer dispatching from /usr/share/assimilation/notification.d
cma/assimcli.py loadbp ./Starting CMA version 1.0.1.1441715487 - licensed under The GNU General Public License Version 3
Neo4j version 2.2.5 // py2neo version 2.0.7 // Python version 2.7.9 // java version "1.7.0_79"
cma: Starting up untraced listener.listen() in foreground; debug=0
[2]CLOSING CONNECTION (closeconn) TO [::ffff:10.10.10.5]:35841
[3]==== Evaluating 6 Best Practices rules on "Login configuration from /etc/login.defs"
FAIL: security ID nist_V-38475 MUST(GE($PASS_MIN_LEN, 14))
FAIL: security ID nist_V-38477 MUST(GE($PASS_MIN_DAYS, 1))
FAIL: security ID nist_V-38479 MUST(LE($PASS_MAX_DAYS, 60))
PASS: security ID nist_V-38480 MUST(EQ($PASS_WARN_AGE, 7))
PASS: security ID nist_V-38576 MUST(EQ($ENCRYPT_METHOD, SHA512))
PASS: security ID nist_V-38645 MUST(EQ($UMASK, 077))
[4]==== Evaluating 11 Best Practices rules on "sshd configuration from /etc/ssh/sshd_config"
PASS: security ID nist_V-38484 MUST(IN($PrintLastLog, "yes", True))
PASS: security ID nist_V-38607 MUST(IN($Protocol, 2))
FAIL: security ID nist_V-38608 MUST(LE($ClientAliveInterval, 900))
FAIL: security ID nist_V-38610 MUST(EQ($ClientAliveCountMax, 0))
PASS: security ID nist_V-38611 MUST(IN($IgnoreRhosts, "yes", True))
PASS: security ID nist_V-38612 NONEOK(IN($HostbasedAuthentication, "no", False))
FAIL: security ID nist_V-38613 MUST(IN($PermitRootLogin, "no", False))
PASS: security ID nist_V-38614 NONEOK(IN($PermitEmptyPasswords, "no", False))
FAIL: security ID nist_V-38615 MUST(EQ($Banner, "/etc/issue"))
FAIL: security ID nist_V-38616 MUST(IN($PermitUserEnvironment, "no", False))
FAIL: security ID nist_V-38617 MUST(IN($Ciphers, aes128-ctr,aes192-ctr,aes256-ctr,aes128-cbc,3des-cbc,aes192-cbc,aes256-cbc))
[5]REQUESTING CHECKSUM MONITORING OF 88 files
**don't know how to monitor ['/usr/sbin/apache2', '-k', 'start']
**don't know how to monitor ['/usr/sbin/dnsmasq', '--no-resolv', '--keep-in-foreground', '--no-hosts', '--bind-interfaces', '--pid-file=/run/sendsigs.omit.d/network-manager.dnsmasq.pid', '--listen-address=127.0.1.1', '--conf-file=/var/run/NetworkManager/dnsmasq.conf', '--cache-size=0', '--proxy-dnssec', '--enable-dbus=org.freedesktop.NetworkManager.dnsmasq', '--conf-dir=/etc/NetworkManager/dnsmasq.d']
**don't know how to monitor ['/home/alanr/.dropbox-dist/dropbox-lnx.x86_64-3.8.8/dropbox']
START monitoring neo4j using ocf agent
**don't know how to monitor ['/usr/lib/x86_64-linux-gnu/libexec/kdeconnectd']
**don't know how to monitor ['/usr/sbin/minidlnad', '-f', '/etc/minidlna.conf', '-P', '/run/minidlna/minidlna.pid']
START monitoring named using ocf agent
START monitoring rpcbind using lsb agent
NEVER monitor skype
START monitoring check_ssh using nagios agent
START monitoring host servidor using nagios:check_load agent
START monitoring host servidor using nagios:check_sensors agent
[6]==== Evaluating 11 Best Practices rules on "PAM configuration in /etc/pam.d"
FAIL: security ID nist_V-38482 MUST(LE(PAMMODARGS(OR($system-auth, $common-auth), auth, pam_cracklib, dcredit),-1))
PASS: security ID nist_V-38497 NONEOK(NE(True, PAMMODARGS(OR($system-auth, $common-auth), auth, ANY, nullok)))
PASS: security ID nist_V-38501 NONEOK(GE(PAMMODARGS(OR($system-auth, $common-auth), password, pam_unix, nullok), 900))
FAIL: security ID nist_V-38569 MUST(LE(PAMMODARGS(OR($system-auth, $common-auth), password, pam_cracklib, ucredit),-1))
FAIL: security ID nist_V-38570 MUST(LE(PAMMODARGS(OR($system-auth, $common-auth), password, pam_cracklib, ocredit),-1))
FAIL: security ID nist_V-38571 MUST(LE(PAMMODARGS(OR($system-auth, $common-auth), password, pam_cracklib, lcredit),-1))
FAIL: security ID nist_V-38572 MUST(GE(PAMMODARGS(OR($system-auth, $common-auth), password, pam_cracklib, difok),4))
FAIL: security ID nist_V-38574 MUST(EQ(True, PAMMODARGS(OR($system-auth, $common-auth), password, pam_unix, sha512), PAMMODARGS(OR($system-auth-ac, $common-auth), password, pam_unix, sha512)))
FAIL: security ID nist_V-38658 MUST(GE(PAMMODARGS(OR($system-auth, $common-auth), password, pam_faillock, remember),24))
FAIL: security ID nist_V-38693 MUST(LE(PAMMODARGS(OR($system-auth, $common-auth), password, pam_cracklib, maxrepeat),3))
FAIL: security ID nist_V-51875 MUST(EQ(PAMMODARGS(OR($system-auth, $common-auth), session, pam_lastlog, showfailed),True))
[7]==== Evaluating 19 Best Practices rules on "/proc/sys directory"
FAIL: networking ID itbp-00001 IN($net.core.default_qdisc, fq_codel, codel)
FAIL: security ID nist_V-38511 EQ($net.ipv4.ip_forward, 0)
PASS: security ID nist_V-38523 EQ($net.ipv4.conf.all.accept_source_route, 0)
PASS: security ID nist_V-38524 EQ($net.ipv4.conf.all.accept_redirects, 0)
FAIL: security ID nist_V-38526 EQ($net.ipv4.conf.all.secure_redirects, 0)
FAIL: security ID nist_V-38528 EQ($net.ipv4.conf.all.log_martians, 1)
FAIL: security ID nist_V-38529 EQ($net.ipv4.conf.default.accept_source_route, 0)
FAIL: security ID nist_V-38532 EQ($net.ipv4.conf.default.secure_redirects, 0)
FAIL: security ID nist_V-38533 EQ($net.ipv4.conf.default.accept_redirects, 0)
PASS: security ID nist_V-38535 EQ($net.ipv4.icmp_echo_ignore_broadcasts, 1)
PASS: security ID nist_V-38537 EQ($net.ipv4.icmp_ignore_bogus_error_responses, 1)
PASS: security ID nist_V-38539 EQ($net.ipv4.tcp_syncookies, 1)
PASS: security ID nist_V-38542 EQ($net.ipv4.conf.all.rp_filter, 1)
PASS: security ID nist_V-38544 EQ($net.ipv4.conf.default.rp_filter, 1)
FAIL: security ID nist_V-38548 EQ($net.ipv6.conf.default.accept_redirects, 0)
PASS: security ID nist_V-38596 EQ($kernel.randomize_va_space, 2)
n/a: security ID nist_V-38597 EQ($kernel.exec-shield, 1)
FAIL: security ID nist_V-38600 EQ($net.ipv4.conf.default.send_redirects, 0)
FAIL: security ID nist_V-38601 EQ($net.ipv4.conf.all.send_redirects, 0)
PROCESSING CHECKSUM DATA
[8]UPDATING CHECKSUM DATA for 308 files
MESSAGE: Service servidor:ocf:neo4j::bc867df5300476bcf5a53bae3bf21750 is now operational
MESSAGE: Service servidor:ocf:named::9b03625830b70c90301781cefc9ff050 is now operational
MESSAGE: Service servidor:lsb:rpcbind::f4074868e84dc00ed7c9e8b3feea4983 is now operational
MESSAGE: Service servidor:nagios:check_ssh::0fffa678561bf2e8636bd2806384eb8f is now operational
MESSAGE: Service servidor:nagios:check_load::c400e6551ae7af9414c78b4fdc047cee is now operational
MESSAGE: Service servidor:nagios:check_sensors::e39857f8fe07267a256c797f472b3747 is now operational
[9]nanoprobe: NOTICE: nanoprobe: exiting on SIGINT.
System servidor at [::ffff:10.10.10.5]:35841 reports graceful shutdown.

What Happened in the Demo Above?

I added the bold [1] notation to the output above so I could explain it step-by-step below.

  1. This step starts up the CMA. The –erasedb option erases everything in the database, including security rules and canned query descriptions. The assimcli.py loadbp command loads the best practice rules into the database.
  2. The “closing connection” message occurs when the nanoprobe first connects, and the CMA resets the connection (if any). The CMA then sends the nanoprobe a series of discovery commands. One of discovers the system’s /etc/login.defs settings, another discovers the system’s /etc/sshd settings, still another discovers TCP client and server processes, and so on.
  3. When the /etc/login.defs discovery information comes in, then we evaluate the 6 best practice rules that relate to login configuration. We fail 3 and pass 3.
  4. Next the /etc/sshd configuration arrives, and we compare it against 11 best practice rules. We fail 6 and pass 5.
  5. The next discovery information to arrive is from client/server processes. This triggers a request to discover the checksums of 88 files related to the network-facing processes we discovered. In addition, we send commands to monitor the services being offered. In this example, we monitor 6 services, don’t know how to monitor 5 of them, and we know we shouldn’t ever monitor Skype. Two of the things being monitored are not related to services but to the machine itself.
  6. The next discovery information to arrive is the /etc/pam.d configuration information. There are 11 rules associated with PAM configuration, and we pass two of them and fail 9.
  7. The next thing to arrive is the /proc/sys discovery results. There are 19 rules associated with /proc/sys. We pass 8 of those tests, fail 10, and one is not applicable. These rules are designed for an older Red Hat kernel, and the value /proc/sys/kernel/exec-shield is not present in the kernel I ran this test on. Next, the checksum discovery we requested in step 3 above arrives.
  8. We store away the checksums for 308 files. The observant reader will notice that we only asked for the checksums of 88 files, yet we came back with 308 checksums. This is because we also compute the checksums of every dependent library – using ldd(1). Back in step 3, when we had discovered the services, we sent some monitoring commands for the nanoprobe to perform. In this step, we observe that the services we’re monitoring are now all working correctly.
  9. I’d started the nanoprobe in the foreground. I press Ctrl-C, and it exits with the exiting on SIGINT message. The CMA is still running and reports that the graceful shutdown. There is a specific query in the database to ask for which systems were shut down gracefully, and another one to ask for those which appear to have crashed.