In our previous article, we went noted: No matter your threat model, you need to understand what you have (“know yourself“). In that article, we took you through the Assimilation installation process, and showed you how to create a visualization of your attack surface, how to retrieve server security scores, and pointed you at the web interface to Neo4j. In An Hour Towards Better Security, we’ll go further and show you how to triage your security issues to create an efficient attack plan to get your site to follow security best practices.
An Hour Towards Better Security
In the previous article we only got as far as getting an overall per-server security score. Although overall security scores are nice, and one over the whole site is really helpful to see how you are improving over time, but they don’t tell you exactly what problems you have. The next command will show you in detail which areas and rules are not being followed. To do that, issue this command:
sudo assimcli --ruleids secdtype
It will produce output which looks something like this – but probably with more lines of output:
proc_sys,9.0,nist_V-38601,1.0 proc_sys,9.0,nist_V-38600,1.0 proc_sys,9.0,nist_V-38548,1.0 proc_sys,9.0,nist_V-38533,1.0 proc_sys,9.0,nist_V-38532,1.0 proc_sys,9.0,nist_V-38529,1.0 proc_sys,9.0,nist_V-38528,1.0 proc_sys,9.0,nist_V-38526,1.0 proc_sys,9.0,nist_V-38511,1.0 sshd,6.0,nist_V-38617,1.0 sshd,6.0,nist_V-38616,1.0 sshd,6.0,nist_V-38615,1.0 sshd,6.0,nist_V-38613,1.0 sshd,6.0,nist_V-38610,1.0
At first glance, this may be a little confusing. It’s obviously CSV format, but what the heck does it mean? Let’s go over it one field at a time.
- Rule Functional area. Each best practice rule belongs to a specific functional area. Examples of functional areas are PAM rules, /etc/login.defs contents, /etc/ssh/sshd_config, contents of /proc/sys, and so on. Each corresponds a single discovery agent.
- Functional area total score. This is the sum of all the scores for all the rules in this functional area across all the servers monitoring. So far in this exercise, that’s just one server.
- Rule ID. This is a unique identifier for this rule – taken from the IT Best Practices project. The ITBP project maintains descriptions and remedies for all their rules. They are currently stored in GitHub, and will soon be available through their web site. I manually enhanced the Rule IDs on this page to point to the individual ITBP JSON files in GitHub. This format is great for machines, but are a little lacking for human readers. For human readers, I recommend the IT Best Practices lookup site. The URLs that go with failed rules also appear in the system logs.
- Individual rule score. This is the sum of all the scores for this rule across all the systems we are monitoring.
Now, let’s talk about the sort order, and why it’s helpful to you. We sort first by functional area total score, then by functional area, then by rule score, then by rule id.
This has the effect of showing you all the rules associated with the biggest problem area first. In other words, if you address this area first, you will have the biggest impact on your overall security posture. This is ideal if you’re using something like Ansible, Chef or Puppet to manage your systems.
Here’s a good way to approach this: learn about the top functional area (configuration file) and what these errant settings mean in this one area, then fix the problems in that area in your base configuration, push out the change across your environment. This will give you the biggest impact on your security posture while minimizing brain context switch overhead – making your more efficient. Whoda thought sort order could be so cool?
One other really cool thing to keep in mind – these scores are continually updated as you make changes to your systems. Once you get one area taken care of, you can do the query again and find out a new area – and your risk scores will keep going down and down and down :-D.
So far, most of the rules we’ve put in place for best practices are in a single directory. When you get your output, you’ll want to go learn about the on the IT Best Practices github repository – or more specifically in the directory mentioned previously.
Finding the Active IP and MAC Addresses On Your Subnet
For the next activity in an hour towards better security, let’s look at the active IP and MAC addresses on your network. By now we should have most of the active IP and MAC addresses on your subnet. Depending on the type of network gear, and its activity level some devices (notably Cisco switches) may take up to 20 minutes to show up. Since Verizon has stated that 30% of all outsider attacks use servers people have lost track of, and other studies show that 30% of all servers are zombie servers, so don’t be surprised if you don’t recognize them all.
To find the set of active IP and MAC addresses on your subnet, issue the following command:
sudo assimcli query allips
When you do, you’ll get a good bit of output that looks something like this:
::ffff:10.10.10.249 a0-21-b7-a1-83-5f undefined ::ffff:10.10.10.250 c4-3d-c7-96-9c-e4 c4-3d-c7-96-9c-e4 ::ffff:169.254.254.254 00-0e-f3-31-24-c4 undefined ::ffff:172.17.42.1 56-84-7a-fe-97-99 servidor
The first value is the IP address (in IPv6 format), the second is the MAC address, and the third is the system’s idea of its name. You may get lots of undefined values in the last column. These are IP addresses that correspond to systems we don’t have agents on. The one with the funky mac-address-looking name is actually from a switch that’s sending us LLDP packets. Part of what it told us was what it thought it’s unique name was. The system servidor is the name of the system that I was running the Assimilation Suite on. These queries don’t (yet) display it, but the database has the OUIs (manufacturer information) from the MAC address, and the DNS name of the IP addresses. If you only see 2 or 3 entries, you’re likely running in an environment where ARP packets are filtered – like a cloud environment.
Like the scoring above, this data is continually updated. A new device only has to be on the network a few seconds for it to show up. Once it’s on the network, it appears the database a few seconds later. There’s a really funny story about this – but I’ll save that for another time.
Exploring The Neo4j database
We store all our data in the open source Neo4j database. It’s a graph database, or more specifically a property-graph database. It represents all its data as nodes and relationships. This is pretty much a digital representation of a typical whiteboard drawing of a technical system. When people draw such systems, they consist of circles or boxes and arrows or lines. In graph terminology the boxes and circles are called nodes, and the lines and arrows are called relationships. Both nodes and relationships can have associated properties. In our implementation, each node is of a particular type, which we represent by the nodetype property. Relationships also have types as well. Graphs we produce currently can include 13 different node types and 12 different relationship types.
This kind of database is much better for modeling IT infrastructure than a more traditional relational database – explaining why is worth an article of its own.
To get started exploring the database, you need to open up a browser to http://localhost:7474 on the machine neo4j is running on. By default the Neo4j web server only serves localhost. It will ask you for authentication – the user id and password we set will be found in /usr/share/assimilation/crypto.d/neo4j.creds. The first line is the user id, and the second is the password. These are stored in a protected file, in a protected directory – and are needed by the Assimilation software for its connection to the database.
In this article, we’ll be doing very basic exploration of the database, just to give you a feel for what kind of data is in the database. To start this exploration, you need to click on the upper-left icon in the screen – which looks like the icon to the left of this text. Once you do that, a Database Information tab will expand under it. It has two interesting headings under it: Relationship Types and Property Keys. Let’s start with the Property Keys. If scroll down, you’ll find the property name nodetype. If you click on nodetype, you’ll see all the different node types that we have on your system. And at the top of that window is the Cypher query that produced that result. The Cypher they generated for this is a bit of overkill for our situation. A simpler query that gets equivalent results is MATCH (n) WHERE has(n.nodetype) RETURN DISTINCT n.nodetype LIMIT 25. If you paste that in on top of their query (the multi-colored text to the right of the $ symbol), and press enter, it will execute that query for you.
I’m not going to give a tutorial on the Neo4j Cypher query language, but you can see it has elements in common with SQL. What I would suggest you do is click on all the following different property keys, and look at the kinds of data we’ve discovered and stored away for you.
- pathname – this will show you the full pathnames of every binary talking on the network via TCP.
- argv – this will show you the full argument list of these same binaries
- designation – the names of all “systems”. A system is either a network device that we’ve discovered by LLDP or CDP or its a system running our nanoprobe.
- uid, gid – the real user ids (group ids) of all the active network processes
- procinfo – JSON describing each active network process we’ve discovered
- processname – unique identifiers for each active network process we’ve discovered. Each procinfo item is associated with a processname.
- macaddr – all the MAC addresses we’ve discovered
- ipaddr – all the IP addresses we’ve discovered
- monitorname – unique identifiers for each service we’re monitoring
Although this is fun to do and gives a flavor for what kind of data we capture, you usually need to do more sophisticated queries to tell you really useful things things. Fortunately, we’ve written a number of canned queries – like the IP address query above.
More Queries
So far in an hour towards better security, if you did the exercises above, and spent time understanding and investigating the data in front of you, you probably spent the better part of a half hour – and you’ve only scratched the surface of what this data can do for you – and so far it’s all been pretty easy to do – and you probably get why this is pretty cool. Something for you to explore is what other kinds of queries you can do. Naturally, “there’s a query for that” ;-). It’s named the “list” query. To get the set of canned queries, issue this command:
sudo assimcli query list
When you do, you’ll be presented with a list of canned queries that look something like this:
allbpscores: return all host BP scores allipports: get all port/ip/service/hosts allips: get all known IP addresses allservernetscores: return host network BP scores allservers: get known servers allserversecscores: return host security scores allservicestatus: status of all monitored services allswitchports: get all switch port connections crashed: get 'crashed' servers down: get 'down' servers downservices: get 'down' services findip: get system owning IP findmac: get system owning MAC addr hostdependencies: host's service dependencies hostipports: get all port/ip/service/hosts hostservicestatus: monitored service status on host hostswitchports: get switch port connections list: list all queries nodecount: get list of node types and their counts shutdown: get gracefully shutdown servers unknownips: find unknown IPs unmonitored: find unmonitored services
Exploring these will likely keep you busy for the remaining half-hour And in case you’re wondering – yes, the canned queries really are objects in the Neo4j graph database ;-). You can find our source for these queries in github. And this means you can easily add your own. If you read those query files, the actual Cypher query is in the field named cypher. The rest of that JSON is metadata to help us properly issue the query.
Coming Next
In this article, we showed you how to figure out where your security issues are – but so far for only a single server. In the next article, we’ll show you how to extend your security exploration into multiple systems. This is where the Assimilation System really shines – because it scales like nothing else.
Please note: I reserve the right to delete comments that are offensive or off-topic.