In previous articles we gave some introductory material on how to get started with the Assimilation software – in 15 minutes and also in an hour. In A Half-Day To Better Security we go into more depth and suggest a good way to improve your security by spending a half-day with the Assimilation software. In this article, we’ll cover setting up email alerts for security changes, fixing your security issues, and setting up the Assimilation software on more systems.

A Half-Day To Better Security – Getting Security Alert E-Mails

In an earlier article, I wrote about how to use the Assimilation API to get emails for server and service changes. I’ve recently updated the script described in that email to also send security alerts.

You can find our sample script on GitHub. You’ll want to make a copy of this script in the /usr/share/assimilation/notification.d directory and make it executable (mode 0755 is a good choice). Since it is a sample script, you’ll have to edit it a little to customize it for your use. In the script are two default values that you must change, and one default value that you might want to change. Of course, it’s important that the machine you have set up as the CMA be able to send email. If it can’t send email – you need to get that working.

The first value in the script that you must change is EMAILDESTS. You should change this to the destination email address(es) that you want to send alerts to.

The second value you must change is RETURNADDR. You should change this to what you want the return address of these emails to show as. It could be a real address in your mail domain or something like “do-not-reply@your-email-domain.com”.

A value you might want or need to change is the EMAIL_METHOD value. This variable controls which command we use to send email. The possible values are currently:

• mail_email – use the “mail” command to send email
• mailx_email – use the “mailx” command to send email
• mutt_email – use the “mutt” command to send email

As written, the script defaults to use the mail command. Make sure the command you select is installed on the machine and can successfully send email.

Once you’ve installed the script, made it executable, and verified that the selected email command can successfully send email, you can go on to the next activity – which will trigger some security emails.

A Half-Day To Better Security – Fixing Problems

Before we dive right into fixing a problem, it’s worthwhile explaining how discovery works in a little more detail. Although the CMA scales really well largely because it delegates nearly everything and never polls for anything itself, many things have to be polled for by the nanoprobes. Once a discovery script completes, it is repeated at its designated repeat-interval. So, if a script takes approximately 10 seconds to run, and its repeat-interval is 60 seconds, then it will run approximately every 70 seconds. If the output from the script changes, it is immediately sent to the CMA and the database is updated and alerts scripts are run and so on. When nothing has changed (the usual situation), the nanoprobe throws away the results. The default repeat-interval for discovery scripts is currently 60 seconds. You can change the default and the value for any given discovery script – but we’ll defer configuring that to a future article.

Fixing something in /proc/sys

On my systems, the two discovery categories that have the most best practice complaints are those that come from /proc/sys discovery and rules and those that come from PAM. On the whole, the /proc/sys updates are easier to make correctly and easier to understand, so I’ve chosen a sample /proc/sys rule that needs fixing on my machines. If this your machine has this one right, then you can either break it as described below or choose another one.

For this exercise, I’ve chosen best practice nist_V-38528 – The system must log martian packets. I chose this rule because changing the value is unlikely to impact the function of the system involved and it’s easy to fix (or break). Borrowing from the STIG text, it says to determine if you have this problem to execute this command on the system running the nanoprobe:

sysctl net.ipv4.conf.all.log_martians

If it is set according to this particular best practice, the value you get should be 1. If not, you’re not in compliance with this recommendation. Fixing it temporarily is equally simple:

sudo sysctl -w net.ipv4.conf.all.log_martians=1

If your system was out of compliance before, you will see system log messages on the CMA system about the change, and get an email stating that it’s now in compliance. The email should have a subject line that says something like “WARNING: System <systemname> out of compliance with security best practice nist_V-38528“.This may take a bit over a minute. If you see syslog messages, but don’t get emails, then you need to check the “usual suspects” with the modified email script (mode 0755, syntax errors, right directory, etc).

If your system is already in compliance with this best practice, then executing this command will bring you out of compliance:

sudo sysctl -w net.ipv4.conf.all.log_martians=0

If you were in compliance before, you should see syslog messages about the change, and get an email stating that it’s out of compliance. If you want to bring it back into compliance, please wait for this email before fixing it – so that you get both the break and the fix emails. Feel free to break and fix it to your heart’s content ;-). As long as you wait for the system to discover the change, you should get an email at each transition. To fix this permanently, refer to the full description from the IT best practices project.

A Half-Day To Better Security – Assimilating More Machines

Now that you’ve gotten an installation under your belt, and gotten a feel for how the security rules are evaluated, it’s time to move on to assimilating the rest of the galaxy – starting with one more system. Like before, there’s a bit more learning to do before diving right in.

How do nanoprobes authenticate the CMA?

One of the key things to know is that all communication with the CMA is encrypted in both directions. As noted in an earlier crypto article, nanoprobes are able authenticate the CMA because they have a copy of its public key. So, when you install a nanoprobe, you need to install the CMA’s public keys at the same time. In the Assimilation software, all encryption keys are stored in /usr/share/assimilation/crypto.d. The CMA’s public keys are in that directory and are named #CMA#<serial-number>.pub. Those keys have to be present on each nanoprobe system. In our single system work, they were naturally available because the nanoprobe is running on the CMA system.

How do nanoprobes find the CMA?

Another thing to understand is how the nanoprobes locate the CMA. You might think that they “know” they’re on the same machine as the CMA so far, but that’s not the case.

There are two ways that the nanoprobes can find the CMA – through multicast, or by manual configuration. By default, nanoprobes find the CMA by sending out a single multicast packet to port 1984, multicast address 224.0.2.5 – which is reserved by IANA for the Assimilation software. Our choice of defaulting to UDP port 1984 has a more complex history. This often works quite well, particularly for demonstrations.  We set the multicast TTL (scope) to 31 (restricted to the same site, organization or department). The exact meaning of multicast TTL is determined by routing policy in your organization.

However, if multicast doesn’t work on your network, or the scope doesn’t match up with local routing policies, then you’ll have to manually configure the CMA address on your other nanoprobes. Multicast is unlikely to work if you’re in a cloud environment.

How to tell a nanoprobe where to find the CMA

The nanoprobe has a command line option that tells it where to find the CMA. That option is --cmaaddr. You must give it the IP address and port in either IPv4 or IPv6 format. So, for address 1.2.3.4, you would give it the option --cmaaddr 1.2.3.4:1984. If you prefer IPv6 notation, you could use --cmaaddr "[::ffff:1.2.3.4]:1984" – which is IPv6 notation for the same address. You can also use the DNS name of the CMA followed by “:1984” – for example if your CMA system were named unimatrixzero, then you could use --cmaaddr unimatrixzero:1984.

There are a variety of ways to set this value when the nanoprobe starts depending on which operating system and which version you’re running. On systems running systemd, the service file assimilation-cma.service has the startup options in ExecStart. On pre-systemd Debian based systems, you can set it in /etc/default/nanoprobe, and for CentOS-based, or SuSE based pre-systemd systems you set it in /etc/sysconfig/nanoprobe. In both cases, the variable to set is NANOPROBE_CMAADDR.

Installing a nanoprobe on a remote machine

You can also use the simple installer script to install a remote machine. This is good for continuing your testing, but of course, you most likely have a way of installing software on systems, and after you complete an initial trial, you’ll want to use that. But for now, the installer is handy and will work no matter what system management scheme you use.

Of course, you have to pick out a second machine to install the software on – at the present time there should be no firewalls between the systems.

To install the software on a remote machine named picard, you can run these commands on the CMA machine:

wget https://github.com/assimilation/assimilation-official/raw/master/buildtools/installme 
chmod 755 ./installme 
sudo ./installme --remote picard

At this point you will be prompted for several passwords – your local password for sudo, passphrases for scp and ssh, and the password for performing sudo on the remote machine. This is potentially a lot of repetitive password requests. The advantage of this method is that it copies the CMA public keys over automatically. Of course, you could run the installme script on the remote machine with the “nanoprobe” option and then copy the CMA’s public keys over using scp manually.

Starting the nanoprobe on the remote machine

Now that you’ve installed the nanoprobe on your nanoprobe-only machine, you can start it manually using one of these two commands:

sudo nanoprobe
sudo nanoprobe --cmaaddr <address-or-hostname-of-CMA>:1984

Once you do, then it should go through the same sequence of events as it did when you installed it on the first machine. Now you have the same detailed information about two systems as you had about one. Additionally, the two systems are now exchanging heartbeats – so if the nanoprobe on either one crashes shuts down gracefully (or the system they’re running on crashes), you’ll get log entries and events generated for them. If you start it without any options and the CMA doesn’t register it, it is likely that multicast doesn’t work in your environment.

This process can be repeated for as many machines as you have the patience – keeping in mind as I mentioned above, that if you need configuration to find the CMA, you’ll have to supply it in the way your OS wants it to be supplied. You’ll no doubt want to eventually get your site-normal installation procedures involved for nanoprobes before you move to production. If you’re running a cloud or container environment, you’ll want to add the nanoprobe, public key and optional configuration to your base image.

A little about firewalls, ports we use and so on

Earlier I mentioned you should avoid firewalls between the systems for your early trials. This is because getting the firewall rules right and going through the corporate politics involved is typically complex and time-consuming.

Although we use port 1984 when it’s available, there is one case where we know it will never be available – and this complicates firewall configuration. The CMA has to run on a well-known port – port 1984 by default. This means when you start up the nanoprobe on the CMA system that it has to bind to a different port. By default, it will bind to a random ephemeral port.

If you have internal firewalls, you typically have to know what ports to allow through. If you let it choose a random port, you can’t do that. So, you have to force it to bind to a fixed port so that your firewall can be programmed to allow heartbeats through. There are two ways to do this

• Bind the nanoprobe to a different well-known port than the CMA (not 1984).
• Bind the nanoprobe to port 1984 but a different IP address than the CMA. This also involves getting the CMA to listen to a specific IP address as well.

Typically, the second method is easier to accomplish – since getting a second IP address is usually easier than getting your firewall team to make their configuration more complicated.

To configure the CMA to listen on a different address you can use the –bind option like this:

--bind 1.2.3.4:1984

To configure the nanoprobe to listen to a specific address, it also uses the –bind option.

--bind 1.2.3.5:1984

This is only necessary for the CMA system – or any other system where UDP port 1984 is already occupied. These options can be set permanently similar to how they were set earlier. As noted before, the nanoprobe running on the CMA cannot be listening to the same port and IP as the CMA itself.

Summary: A Half-Day To Better Security

By now you have a good idea of many of the basic security capabilities of the Assimilation software. However, there are a number pieces of security-related information that we collect and keep up to date but didn’t have time to discuss. These include:

• Checksums of all network-facing binaries, libraries, and JARs
• Names and versions of every package installed on every machine

In addition there are a number of interesting features that aren’t typically considered to be security-related. These include:

• Zero-configuration server monitoring
• Near-zero configuration service monitoring
• Network interconnect attributes and map (via CDP or LLDP)
• Network best practices (similar to the security best practice rules)

We’ll try and get to all these features in future articles.