After a few mental missteps documented by previous blog posts (here and there), this is what I think of as a pretty reasonable approach to packet encryption in the Assimilation Project.  Although those two posts are now obsolete, the background post I wrote is still relevant.  I’ve learned a lot about crypto in the process  – and I suspect I’m not done learning ;-).  Hopefully my future crypto learnings will be smaller tweaks than my previous ones.  The general policies I’m now proposing are:

• All encryption is using the simplified public key authenticated encryption APIs from libsodium.  Libsodium is a more-generally-usable implementation of Dan Bernstein’s NaCl library.
• Libsodium uses Curve25519 for key exchange, XSalsa20 for the stream cipher, and Poly1305 MAC for authentication.  XSalsa20 is an improved version of Salsa20.
• Each nanoprobe has its own Curve25519 keypair, and the CMA has two keypairs.
• The secondary CMA private key is hidden away separately by a manual process by human beings.  It is intended to be used when the primary CMA private key is compromised.
• The CMA’s public keys are distributed to each nanoprobe machine as part of the installation of the nanoprobe software using an installation-defined method.
• Nanoprobes are authenticated using a Trust On First Use (TOFU) arrangement – analogous to that used by ssh.

The remainder of this post covers how we carry the actions associated with these policies by detailing how each of the several scenarios are carried out.  The scenarios detailed below are:

• CMA one-time initialization
• Nanoprobe one-time initialization
• Nanoprobe startup
• CMA to Nanoprobe command flow

In the sections below, there are several diagrams which follow these conventions:

• CMA actions are in green boxes
• Nanoprobe actions are shown in blue boxes
• Actions protected by encrypted communication are shown in solid boxes
• Actions not involving encrypted communication are shown in dashed boxes

CMA one-time initialization

This step occurs exactly once – when the CMA is first installed.

Although the diagram calls this a single CMA keypair, the code actually generates two keypairs.  When the CMA starts, if there is only one public CMA key, it will generate a second keypair.  Any time the CMA starts and finds two private CMA keys, it logs complaints to standard output and also to the system log.  The process for hiding the second private CMA key is purely manual.  The administrator should move it to reliable removable media and store it in a lock box or similar, or they should encrypt it in place under a different name using GPG or similar, and remove the plain text version of the secondary private key.  Restoring the private key is a similarly manual operation.

Since the administrator has to install the nanoprobe software on each machine, it is expected that they will install the CMA’s public key on nanoprobe systems at the same time as they install the software.   It is expected that the administrator will use a mechanism similar to ssh for this key distribution.  On the other hand, the CMA’s public key does not need to be kept secret.

Nanoprobe one-time initialization

This step occurs one time for each nanoprobe – when it’s started for the first time.

It looks to see if it has a nanoprobe key pair, and if not it generates it and saves it to disk.  The effects of cloning a nanoprobe system are described in the FAQ at the end of this document.

Nanoprobe startup

These steps occur each time a nanoprobe starts up.

When a nanoprobe starts up, it sends an unencrypted initial packet declaring that it’s up and providing a small amount of configuration information.  We have added two fields to that packet – one containing the key id and one containing the public key for the nanoprobe.   If the CMA has talked to this particular nanoprobe before, then it validates its identity by comparing the key id and public key.  The CMA marks the IP address that the nanoprobe sent the message from as being associated with the nanoprobe’s key id.  This is the key id used when sending future communications to that IP address.

All CMA packets are authenticated as being signed by one of the known CMA public keys before they are acted on.  The configuration packet for the CMA contains the set of IP:port addresses that the CMA might use in communication.  These IP addresses are marked as needing to be encrypted using the CMA key id used to send the configuration packet.  All future communication sent to any of these addresses will be encrypted using the public key that was used to encrypt the CMA’s configuration packet.  Before this command packet is received, the nanoprobe will not encrypt any communication.  Fortunately, it’s the first command sent to a nanoprobe after it starts up.

After all this, the configuration packet is then obeyed – that is, it updates the local configuration information.

Command Processing Flow

The diagram below describes the complete command processing flow for a command.

As before, solid boxes indicate encryption is in effect – green boxes are from the CMA, blue boxes from the nanoprobe.  So, the sequence we expect to use is:

1. CMA sends a command encrypted by both keys using crypto_box_easy(), and signed simply with a checksum.
2. The nanoprobe receives the command, and verifies that it’s signed by a key belonging to the CMA.
3. The nanoprobe then obeys the command.
4. The nanoprobe sends a response encrypted by both its key and the preferred key from the CMA.
5. The CMA then receives the packet from the nanoprobe, and verifies that it knows which nanoprobe sent it.
6. The CMA then processes the packet it received from the nanoprobe.

Note that this data flow is a superset of what happens when a CMA has something to report asynchronously.

 Assimilation Crypto FAQ

1. How are keys named?
   Each keypair has a unique key id.  CMA key ids are named “#CMA#nnnnn” where nnnnn is a 5-digit sequence number.  Nanoprobe keys are named “hostname::md5sum-of-public-key“.  In a multi-tenant environment, it is possible for host names to be duplicated.  However, by adding the md5 sum to the key id, it is wildly improbable that there would be a conflict of key ids.
2. How is the CMA’s key distinguished from a nanoprobes keys?
   The key id conventions make it impossible for a nanoprobe key to be mistaken for a CMA key.  In addition, the Assimilation system provides no mechanism for distributing public keys to nanoprobes.
3. What are the modes and ownerships of key pairs?
   Public keys are mode 0644 and private keys are mode 0600.  CMA keys are owned by the user id assimilation, and the nanoprobe keys are owned by root.
4. What happens if you clone a nanoprobe system?
   When the nanoprobe starts up, it looks for its own key pair.  Since the system name will no longer match, it will be unable to find a key pair matching its system name, and will generate a new key pair for its new system name.  The old key pair will remain where it is and be ignored.
5. What happens when packets are lost or reordered?
   The underlying reliable communication protocol is robust with respect to packet loss and reordering.  It also has its own replay protection scheme.
6. Can a nanoprobe running on the CMA system read the CMA’s private key?
   Yes.  Although the CMA runs as user id assimilation, many nanoprobe operations require root permissions, so nanoprobes run as root.  At startup, the nanoprobe reads in all key pairs looking for its own.  The only packets that a nanoprobe will accept from another nanoprobe are heartbeats, which only update a time stamp.  This makes it very difficult to compromise a nanoprobe without access to the CMA’s private key.

If this description has left questions in your mind – then let’s get going on a great conversation using the comment box below!  As always, any major changes will be announced on our blog and on @OSSAlanR. If you have an aversion comment forms – email alanr@unix.sh and I’ll incorporate them – I even have a GPG key [717A640E], or join the Assimilation development mailing list here.