Di

Original dated Aug 2, 2017 found at https://www.peerlyst.com/posts/sharing-secrets-with-containers-using-custodia-alan-robertson

Distributing secrets in container environments is done dangerously more often than safely. This article gives an overview of secrets distribution using the open source Custodia package to distribute secrets safely in a really cool and novel way. Custodia will work in many more environments than this, but we only cover using Custodia with containers.

Distributing secrets is a hard thing to do right, and the consequences can be serious. If you make a mistake in secret sharing with computers, they can compound the problems you made much more rapidly than with humans.

It’s a problem with somewhat reasonable and accepted solutions when your software is running on bare metal using hardware security modules (HSMs). But when your software is running in the cloud, or worse yet running in containers in the cloud, HSM use breaks down. Custodia aims at solving the problem of getting keys distributed in this environment without any secrets being stored in the client containers.

What’s needed is a method that allows services running in containers to receive their secrets in a trustworthy and reasonably secure way. This is a separate problem from storing secrets – which has its own issues which we conveniently ignore in this article – since Custodia isn’t a secret vault.

What does reasonably secure mean?

Let’s start by defining what “reasonably secure” means. A reasonably secure solution would have these properties:

• Mutual authentication between the secret giver and the secret recipient.
• No secrets stored on disk in plain text.
• Secret distribution protocols are always fully encrypted by TLS or something similar and well-proven when going “over the wire”.
• Reasonably fine-grained authorization protocols. Just because one container has access to a secret doesn’t mean that all containers have equal access to it.
• Avoid storing secrets in container images or hard drives.

This list was inspired by the even better list that Jack Nickoloff gave in his review of secrets vaults

The Problem Is Trust

What you want is a trusted chain of custody from the secret vault to the application. What we need is for the secrets distribution mechanism to be trusted by the secret vault and be trustable by the recipients of the secrets. This mechanism has to have a way to identify which applications in which containers can receive which secrets –ideally without storing any secrets in the containers.

At first glance, it might seem odd that the normal approach is to give the application a secret that it can use to get the secrets it wants.It’s like saying we have to give you a passport before you can get access to the things you want to do — like travel to another country. The problem is that any secret you give an application is easily leaked if the application is compromised. When you compromise an application and get access to its operating environment (its container), then the chances are that the attacker has access to any credentials that the application has.

The approach taken by Custodia is to effectively replace the HSM by a host service on each host OS which is running containers. From the container model, the host OS can be viewed as the container’s “hardware”. That is, it is always trusted (because the container has no choice) and it can’t be easily compromised by applications running inside containers.

Custodia Container Architecture

Below is a diagram which covers deploying Custodia in a Kubernetes or similar Linux container environment.

In this diagram, Custodia runs on each VM including the system running container control (Kubernetes master). Inside each client machine (kubelet) it talks to each container using UNIX sockets, and the master instance using TLS. The advantage of UNIX sockets is Custodia can accurately identify each process which is trying to talk to it –without having to have each client process its own TLS certificate.In addition, UNIX sockets cannot be “sniffed” and allow each side to identify the process on the other end. IMHO, Custodia’s use of UNIX sockets is pretty slick.

Since Custodia is running in the host PID space, it can see all the details of its client processes in the container. This allows it to accurately identify user id, group id and security context of each of its client processes running in containers. It can also infer which image it’s part of through the cgroup that the process is running as.

Where are the Secrets?

In the diagram above, each secret is indicated by a red spiral icon.

• Each Custodia-kubelet must have a copy of a TLS client certificate identifying itself as a legitimate Custodia client. The master must have a copy of the public portion of this certificate.
• The master Custodia must have its own unique TLS certificate. Each Custodia-kubelet instance must have a copy of this public certificate.
• The master instance of Custodia must have a way of authenticating itself to the key stores (vaults). Each type of vault has its own authentication API.

It’s like looking at the DNA of the process asking for the secret

What’s readily seen is that there are no secrets of any kind stored by or in any of he containers. This is what’s cool. No need to push anything secret-like (no security tokens, no TLS certificates, nothing) into any of the containers. It’s like looking at the DNA of the process asking for the secret. It’s hard to forge, and not a secret you have to give you clients – they naturally have it.

Do container clients mutually authenticate with Custodia?

Since there are no certificates in the client, it’s worth exploring how container clients and Custodia mutually authenticate.

Clients can trust that they connecting to the “real” Custodia because the socket name is well-known and each client container is configured to have that socket be supplied on the host. This trust is conditioned on the assumption that the host OS been set up correctly, and not compromised. This is similar to the trust that an application has regarding hardware, BIOS and OS in a bare metal environment.

Custodia authenticates the clients using UNIX sockets APIs (SO_PEERCRED,SO_PEERSEC) and corresponding system calls. This allows it to determine the user id, security context and a great many other things about the process. This in turn, trusts that the userid that the service is running as is configured correctly when the instance was started. Looking at /proc is in a bit of a hack, but it’s a very useful hack, which potentially allows it to use even more information about a process to ensure that it is the entity it claims to be. This is analogous to the “something you are” method of verifying identity. In a lot of ways, Custodia could be said to be looking at the DNA of the process that’s making the request.

What’s the result of this?

• Only one VM has a copy of the vault certificates/tokens.
• No secrets are stored in any client containers. Container images are completely secret-free.
• Secrets are never stored unencrypted (they’re not stored at all)
• Secrets are always transmitted by methods which are very hard to intercept (TLS and UNIX sockets)
• Containers are built without having any secrets installed, or injected after startup.
• Host VMs (or bare metal depending on your situation) do need to have secrets installed.
• You need a Custodia plugin for each type of secrets vault you want to access.

How does this make it harder for an attacker?

One of the most common ways that adversaries compromise systems is by finding holes in locally-written software. If you have a microservices architecture, you potentially have a much larger attack surface for this kind of attack. For the purposes of this article, let’s assume the attacker is after our secrets. After all, this article is about protecting secrets 😉

If an attacker succeeds in compromising one of our applications, they are now in a container. If there is a secret stored on one of these containers (as is common practice), then they are likely running now running as the user id of the service – so they can have immediate access to whatever secrets this application has access to. In addition, the master instance of Custodia makes sure that the distributed instances are only asking for secrets they’re entitled to see. If one asks for a secret it’s not entitled to see, they immediately log the violation.

If on the other hand, they perform the same attack using this method of secret sharing, then there are no secrets for them to steal. If you’ve secured the application (mode 0555 or better), and made it the only thing capable of running as that user and in that security context, then they have a much higher barrier to get over – they have to break out of the container into the host, and then compromise the Custodia software, or get access to the host-server Custodia authentication secrets. This is a much higher barrier for the attacker. Since the best you can do is make a higher barrier for an attacker, that sounds like a win to me.

How To Do This Right?

• Each unique container service should have its own unique user id. You’re not still running your containers as root are you?
• Each unique container service should be running in its own security context.
• Only run trusted container images in your collection. [Always good advice ;-)]
• Take special care of the kubelet keys and the kubelet master key.
• Use good secrets vaults (one that follows the rules mentioned previously) and make sure they are properly installed and maintained.
• Use good methods for distributing secrets to VMs.
• Custodia allows http (non-TLS) communication. Always use TLS and mutual authentication between the client and server copies of Custodia.

What about verifying the container identity?

One of the potential benefits of this approach is that one can use the container image as part of the authentication process. It does this by using cgroups. As we noted before, this is a bit of a hack. Regardless of this, the inability to use this securely doesn’t detract from the general approach – particularly if each application runs under its own userid.

Conclusion

Custodia seems like a well-thought-out method of distributing secrets in a container environment. It is not yet a supported product, but seems to be a very promising technology preview. It completely eliminates the need for the container images to include secrets at the expense of requiring secrets at the VM layer. It provides fine-grained authorization while reducing the number of administrative secrets to be managed – since each application doesn’t have to have its own unique certificate or token. Adding distinct user ids and/or security contexts per application is a good bit less overhead than managing a unique certificate for each application.

What’s Next?

Obviously once you decide you want to use Custodia, you have to connect it to Vault or whatever secret storage mechanism you decide on.

I’ve also written a related article on a much simpler implementation using this same UNIX domain socket idea – without using Custodia. Naturally, I think it’s even cooler 😉