How to make your microservice architecture end-to-end confidential

Service Mesh is one of the most hyped technologies in the cloud native ecosystem. But behind all the noise there is some real important value that it brings to the table. "The service mesh gives you features that are critical for running modern server-side software in a way that's uniform across your stack and decoupled from application code," said William Morgan, CEO of Linkerd.

Critical, because when you are building modern server-side software, you are likely to build a collection of services that synchronously talk to each other. Uniform across your stack, because these features apply to every service in your application, regardless of its technical details. Decoupled, because those features are implemented on the service mesh layer, requiring no changes to your applications. The features can be grouped into three classes: Reliability, Observability, and Security.

The key problem here is trust: you need to establish trust throughout your service mesh and especially you need a root of trust in your mesh's control plane.

Confidential service mesh

If you are building confidential server-side software, you have the same kind of feature demands to a service mesh when it comes to reliability and observability. With security, however, you are faced with a bunch of additional challenges. The key problem here is trust: you need to establish trust throughout your service mesh and especially you need a root of trust in your mesh's control plane.

In the following, we'll explore the aspects of service mesh in confidential computing by deploying the Open Service Mesh (OSM) together with the confidential-computing control plane MarbleRun.

Open Service Mesh (OSM) is a lightweight and extensible cloud native service mesh. Using the CNCF Envoy project, OSM implements the Service Mesh Interface (SMI). One of OSM's distinct features are access control policies that define your cluster's internal and external network communications, essentially "service to service" and "outside-world to service" communications. It follows a whitelisting approach to protect against common misconfigurations leading to service exposure in Kubernetes.

MarbleRun is an open-source control plane that simplifies deploying, scaling, and verifying SGX-based apps. It is designed to run on Kubernetes --- whether alongside a service mesh or as a standalone service.

Bookstore demo

OSM has a great demo application called the bookstore. Books are stored in a database called the bookwarehouse. The bookwarehouse is accessed and managed by the bookstore, a simple web-based online store. The demo comes with two services accessing the bookstore. A legitimate service called bookbuyer is intended to access the bookstore and retrieve books. The other service called bookthief represents a malicious or wrongly configured service that shouldn't be able to access the bookstore. When such access happens, the books are considered to be "stolen". The demo is a great showcase for us because it has a clear focus on the security features of a service mesh.

Threat models

Let's look at the threat model in the bookstore demo. The bookthief as the central aspect of this demo showcases the importance of isolation in your cluster. Misconfigured or vulnerable services can be an entry point for attackers and can lead to data leaks in any part of your application. OSM helps you to protect against these threats by enforcing strict access control policies.

While not explicitly highlighted in the demo, attackers with access to the network traffic can try to exfiltrate information by tapping into service-to-service communication. To prevent this, OSM enables encrypted communication. It uses sidecar containers to establish mutual TLS connections between your services.

In short, the usual Kubernetes security protections and isolations are insufficient to protect against internal threats or powerful external attackers.

In confidential computing, we extended this threat model. We assume no part of the existing infrastructure can be trusted. Imagine that an attacker can gain access to every part of your cluster's infrastructure. Either as a powerful external attacker able to compromise nodes in your cluster or in the form of an internal threat, for example, a malicious cloud administrator. Someone with such capabilities can leak data by accessing the bookwarehouse directly or by replacing the trustworthy bookbuyer service with a malicious one that exfiltrates data. They could also intercept the sidecar to service communication, hence, breaking the mutual TLS protection. We could go on and on here, but you get the idea.

In short, the usual Kubernetes security protections and isolations are insufficient to protect against internal threats or powerful external attackers. When faced with such strong attackers, we must assume compromise for all nodes and the entire Kubernetes control plane.

Confidential bookstore

MarbleRun addresses this seemingly unsurmountable challenge by bringing the virtues of confidential computing to your Kubernetes deployment. In the following, we'll demonstrate how using bookstore demo as example.

Let's assume books represent our sacred information (they do, right?) and should stay confidential. First, we need to transform our bookstore application into confidential microservices. All services are written in Go so our natural choice here is to use EGo. No code changes needed, easy! That already means an attacker can't steal any books from accessing our services directly or dumping their memory.

EGo is an open-source SDK that enables you to develop your own confidential apps in the Go programming language.

Next, let's secure our bookwarehouse database. We need to make sure the data is protected at runtime and on disk, plus we need to make sure only our legitimate bookstore service can access the database. EdgelessDB is the open-source database designed for confidential computing that ticks all these boxes and it's MySQL-compatible so it's just a simple drop-in replacement. Great!

EdgelessDB is a full SQL database that runs entirely inside runtime-encrypted Intel SGX enclaves. In contrast to conventional databases, EdgelessDB ensures that all data is always encrypted --- in memory at runtime as well as on disk.

Finally, we need to encrypt our communications and ensure services are authenticated using a form of identity. This is where we want to build on the service mesh philosophy. MarbleRun augments service meshes by implementing the control plane for confidential computing. Providing enclave to enclave mutual TLS, without sidecars, and remote attestation-based authentication of services.

Looking at our threat model, MarbleRun essentially establishes a trusted party in the Kubernetes control plane. Hence, it can handle identity, authentication, secret distribution, and bootstrapping of your confidential services. MarbleRun follows the cloud native philosophy of doing one thing and doing it well. Thus, it augments existing service meshes like OSM and does not reimplement their whole feature set. In the bookstore deployment, you install MarbleRun alongside OSM. That way you keep the existing features and benefits of OSM while solving the confidential computing-related tasks with MarbleRun.

What does that mean in practice? MarbleRun verifies and authenticates all services, provides them with credentials and configurations, and establishes secure TLS connections between them. Take for example the EdgelessDB instance. It is configured to only allow access to an authenticated bookstore service via TLS using MarbleRun credentials. After deploying the bookstore application, we can get a concise remote attestation statement of the whole deployment. Such a statement can be used to prove the integrity and confidentiality of your application to 3rd parties. More importantly, it can be used by clients as ground truth to establish trust before any interaction with your application.

That concludes our confidential bookstore transformation. Your books have never been safer!

Let's revise the service mesh principles here:

• Confidential computing introduces a distinct set of challenges that are critical, uniform across your stack, and should be solved decoupled from your application code.
• Services meshes like OSM have a rich feature set for the already large number of challenges with modern server-side software.
• Having all of them add features for confidential computing is tedious, redundant, and considering their implementation non-trivial.
• Instead, we believe that following the cloud native philosophy with a modular approach is the key to success.

→ Decoupling the confidential control plane from your service mesh control plane by using MarbleRun.

Summary

We have seen how MarbleRun augments OSM (and other service meshes) to solve the daunting challenges of modern confidential server-side software. We made the bookstore demo fully confidential in a few simple steps using our modular toolset of EGo, EdgelessDB, and MarbleRun. We strongly believe in the cloud native philosophy and see immense value of bridging the gap to confidential computing, making the cloud a safer place for your data. Hopefully, this story got you excited to join us on this mission! If you are building things with confidential computing or are about to get started, please get in touch! We are eager to hear from you 😊

For a full hands-on walk-through of the confidential bookstore demo please see our GitHub repository.