Introduction

While an organization is gradually moving the development of "monolithic" applications towards distributed microservices, several new challenges are emerging. We keep hearing about how can microservices ("workloads") should safely discover, authenticate and connect on potentially unreliable networks.

Resolving this issue by workload, although it is possible, is unmanageable for a software developer and increasingly difficult to manage with more workloads. Instead, large and small organizations are starting to use proxies (such as  Envoy) to manage discovery, authentication, and encryption for a workload. When workloads connect in this manner, the resulting design model is popularly described as a service mesh.

The issue arises when the services need to communicate securely. To do so the services authenticate themselves by generating an X.509 certificate associated with a private key. Doing this manually is sufficient for simple scenarios, but it's not always scalable.

One great way to remove this overhead and scale the security issue will be to use SPIRE, which provides fine-grained, dynamic workload identity management. SPIRE provides the control plane to provision SPIFFE IDs to the workloads. These SPIFFE IDs can then be used for policy authorization. It enables teams to define and test policies for workloads operating on a variety of infrastructure types, including bare metal, public cloud (such as GCP), and container platforms (like Kubernetes). We will be coupling CIilium with SPIRE to demonstrate this scenario.

The Scenario Setup

The scenario's purpose is to upgrade the connection between carts and front-end pods from HTTP to HTTPS. To enhance the connection, a Cilium Network Policy (CNP) will be used.

To try this, we'll clone the cilium-spire-tutorials repo from the KubeArmor repository. We use a GKE cluster and the sock-shop as our sample application.

  1. Create a cluster under the GCP project
#!/bin/bash
export NAME="test-$RANDOM" 
gcloud container clusters create "${NAME}" --zone us-west2-a --image-type=UBUNTU
gcloud container clusters get-credentials "${NAME}" --zone us-west2-a

2. Deploy the sample application

kubectl apply -f https://raw.githubusercontent.com/kubearmor/KubeArmor/main/examples/sock-shop/sock-shop-deployment.yaml

3.Deploy manifest (cilium-control-plane + spire-control-plane + dependencies).

kubectl apply -f https://raw.githubusercontent.com/accuknox/microservices-demo/main/cilium-spire/cilium-gke.yaml \
              -f https://raw.githubusercontent.com/accuknox/microservices-demo/main/cilium-spire/spire.yaml
Note: Check the status of all the pods. The spire-control plane (spire-agent and spire-server) should be running as well as the cilium-control plane.
kube-system   cilium-26c86                                           1/1     Running   0          16h
kube-system   cilium-jdx87                                           1/1     Running   0          16h
kube-system   cilium-node-init-2zrjf                                 1/1     Running   0          16h
kube-system   cilium-node-init-7xdl9                                 1/1     Running   0          16h
kube-system   cilium-node-init-lk4kz                                 1/1     Running   0          16h
kube-system   cilium-operator-7c97784647-nk86q                       1/1     Running   0          16h
kube-system   cilium-operator-7c97784647-xrhd2                       1/1     Running   0          16h
kube-system   cilium-spsm8                                           1/1     Running   0          16h
spire         spire-agent-2glk2                                      0/1     Evicted   0          13m
spire         spire-agent-bs679                                      1/1     Running   0          123m
spire         spire-agent-lv66b                                      1/1     Running   0          5h32m
spire         spire-server-0                                         1/1     Running   0          38m

Configuring SPIRE

A trust domain represents workloads that implicitly trust each other's identity papers, and a SPIRE Server is bound to it. I'm going to use example.org as an example.

SPIRE employs an attestation policy to determine which identity to issue to a requesting workload for workloads within a trusted domain. These policies are usually characterized in terms of

  1. The infrastructure that hosts the workload (node attestation), and
  2. The OS process 'hooks' that identify the workload that is operating on it (process attestation).

For this example, we'll use the following commands on the SPIRE Server to register the workloads (replace the selector labels and SPIFFE ID as appropriate):

kubectl exec -n spire spire-server-0 -- \
    /opt/spire/bin/spire-server entry create \
    -spiffeID spiffe://example.org/ciliumagent \
    -parentID spiffe://example.org/ns/spire/sa/spire-agent \
    -selector unix:uid:0

# agent
kubectl exec -n spire spire-server-0 -- \
    /opt/spire/bin/spire-server entry create \
    -node  \
    -spiffeID spiffe://example.org/ns/spire/sa/spire-agent \
    -selector k8s_sat:cluster:demo-cluster \
    -selector k8s_sat:agent_ns:spire \
    -selector k8s_sat:agent_sa:spire-agent
    
# frontend pod
kubectl exec -n spire spire-server-0 -- \
    /opt/spire/bin/spire-server entry create \
    -spiffeID spiffe://example.org/frontend \
    -parentID spiffe://example.org/ns/spire/sa/spire-agent \
    -selector k8s:pod-label:name:front-end
    
# carts pod
kubectl exec -n spire spire-server-0 -- \
    /opt/spire/bin/spire-server entry create \
    -spiffeID spiffe://example.org/cartservice \
    -parentID spiffe://example.org/ns/spire/sa/spire-agent \
    -selector k8s:pod-label:name:orders

We’ll check the endpoint list in cilium to make sure that a SPIFFE ID is assigned to the pods

ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                                  IPv6   IPv4          STATUS   
           ENFORCEMENT        ENFORCEMENT                                                                                                                    
479        Disabled           Disabled          28692      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system          10.16.1.115   ready   
                                                           k8s:io.cilium.k8s.policy.cluster=default                                                                  
                                                           k8s:io.cilium.k8s.policy.serviceaccount=kube-dns                                                          
                                                           k8s:io.kubernetes.pod.namespace=kube-system                                                               
                                                           k8s:k8s-app=kube-dns    
1015       Disabled           Disabled          1886       k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=sock-shop            10.16.1.10    ready   
                                                           k8s:io.cilium.k8s.policy.cluster=default                                                                  
                                                           k8s:io.cilium.k8s.policy.serviceaccount=default                                                           
                                                           k8s:io.kubernetes.pod.namespace=sock-shop                                                                 
                                                           k8s:name=catalogue                                                                                        
                                                           spiffe://example.org/catalogue                                                                            
1478       Disabled           Disabled          12082      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system          10.16.1.127   ready   
                                                           k8s:io.cilium.k8s.policy.cluster=default                                                                  
                                                           k8s:io.cilium.k8s.policy.serviceaccount=konnectivity-agent                                                
                                                           k8s:io.kubernetes.pod.namespace=kube-system                                                               
                                                           k8s:k8s-app=konnectivity-agent                                                                            
1579       Disabled            Disabled          11840     k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=sock-shop            10.16.1.156   ready   
                                                           k8s:io.cilium.k8s.policy.cluster=default                                                                  
                                                           k8s:io.cilium.k8s.policy.serviceaccount=default                                                           
                                                           k8s:io.kubernetes.pod.namespace=sock-shop                                                                 
                                                           k8s:name=front-end                                                                                        
                                                           spiffe://example.org/frontend                                                                            

Perfect! everything is attached nicely. Let's go ahead and create a policy that will upgrade the egress connection from the carts pod from HTTP to HTTPS and does the vice-versa in the front-end pod.

---
apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy
metadata:
  name: frontend-kube-dns
spec:
  endpointSelector:
    matchLabels:
      name: front-end
  egress:
  - toEndpoints:
        - matchLabels:
            io.kubernetes.pod.namespace: kube-system
            k8s-app: kube-dns
    toPorts:
        - ports:
            - port: "53"
              protocol: UDP
          rules:
            dns:
              - matchPattern: "*"
---
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "frontend-upgrade-carts-egress-dns"
spec:
  endpointSelector:
    matchLabels:
      name: front-end
  egress:
  - toPorts:
    - ports:
      - port: "80"
        protocol: "TCP"
      originatingTLS:
        spiffe:
          peerIDs:
            - spiffe://example.org/carts
            - spiffe://example.org/catalogue 
        dstPort: 443
      rules:
        http:
        - {}
---
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "carts-downgrade-frontend-ingress"
spec:
  endpointSelector:
    matchLabels:
      name: carts
  ingress:
  - toPorts:
    - ports:
      - port: "443"
        protocol: "TCP"
      terminatingTLS:
        spiffe:
          peerIDs:
            - spiffe://example.org/front-end
        dstPort: 80
      rules:
        http:
        - {}
---
apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "carts-pod-open-port-80"
spec:
  endpointSelector:
    matchLabels:
      name: carts
  ingress:
  - toPorts:
    - ports:
      - port: "80"
        protocol: "TCP"

Since we have the policy ready let us go ahead and apply it to the cluster view the result

kubectl apply -f policy.yaml -n sock-shop 	
ciliumnetworkpolicy.cilium.io/frontend-kube-dns created
ciliumnetworkpolicy.cilium.io/frontend-upgrade-carts-egress-dns created
ciliumnetworkpolicy.cilium.io/carts-downgrade-frontend-ingress created
ciliumnetworkpolicy.cilium.io/carts-pod-open-port-80 created

Inspect the front-end and carts endpoints and see whether the policies are enforced. Remember that we utilized an ingress policy for pod front-end and an egress policy for pod carts. We'll look for the keyword ENABLED in the appropriate columns.

ENDPOINT   POLICY (ingress)   POLICY (egress)   IDENTITY   LABELS (source:key[=value])                                                  IPv6   IPv4          STATUS   
           ENFORCEMENT        ENFORCEMENT                                                                                                                    
479        Disabled           Disabled          28692      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system          10.16.1.115   ready   
                                                           k8s:io.cilium.k8s.policy.cluster=default                                                                  
                                                           k8s:io.cilium.k8s.policy.serviceaccount=kube-dns                                                          
                                                           k8s:io.kubernetes.pod.namespace=kube-system                                                               
                                                           k8s:k8s-app=kube-dns    
1015       Disabled           Disabled          1886       k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=sock-shop            10.16.1.10    ready   
                                                           k8s:io.cilium.k8s.policy.cluster=default                                                                  
                                                           k8s:io.cilium.k8s.policy.serviceaccount=default                                                           
                                                           k8s:io.kubernetes.pod.namespace=sock-shop                                                                 
                                                           k8s:name=catalogue                                                                                        
                                                           spiffe://example.org/catalogue                                                                            
1478       Disabled           Disabled          12082      k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=kube-system          10.16.1.127   ready   
                                                           k8s:io.cilium.k8s.policy.cluster=default                                                                  
                                                           k8s:io.cilium.k8s.policy.serviceaccount=konnectivity-agent                                                
                                                           k8s:io.kubernetes.pod.namespace=kube-system                                                               
                                                           k8s:k8s-app=konnectivity-agent                                                                            
1579       Enabled            Disabled          11840     k8s:io.cilium.k8s.namespace.labels.kubernetes.io/metadata.name=sock-shop            10.16.1.156   ready   
                                                           k8s:io.cilium.k8s.policy.cluster=default                                                                  
                                                           k8s:io.cilium.k8s.policy.serviceaccount=default                                                           
                                                           k8s:io.kubernetes.pod.namespace=sock-shop                                                                 
                                                           k8s:name=front-end                                                                                        
                                                           spiffe://example.org/frontend                                                                            

Every step is completed and we have attached SPIFFEE IDs to our pods and wrote a cilium network policy to upgrade as well as downgrade the connections on pods carts and front-end. The only thing remaining for us to do is to verify whether the policies are enforced.

Since our dockerized application is immutable, we’ll deploy an Ubuntu pod with the same label as of front-end pod.

kubectl -n sock-shop run -it  frontend-proxy --image=ubuntu --labels="name=front-end"	

We’ll try to initiate a connection to the carts pod via the carts service from the newly deployed frontend-proxy pod.

[email protected]:/# curl http://carts.sock-shop.svc.cluster.local -kv
*   Trying 10.116.21.188:80...
* TCP_NODELAY set
* Connected to carts.sock-shop.svc.cluster.local (10.116.21.188) port 80 (#0)
> GET / HTTP/1.1
> Host: carts.sock-shop.svc.cluster.local
> User-Agent: curl/7.68.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< x-application-context: carts:80
< content-type: application/hal+json;charset=UTF-8
< date: Tue, 28 Dec 2021 09:05:54 GMT
< x-envoy-upstream-service-time: 509
< server: envoy
< transfer-encoding: chunked
< 
{
  "_links" : {
    "items" : {
      "href" : "http://carts.sock-shop.svc.cluster.local/items{?page,size,sort}",
      "templated" : true
    },
    "profile" : {
      "href" : "http://carts.sock-shop.svc.cluster.local/profile"
    }
  }
* Connection #0 to host carts.sock-shop.svc.cluster.local left intact
}

The Validation Phase

In order to validate our hypothesis, we’ll need to collect the traffic that's originating from the frontend-proxy pod and carts pod. Luckily for us, both the frontend-proxy and carts pods are residing on the same node gke-test-28321-default-pool-3956f92a-qz63

kubectl -n sock-shop get pod -o wide
NAME                            READY   STATUS    RESTARTS   AGE     IP             NODE                                        NOMINATED NODE   READINESS GATES
carts-578686fcbc-jw622          1/1     Running   0          6h25m   10.108.2.149   gke-test-28321-default-pool-3956f92a-qz63   <none>           <none>
carts-db-7d7f64c4c7-dqvvv       1/1     Running   0          6h25m   10.108.2.23    gke-test-28321-default-pool-3956f92a-qz63   <none>           <none>
catalogue-8998bcbb8-cdz9z       1/1     Running   0          6h25m   10.108.0.192   gke-test-28321-default-pool-3956f92a-snt2   <none>           <none>
catalogue-db-55464745df-r5vnd   1/1     Running   0          6h25m   10.108.2.224   gke-test-28321-default-pool-3956f92a-qz63   <none>           <none>
front-end-65bc579b99-jh7lk      1/1     Running   0          6h25m   10.108.1.27    gke-test-28321-default-pool-3956f92a-j6cn   <none>           <none>
orders-65996f6d45-vtftg         1/1     Running   0          6h25m   10.108.1.77    gke-test-28321-default-pool-3956f92a-j6cn   <none>           <none>
orders-db-6d957b4499-t2wxf      1/1     Running   0          6h25m   10.108.2.48    gke-test-28321-default-pool-3956f92a-qz63   <none>           <none>
payment-77f89df9b7-7zhmb        1/1     Running   0          6h25m   10.108.2.21    gke-test-28321-default-pool-3956f92a-qz63   <none>           <none>
queue-master-6dddcbbcbb-2znvv   1/1     Running   0          6h25m   10.108.0.60    gke-test-28321-default-pool-3956f92a-snt2   <none>           <none>
rabbitmq-57b5dd7bc-clrl4        2/2     Running   0          6h25m   10.108.2.40    gke-test-28321-default-pool-3956f92a-qz63   <none>           <none>
session-db-864748dffb-5frbl     1/1     Running   0          6h25m   10.108.2.216   gke-test-28321-default-pool-3956f92a-qz63   <none>           <none>
shipping-86f8b9cd97-vsgwv       1/1     Running   0          6h25m   10.108.1.252   gke-test-28321-default-pool-3956f92a-j6cn   <none>           <none>
frontend-proxy                  1/1     Running   0          5h49m   10.108.2.92    gke-test-28321-default-pool-3956f92a-qz63   <none>           <none>
user-7ccf5fc975-ppbzb           1/1     Running   0          6h25m   10.108.2.236   gke-test-28321-default-pool-3956f92a-qz63   <none>           <none>
user-db-6546f9fcfc-qj9c7        1/1     Running   0          6h25m   10.108.1.151   gke-test-28321-default-pool-3956f92a-j6cn   <none>           <none>

We will use tcpdump to capture the packets on every interface and trigger the curl command once again. Once we receive the response we’ll save the tcpdump file and analyze it using Wireshark

SSH into the node and run tcpdump to capture the packets and save them to a dump file using the commands

[email protected]:~$ sudo tcpdump -i any  -w packet.log
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
^C3638 packets captured
4132 packets received by filter
0 packets dropped by kernel

Before decoding the captured packet with Wireshark let's take a look at the IP’s our application use.

kube-system   pod/cilium-5xzwt                                       10.168.0.10   
kube-system   pod/konnectivity-agent-7fbb777cb5-5dgrs                10.108.2.31    
sock-shop     pod/carts-578686fcbc-jw622                             10.108.2.149   
sock-shop     pod/frontend-proxy                                     10.108.2.92    
spire         pod/spire-agent-tc4ct                                  10.168.0.10    
sock-shop     service/carts                                          10.116.21.188   

Conclusion

The use of the mTLS capability via Cilum-SPIRE in the context of Kubernetes is straightforward, as you can see from the above walkthrough. This is partly due to the modified cilium-spire agents and also due to SPIRE taking care of the heavy lifting concerning the workload identities management.

Now you can protect your workloads in minutes using AccuKnox, it is available to protect your Kubernetes and other cloud workloads using Kernel Native Primitives such as AppArmor, SELinux, and eBPF.

Let us know if you are seeking additional guidance in planning your cloud security program.

Read more blogs from Cloud Security Category here.