How to Set Up Kubernetes Performance Monitoring with Prometheus and Grafana

Monitoring the performance of your Kubernetes cluster is crucial for maintaining a healthy and efficient environment. Prometheus and Grafana are popular tools for this purpose. This guide will walk you through the steps to set up performance monitoring for a Kubernetes cluster using Minikube, but the steps can be applied to any Kubernetes cluster.

Prerequisites

Before you start, ensure you have the following installed:

• Kubernetes cluster (Minikube,Kind, EKS, GKE, AKS, etc.)
• kubectl
• Helm

Step 1: Minikube (or Ensure Your Kubernetes Cluster is Running)

If you’re using Minikube, start your Minikube cluster:

minikube start

For other Kubernetes clusters, make sure they are running and kubectl is configured to interact with them.

Step 2: Install Prometheus Operator

1. Add the Helm repository for Prometheus:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts 
helm repo update

2. Install the Prometheus Operator using Helm:

helm install prometheus prometheus-community/kube-prometheus-stack --namespace monitoring --create-namespace

3. Verify the Prometheus components are deployed:

kubectl get pods -n monitoring -l "release=prometheus"

Step 3: Install Grafana

1. Add the Helm repository for Grafana:

helm repo add grafana https://grafana.github.io/helm-charts 
helm repo update

2. Install Grafana using Helm:

helm install grafana grafana/grafana --namespace monitoring

3. Verify Grafana is deployed:

kubectl get pods -n monitoring -l "release=grafana"

Step 4: Access Grafana

1. Get the Grafana admin password:

kubectl get secret --namespace monitoring grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

2. Expose Grafana service:

• For Minikube:

minikube service grafana -n monitoring --url

• For other clusters, you might need to set up port forwarding:

kubectl port-forward svc/grafana 3000:80 -n monitoring

3. Open Grafana in your browser:

Navigate to the URL provided by the minikube service command or localhost:3000 if using port forwarding.

Step 5: Configure Prometheus Data Source in Grafana

1. Log in to Grafana:

Open your browser and go to the URL provided, then log in with admin as the username and the password you retrieved earlier.

2. Add Prometheus Data Source:

• Go to Configuration > Data Sources.
• Click Add data source.
• Select Prometheus.

3. Enter the Prometheus URL:

Use the internal DNS name for the Prometheus service:

http://prometheus-kube-prometheus-prometheus.monitoring.svc.cluster.local:9090

4. Save and Test:

Click Save & Test to ensure Grafana can access Prometheus. You should see a green message indicating the data source is working.

Step 6: Import Kubernetes Monitoring Dashboards into Grafana

Import NewDashboard:

• Go to Create > Import.
• Use the following dashboard IDs to import Kubernetes monitoring dashboards:
• Kubernetes cluster monitoring (ID: 315)
• Kubernetes monitoring (ID: 1471)
• Kubernetes / Prometheus (ID: 6417)
• Prometheus 2.0 Overview (ID: 3662)

Enter one of the IDs and click Load.

Adjust any required settings and click Import.

Ex :- Prometheus 2.0 Overview (ID: 3662)

Step 7: Monitoring Key Metrics

1. Cluster Metrics:

• CPU Usage:

sum(rate(container_cpu_usage_seconds_total{image!=""}[5m])) by (instance

• Memory Usage:

sum(container_memory_usage_bytes{image!=""}) by (instance)

2. Pod Metrics:

• Pod Restarts:

sum(rate(kube_pod_container_status_restarts_total[5m])) by (pod)

Step 8: Configure Alerts in Prometheus

1. Create Alerting Rules:

Create a YAML file for Prometheus alerting rules (e.g., alert-rules.yaml):

groups:
- name: example
  rules:
  - alert: HighCPUUsage
    expr: sum(rate(container_cpu_usage_seconds_total{image!=""}[5m])) by (instance) > 0.5
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High CPU usage detected"
      description: "The CPU usage is above 50% for more than 5 minutes."

2. Apply the Alerting Rules:

kubectl apply -f alert-rules.yaml

3. Verify Alerting Rules:

Ensure the alerting rules are applied correctly by checking the Prometheus UI or using kubectl to describe the Prometheus resource:

kubectl describe prometheus -n monitoring

Conclusion

By following these steps, you have set up a comprehensive monitoring solution for your Kubernetes cluster using Prometheus and Grafana. This setup will help you keep an eye on the performance and health of your cluster, enabling you to react promptly to any issues.