Monitoring and Alerting on your Kubernetes Cluster with Prometheus and Grafana

Photo by Kevin Ku on Unsplash

Why Are Monitoring and Alerting Important?

Why Prometheus and Grafana?

Prometheus

<metric name>{<label name>=<label value>, …}
Prometheus architecture and ecosystem components. Source

Grafana

Prerequisites for Installation

Install Prometheus and Grafana with Helm

Installing Helm

$ helm repo add stable https://kubernetes-charts.storage.googleapis.com/
$ helm repo update
$ kubectl create ns monitoring

Installing Prometheus

$ helm install prometheus stable/prometheus --namespace monitoring
NAME: prometheus
LAST DEPLOYED: Tue Apr 14 09:22:27 2020
NAMESPACE: monitoring
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
The Prometheus server can be accessed via port 80 on the following DNS name from within your cluster:
prometheus-server.monitoring.svc.cluster.local
[...]

Installing Grafana

$ helm install grafana stable/grafana --namespace monitoring
$ kubectl -n monitoring expose pod grafana-5b74c499c6-kt4bw --type NodePort --name grafana-npservice/grafana-np exposed
$ kubectl -n monitoring get svc grafana-npNAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
grafana-np NodePort 10.111.59.2 <none> 80:30368/TCP,3000:31399/TCP 3h8m
kubectl get secret — namespace monitoring grafana -o jsonpath=”{.data.admin-password}” | base64 — decode ; echoADMINPASSWORD

Alerting on Slack Channel

Create the Slack notification channel

Create and test your custom alert

$ kubectl create ns loadtest
namespace loadtest created
$ kubectl create deployment nginx --image=nginx -n loadtest
deployment.apps/nginx created
$ kubectl -n loadtest scale deployment nginx --replicas=100
deployment.apps/nginx scaled
$ kubectl -n loadtest scale deployment nginx — replicas=1
deployment.apps/nginx scaled

Conclusion

Cloud Architect

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store