Skip to content

Deployment

Lets deploy Grafana to read data from Prometheus instances.

This will be simple. Still in our monitoring directory, create new subdirectory called grafana and in it we will create following files:

grafana-pvc.yaml

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: longhorn-grafana-pvc
  namespace: monitoring
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: longhorn
  resources:
    requests:
      storage: 10Gi

This will be our persistent storage, its to keep the dashboards saved. As far as understand Grafana does not keep the data so we don't have to have so much space dedicated for it. ( mine is using like 400MB )

grafana-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: grafana
  name: grafana
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      containers:
      - env: []
        image: grafana/grafana:latest
        name: grafana
        ports:
        - containerPort: 3000
          name: http
        readinessProbe:
          httpGet:
            path: /api/health
            port: http
        resources:
          limits:
            cpu: 200m
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 100Mi
        volumeMounts:
        - mountPath: /var/lib/grafana
          name: grafana-storage
          readOnly: false
      nodeSelector:
        node-type: worker
      securityContext:
        fsGroup: 65534
        runAsNonRoot: true
        runAsUser: 65534
      serviceAccountName: grafana
      volumes:
        - name: grafana-storage
          persistentVolumeClaim:
            claimName: longhorn-grafana-pvc

Fairly standard deployment, I mentioned most of "kinks" I use before like nodeSelector etc...

grafana-serviceAccount.yaml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: grafana
  namespace: monitoring

Just a service account for Grafana

grafana-service.yaml

apiVersion: v1
kind: Service
metadata:
  name: grafana
  namespace: monitoring
spec:
  selector:
    app: grafana
  type: LoadBalancer
  ports:
  - name: http
    port: 3000
    targetPort: http
  loadBalancerIP: 192.168.0.236

Classic for us by now, I'm creating external IP for Grafana, to run on 192.168.0.236 and port 3000

Jump one folder up and apply whole folder.

cd ..
kubectl apply -f grafana/

Check if grafana pod is deployed:

root@control01:/home/ubuntu/grafana# kubectl get pods -n monitoring
NAME                                   READY   STATUS    RESTARTS   AGE
grafana-5b799b4c8c-qzp52               1/1     Running   0          8d
.
.
.

Basic setup

You should be able to connect to the IP of Grafana now.

Default login and password is admin:admin

grafana-ui

Than go down and change you account name, password etc...

grafana-pass

Next we need to define source where Grafana should look for data.

grafana-datasource

Click on Add data source and chose Prometheus, new tab with settings will pop up. Set name for your instance, for example Prometheus-main. This is so we can differentiate sources later. Next important value is URL. If you remember back when we deployed Prometheus file prometheus-service-local.yaml we created ClusterIP and in other file MetalLB IP. You can choose any of them. To check look at the services:

root@control01:/home/ubuntu/grafana# kubectl get services -n monitoring
NAME                  TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)             AGE
.
.
prometheus            ClusterIP      10.43.117.147   <none>          9090/TCP            14d
prometheus-external   LoadBalancer   10.43.49.187    192.168.0.235   9090:30850/TCP      14d
.
.

So in the URL either put IP or NAME so for example using internal ClusterIP enter http://10.43.117.147:9090 should work.

Down click Save & Test it should check and save the data source.

Add another data source, this will be for OpenFaaS ( If you have it ). Same drill as above just check your IP for OpenFaaS Prometheus.

root@control01:/home/ubuntu/grafana# kubectl get services -n openfaas
NAME                TYPE           CLUSTER-IP      EXTERNAL-IP     PORT(S)          AGE
.
.
prometheus          ClusterIP      10.43.238.226   <none>          9090/TCP         35d

So use 10.43.238.226:9090 for URL and name it Prometheus-OpenFaaS, or something that will let you know its OpenFaaS data.

Note

I just noticed that both Prometheus instances have same name for ClusterIP = prometheus therefore I op for IP instead of http://prometheus:9090 as I'm not sure if the internal DNS would not mess something up.

Some Graphs

My final goal is to create my own dashboard with data I wont. But before we get to that we can use already existing collection ( and later on pick and choose what we want from them ).

Click on plus sign and Import.

grafana-import

Next type 8171 into Import via grafana.com

grafana-import

Where did I get the ID ? well here: Grafana Dashboard

Click Load

In next window Name the dashboard if you like, but more important down choose the source, your main Prometheus instance.

grafana-sources

Click Import

Tadaaaa ! your first graphs.. It should take you to them immediately and you can choose data from specific server on the top.

grafana-graph

Here is list of other dashboards that work, mostly, out of the box.

Kubernetes Kubernetes Longhorn OpenFaaS 1 OpenFaaS 2

And thats really it. Next chapter I look into creating my own dashboard but until then I need some drink and chill.


Last update: February 28, 2021

Comments