K8s Series 24: Calico Primary (Monitoring)

“This is the 12th day of my participation in the Gwen Challenge in November. See details of the event: The Last Gwen Challenge 2021”.

preface

Calico officially provides Prometheus monitoring and detailed metrics

Since we installed Calico by default and did not enable Typha components, there are only two components running in the Kubernetes cluster: Felix and Kube-Controlles

Felix detailed indicators: docs.projectcalico.org/reference/f…

Kube – controlles the detailed indicators: docs.projectcalico.org/reference/k…

Calico component configuration

Although Prometheus monitoring is officially provided, it is disabled by default and must be manually enabled, and an endpoint must be provided for Prometheus to pull monitoring data

Felix configuration

Enable the Prometheus indicator for Felix

calicoctl patch felixConfiguration default  --patch '{"spec":{"prometheusMetricsEnabled": true}}'
Copy the code

Create the Felix indicator endpoint

kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: felix-metrics-svc
  namespace: kube-system
spec:
  selector:
    k8s-app: calico-node
  ports:
  - port: 9091
    targetPort: 9091
EOF
Copy the code

Kube – controlles configuration

Kube-controlles Prometheus indicator is enabled by default, need not change, to change its monitoring port, you can run the following command, if the port is 0, it is disabled

# Default port monitoring in9094
calicoctl patch kubecontrollersconfiguration default  --patch '{"spec":{"prometheusMetricsPort": 9094}}'
Copy the code

Create kuBE-Controlles metric endpoints

kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: kube-controllers-metrics-svc
  namespace: kube-system
spec:
  selector:
    k8s-app: calico-kube-controllers
  ports:
  - port: 9094
    targetPort: 9094
EOF
Copy the code

Once the services for both components are successfully created, take a look

Prometheus installation and configuration

Create service accounts and permissions before installing Prometheus

Create a namespace

Create a separate command space for monitoring

kubectl apply -f -<<EOF
apiVersion: v1
kind: Namespace
metadata:
  name: calico-monitoring
  labels:
    app:  ns-calico-monitoring
    role: monitoring
EOF
Copy the code

Creating a Service Account

Create an account with the ability to collect data from Calico, and then grant permission to do so

The following configuration consists of creating roles, creating accounts, and binding role accounts

kubectl apply -f - <<EOF
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: calico-prometheus-user
rules:
- apiGroups: [""]
  resources:
  - endpoints
  - services
  - pods
  verbs: ["get"."list"."watch"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: calico-prometheus-user
  namespace: calico-monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: calico-prometheus-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: calico-prometheus-user
subjects:
- kind: ServiceAccount
  name: calico-prometheus-user
  namespace: calico-monitoring
EOF
Copy the code

Prometheus configuration file

Create a configuration file for Prometheus, and if you have installed Prometheus in binary form, you should find that the following configuration is almost the same. If you want to modify the configuration later, you can edit it directly

kubectl apply -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: calico-monitoring
data:
  prometheus.yml: |-
    global:
      scrape_interval:   15s
      external_labels:
        monitor: 'tutorial-monitor'
    scrape_configs:
    - job_name: 'prometheus'
      scrape_interval: 5s
      static_configs:
      - targets: ['localhost:9090']
    - job_name: 'felix_metrics'
      scrape_interval: 5s
      scheme: http
      kubernetes_sd_configs:
      - role: endpoints
      relabel_configs:
      - source_labels: [__meta_kubernetes_service_name]
        regex: felix-metrics-svc
        replacement: $1
        action: keep
    - job_name: 'typha_metrics'
      scrape_interval: 5s
      scheme: http
      kubernetes_sd_configs:
      - role: endpoints
      relabel_configs:
      - source_labels: [__meta_kubernetes_service_name]
        regex: typha-metrics-svc
        replacement: $1
        action: keep
    - job_name: 'kube_controllers_metrics'
      scrape_interval: 5s
      scheme: http
      kubernetes_sd_configs:
      - role: endpoints
      relabel_configs:
      - source_labels: [__meta_kubernetes_service_name]
        regex: kube-controllers-metrics-svc
        replacement: $1
        action: keep
EOF
Copy the code

Install the Prometheus

After the previous steps are successful, perform the following installation steps

kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: prometheus-pod
  namespace: calico-monitoring
  labels:
    app: prometheus-pod
    role: monitoring
spec:
  serviceAccountName: calico-prometheus-user
  containers:
  - name: prometheus-pod
    image: prom/prometheus
    resources:
      limits:
        memory: "128Mi"
        cpu: "500m"
    volumeMounts:
    - name: config-volume
      mountPath: /etc/prometheus/prometheus.yml
      subPath: prometheus.yml
    ports:
    - containerPort: 9090
  volumes:
  - name: config-volume
    configMap:
      name: prometheus-config
EOF
Copy the code

Check the installation progress. If Running is displayed, the installation is complete

kubectl get pods prometheus-pod -n calico-monitoring
Copy the code

Visit Prometheus

Since we did not create a Service for Promethesu, port forwarding is used here to verify that Prometheus has obtained calico data

kubectl port-forward --address 0.0. 0. 0 pod/prometheus-pod 9090:9090 -n calico-monitoring
Copy the code

Access http://ip:9090 port

Grafana installation configuration

Before configuring Grafana, you need to declare Prometheus access for easy access to data display charts

Create Prometheus Service

kubectl apply -f - <<EOF
apiVersion: v1
kind: Service
metadata:
  name: prometheus-dashboard-svc
  namespace: calico-monitoring
spec:
  selector:
      app:  prometheus-pod
      role: monitoring
  ports:
  - port: 9090
    targetPort: 9090
EOF
Copy the code

Create the Grafana configuration

Creates the type, address, port, and connection mode for grafana to connect to the database

kubectl apply -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-config
  namespace: calico-monitoring
data:
  prometheus.yaml: |-
    {
        "apiVersion": 1."datasources": [{"access":"proxy"."editable": true."name": "calico-demo-prometheus"."orgId": 1."type": "prometheus"."url": "http://prometheus-dashboard-svc.calico-monitoring.svc:9090"."version": 1
            }
        ]
    }
EOF
Copy the code

Felix dashboard configuration

kubectl apply -f https://docs.projectcalico.org/manifests/grafana-dashboards.yaml
Copy the code

Install Grafana

Applying the following configuration directly will download the latest image from grafana

kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: grafana-pod
  namespace: calico-monitoring
  labels:
    app:  grafana-pod
    role: monitoring
spec:
  containers:
  - name: grafana-pod
    image: grafana/grafana:latest
    resources:
      limits:
        memory: "128Mi"
        cpu: "500m"
    volumeMounts:
    - name: grafana-config-volume
      mountPath: /etc/grafana/provisioning/datasources
    - name: grafana-dashboards-volume
      mountPath: /etc/grafana/provisioning/dashboards
    - name: grafana-storage-volume
      mountPath: /var/lib/grafana
    ports:
    - containerPort: 3000
  volumes:
  - name: grafana-storage-volume
    emptyDir: {}
  - name: grafana-config-volume
    configMap:
      name: grafana-config
  - name: grafana-dashboards-volume
    configMap:
      name: grafana-dashboards-config
EOF
Copy the code

Visit grafana

Because the write Service is not configured temporarily, the port is first accessed to verify whether the monitoring is normal

kubectl port-forward --address 0.0. 0. 0 pod/grafana-pod 3000:3000 -n calico-monitoring
Copy the code

Visit http://IP:3000 to access Grafana’s web-UI login page. The default account password is admin/admin

After a successful login, the system prompts you to change the password or skip the login. Change the password in the subsequent Settings

Login good look, is nothing, need to visit this address: http://ip:3000/d/calico-felix-dashboard/felix-dashboard-calico? orgId=1

The Dashborad provided by Calico will be opened. Click the star here and you will find the face plate on the home page later

Create a Service

Create a NodePort Service directly using the Expose command

Kubectl expose Pod grafana-pod --port=3000 --target-port=3000Kubectl get SVC -n calico-monitoring --type=NodePort -n calico-monitoringCopy the code

To open grafana, access the cluster node IP +30538 port

uninstall

If you feel that the monitoring system occupies cluster resources, you can run the following command to delete the monitoring system

kubectl delete service felix-metrics-svc -n kube-system
kubectl delete service typha-metrics-svc -n kube-system
kubectl delete service kube-controllers-metrics-svc -n kube-system
kubectl delete namespace calico-monitoring
kubectl delete ClusterRole calico-prometheus-user
kubectl delete clusterrolebinding calico-prometheus-user

kubectl delete namespace calico-monitoring
Copy the code