Load balancing with k8s and kedacore

INTRODUCTION

Kubernetes is a well know container orchestration tool and if you’re reading this article then there’s a strong chance that you’re already familiar with internal workings of this state of the art technology made by google. This article is going to focus mainly on scaling your kubernetes workloads on multiple machines also known as nodes in the kubernetes world. We are going to use kedacore to write our HPAs (Horizontal pod autoscalers) and we are going to see how you can configure these HPAs based on the metrics of your choice.

APPLICATION

You can setup horizontal scaling for any kind of application using kedacore and kubernetes, for the sake of this tutorial, we are going to demonstrate it using a nestjs application. We are going to create a nestJS application with HPA from scratch in the next sections of this article but you can also get the full source code of this demo from https://github.com/A7ALABS/nest-kedacore-demo.

Create a new nest project using the nest-cli

1nest new nest-kedacore-demo

Nest-cli will create a new folder named `nest-kedacore-demo` with a bunch of files in it. The files we are going to focus on are as followings:

1src/app.module.ts
2src/app.controller.ts

Upon creation of the project using nest-cli, next step is to install two modules

1nestjs-prometheus
2prom-client

The nestjs-prometheus with the help of prom-client module will expose a set of metrics which can be monitored by our HPA (kedacore in this case) using prometheus. Prometheus is a metrics scraper which can listen to metrics like CPU usage, memory usage, network usage and a lot more. nestjs-prometheus acts as a bridge between our nestjs app and prometheus.

Install the required modules using the following command

1yarn add prom-client 
2yarn add @willsoto/nestjs-prometheus

Now that we have nestjs-prometheus installed in our nestjs app, we can configure it so it can expose the metrics, the first step is to import this module in `app.module.ts` file using the following import statement:

1import { PrometheusModule } from '@willsoto/nestjs-prometheus';

Once imported, the next step is to include this module in imports section of our `app.module.ts` file located at `nest-kadacore-demo/src/app.module.ts`

1imports: [ PrometheusModule.register(), ],

Your `app.module.ts` file should look something like this after the edits:

1// app.module.ts
2
3import { Module } from '@nestjs/common';
4import { AppController } from './app.controller';
5import { AppService } from './app.service';
6import { PrometheusModule } from '@willsoto/nestjs-prometheus';
7@Module({
8  imports: [ 
9    PrometheusModule.register(), 
10  ],
11  controllers: [AppController],
12  providers: [AppService],
13});
14
15export class AppModule {}

Let’s also add a /health endpoint in our app.controller.ts file so we can check if our app is running well or not, this /health endpoint is also going to be used by kubernetes as a liveliness/readiness probe:

1// app.controller.ts
2
3import { Controller, Get, Req } from '@nestjs/common';
4import { AppService } from './app.service';
5
6@Controller()
7export class AppController {
8  constructor(private readonly appService: AppService) {}
9
10  @Get('/health')
11  async k8s(@Req() req) {
12    return 'Working - 0.0.1'
13  }
14}
15@Get('/health')
16  async k8s(@Req() req: Request) {
17    const eventID = uuidv4();
18    return 'Working - 0.0.1'
19  }

You can now run your nestjs app using the yarn start:dev command and verify that prometheus is exposing all metrics on /metrics route. Go to http://localhost:3000/metrics endpoint and you will be greeted by a list of useful metrics.

DOCKER

As you might already know, we need a container runtime to run an application in kubernetes since kubernetes just orchestrates the containers. We are going to use docker as our container runtime to run our app in the kubernetes system. And as you have already guessed, we are going to need a Dockerfile for it, create a file named Dockerfile in the root directory of the project with the following contents:

1# Dockerfile
2
3FROM ubuntu:20.04
4WORKDIR /app
5
6RUN apt update
7RUN apt install -y tzdata
8
9RUN apt update
10RUN apt -y upgrade
11RUN apt -y install curl
12RUN apt -y install curl dirmngr apt-transport-https lsb-release ca-certificates
13RUN curl -sL https://deb.nodesource.com/setup_16.x | bash -
14RUN apt -y install nodejs
15RUN apt -y  install gcc g++ make
16COPY package.json /app
17RUN npm install
18RUN npm install -g @nestjs/cli
19EXPOSE 3000
20COPY  . /app
21CMD ["nest", "start"]

The Dockerfile is pretty self explanatory - it’s just copying all the files to the /app directory of the docker image and then install the dependencies from package.json file.

NOTE: I am using ubuntu:20.04 as the base image for this application but you can also use the nodejs image if you are looking for a smaller footprint of your application image.

Now that we have the Dockerfile ready, we are going to build a docker image using it. Run the following command from the project directory to build the docker image:

1docker build . -t nest-kedacore-demo

Then confirm if you have this image in your system using the following command:

1docker images

The above command should print a list of all available docker images on your system along with an image named nest-kedacore-demo. We can run containers based on this image that we just built, these containers will then be used to run our nestjs app that we built in the previous section.

KUBERNETES

We have a docker image which contains all our nestjs application files and now is the time to use this docker image to run container(s) in our kubernetes cluster.

Let’s start writing our configuration files, we are going to need the following files to run this container in the k8s cluster:

• deployment.yaml
• service.yaml
• hpa.yaml

Create a new folder inside the root folder of the project and name it kubernetes. Upon creation of this folder, create the above three files in it.

We are going to write the deployment file first, the deployment file is where we are going to use the docker image that we built in the previous section of this article. The deployment file has the following contents:

1# deployment.yaml
2
3apiVersion: apps/v1
4kind: Deployment
5metadata:
6  name: nest-kedacore-demo
7  labels:
8    app: nest-kedacore-demo
9
10spec:
11  replicas: 1
12  selector:
13    matchLabels:
14      app: nest-kedacore-demo
15
16  # template is abstraction over pods
17  # one pod can have multiple containers
18  template:
19    metadata:
20      labels:
21        app: nest-kedacore-demo
22      annotations:
23        prometheus.io/scrape: 'true'
24        prometheus.io/path: /metrics
25        prometheus.io/port: '3000'
26    spec:
27      containers:
28        - name: nest-kedacore-demo
29          livenessProbe:
30            failureThreshold: 10
31            httpGet:
32              path: /health
33              port: 8001
34              scheme: HTTP
35            initialDelaySeconds: 60
36            periodSeconds: 300
37            successThreshold: 1
38            timeoutSeconds: 300
39          readinessProbe:
40            failureThreshold: 10
41            httpGet:
42              path: /health
43              port: 8001
44              scheme: HTTP
45            initialDelaySeconds: 60
46            periodSeconds: 300
47            successThreshold: 1
48            timeoutSeconds: 300
49          env:
50          - name: LEVEL
51            value: "production"
52          image: nest-kedacore-demo:latest
53          resources:
54            limits:
55              cpu: 1000m
56              memory: 1Gi
57            requests:
58              cpu: 1000m
59              memory: 1Gi
60          ports:
61            - containerPort: 3000

There’s one very special section in this deployment.yaml file, and that is the annotations section:

1 annotations:
2        prometheus.io/scrape: 'true'
3        prometheus.io/path: /metrics
4        prometheus.io/port: '3000'

The prometheus.io/scrape: 'true' is telling prometheus to scrape on the /metrics endpoint on port 3000 of the container running in this pod.

Next step is to write our service.yamlfile, this file has the following contents and is self-explanatory:

1# service.yaml
2
3apiVersion: v1
4kind: Service
5metadata:
6  name: nest-kedacore-demo
7spec:
8  type: NodePort
9  selector:
10    app: nest-kedacore-demo
11  ports:
12    - protocol: TCP
13      port: 80
14      targetPort: 3000

The last file that we are going to need is the hpa.yaml

1# hpa.yaml
2
3apiVersion: keda.sh/v1alpha1
4kind: ScaledObject
5metadata:
6  name: nest-kedacore-demo
7  labels:
8    app: nest-kedacore-demo
9spec:
10  maxReplicaCount: 5
11  minReplicaCount: 3
12  pollingInterval: 15
13  scaleTargetRef:
14    name: nest-kedacore-demo
15  triggers:
16    - type: prometheus
17      metadata:
18        serverAddress: http://prometheus-server
19        metricName: cpu_usage
20        query: sum(irate(process_cpu_seconds_total{app="nest-kedacore-demo"}[2m])) * 100
21        threshold: '30'

The hpa.yaml file is basically saying that we need minimum 3 replicas and maximum 5 replicas of our nestjs app. The triggers section of this file creates a scale trigger which polls the prometheus server every 15 seconds and creates a sample over 2 minutes, and based on this sample, if the cpu_usage crosses the defined threshold: 30 then it scales up or scales down if the threshold is below 30.

KEDA

Note that the kind of the component here is ScaledObject which is a custom kind by kedacore and is not a part of kubernetes core components. To run this kind of component in our cluster, we need to install keda in our cluster, you can do so by running the following command on master node of your cluster:

1helm repo add kedacore https://kedacore.github.io/charts
2helm repo update
3helm install keda kedacore/keda --namespace monitoring --create-namespace

PROMETHEUS

Then install prometheus in your cluster:

1helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
2helm install prometheus prometheus-community/prometheus -n monitoring --create-namespace

Note: Considering prometheus and keda are installed in your cluster, this scaledobject by kedacore is going to query prometheus server and get the metrics every 2 minutes based on the trigger we made in the hpa.yaml file.

Now that we have all the kubernetes files ready with us, we can install them all in our kubernetes cluster, run the following command from your master node from the root directory of the project to do so:

1kubectl apply -f kubernetes/

The above command will install the deployment, service and the hpa file in your cluster, you can verify if they were installed or not by the following commands:

1kubectl get deployments
2kubectl get svc
3kubectl get hpa

Now send some load to your deployment after exposing your service via an ingress and see your deployment scale up/down based on the traffic it’s getting.

Thanks for reading!