How I build my own kubernetes cluster part. 1

Jun 18, 2023

As part of trying to learn more about Kubernetes, I figured it would be fun to create my own hosting, expose it to the internet and in the process gain some knowledge and experience.

In this post, I'll share tips and cover the entire cluster setup process. I'll also show how to automate deployment and configuration using tools such as Ansible, Cloud-init and Argo CD.

Before starting, let me explain a few important concepts here.

Container

Container is a lightweight and portable unit that packages together an application and its dependencies, libraries, and configuration files needed to run reliably and consistently across different computing environments. Containers provide a way to isolate applications from the underlying system, ensuring that they work the same way regardless of where they are deployed.

One of the most popular containerization technologies is Docker, which allows developers to create, distribute, and run containers easily

Why kubernetes?

The primary purpose of Kubernetes is to simplify the management of containerized applications in a scalable and resilient manner. It allows developers to deploy and manage containers without needing to worry about the underlying infrastructure details. Kubernetes abstracts away the complexities of handling multiple containers, networking, storage, and load balancing, providing a unified API and toolset to manage and automate containerized applications.

In this project we will be using k3s which is lightweight Kubernetes distribution designed by Rancher Labs. It is trimmed of unnecessary cloud integrations and various kinds of drivers. Unlike full Kubernetes it is contained in a single binary complete with all the necessary add-ons such as containerd, flannel, CoreDNS or traefik.

More about k3s: https://k3s.io/ and Kuberentes: https://kubernetes.io/.

Hardware

This is the list of hardware I used.

3x Raspberry Pi 4B

1x Cluster case

3x 0,25m Patch cords

3x Samsung FIT Plus 32GB Usb disk (used for operating system)

2x Samsung FIT Plus 64GB Usb disk (used as a storage for pods)

3x Heat sink with pwm controlled fan

1x TP-Link 8-port switch

Installation

I choose Ubuntu Server 64-bit As the operating system for my cluster. To flash up micro SD cards I used Raspberry pi imager.

Next I modify user-data and network-config files within /boot directory. that are part of the cloud-init standard.

user-data file:

This file enable remote access with the SSH key, set the hostname and specify a new user on each node. To get a ssh keys pair use ssh-keygen command

network-config:

After a boot It will assign a static ip address for each Pi.

To make it easier to access nodes from my computer, I added below configuration to my ~/.ssh/config file:

Ansible

To make the configuration faster I'm going to use ansible. You can find more about ansible at: https://www.ansible.com/

Let's create our inventory.yml file:

This file is used to define host groups that Ansible will manage.

all: section defines all host groups.

Inside children: groups are defined, in this case master and worker.

Each group (master and worker) contains hosts to be managed by Ansible. Each host has assigned attributes, such as its name (pi1, pi2, pi3) and IP address (ansible_host).

Next I'll setup ansible.cfg file:

private_key_file = ~/.ssh/rpi: Specifies the path to the private SSH key that will be used for authentication on remote hosts.

remote_user = pi: Enables privilege escalation. By default, Ansible will attempt to use the sudo option to execute commands with administrator privileges on remote hosts.

Lastly we are going to check if Ansible is working fine and can connect to all nodes. To do that you can use ping module specified by -m flag

Os setting

Updating system:

If you don't plan to use snap, you can just remove it to get some free resources

Since the Raspberry PIs in the cluster are configured as a headless server, reserved GPU Memory can be set to lowest possible (16M). To set gpu memory edit boot/firmware/config.txt by adding at the end:

Get the vxlan support. Starting with Ubuntu 21.10, vxlan support on Raspberry Pi has been moved into a separate kernel module. https://docs.k3s.io/installation/requirements?os=pi

Enable cgroups:

Reboot the Raspberry Pi

Fan control

Due to the fact that my cluster, will run for many days, I decided to purchase more robust cooling heat sink with a fan. The aim is to reduce throthling and increase the life span of the PI’s.

I wrote a simple program in python which sets the speed of the fans based on the current temperature:

We need to install required python libraries.

To keep the program running even after a reboot you can create systemd service

Upload program to server:

Upload the service in similar manner:

Enable service:

Installing kubernetes

Control plane (pi1)

To install k3s on the master node execute the following command:

--write-kubeconfig-mode 644: set the file permission mode for the kubeconfig file to 644.

--disable servicelb: This disables the servicelb, which is used for load balancing.

--node-taint CriticalAddonsOnly=true:NoExecute: This adds a taint to the node, indicating that only critical addons should be scheduled on it.

--disable traefik: This disables the traefik , which is an HTTP reverse proxy and load balancer. Traefik will be installed from helm chart.

--disable local-storage: This disables the local storage provisioner. Longhorn will be used instead

After installation obtain node token:

Workers

We will use ansible to join rest nodes to our cluster. Replace <node token> with a node token previously obtained:

Now wait for nodes to join the cluster:

Kubeconfig

We want to copy kubeconfig file from cluster to communicate with the Kubernetes cluster's API server from our computer:

Edit kubeconfig on computer

Networking

In K3s, key network components such as CoreDNS, Traefik Ingress Controller, Flannel and ServiceLB are pre-configured by default.

MetalLB

For bare-metal clusters, Kubernetes does not offer network load balancer implementations (LoadBalancer services) by default. However other options are available, such as NodePort and externalIPs, but both of these options have significant downsides for production use, which makes bare-metal clusters second-class citizens in the Kubernetes ecosystem.

MetalLB addresses these problems by offering a network load balancer implementation that integrates with standard network equipment. This allows external services in bare-metal clusters to run as efficiently as in cloud clusters.

How MetalLB works

MetalLB can work in two modes, BGP and Layer 2. In layer 2 mode, a node advertises a service to the local network, appearing to the network as a machine with multiple IP addresses. The main benefit of using layer 2 mode in building a Kubernetes cluster is its universality, as it operates on any Ethernet network without the need for special hardware or advanced routers.

In BGP mode, each node in your cluster establishes a BGP peering session with your network routers, and uses that peering session to advertise the IPs of external cluster services. More About metalb: https://metallb.universe.tf/

MetalLB installation

Install metalb using helm:

You need to deploy the following manifest as MetalLB requires an IP range to use for external IPs:

Now services that use LoadBalancer should have an external IP assigned to them:

Traefik

Traefik will be used as Ingress Controller for the cluster. Normally traefik is bundled with k3s by default, but I disabled It during installation, so it can be installed manually to have full control over the version and its initial configuration.

Traefik's purpose is to handle network traffic by routing HTTP/HTTPS requests to the appropriate services and exposing them to the outside world through domain

Instalation

Add helm repo, get the latest charts from the repository and create namespace:

Create traefik-values.yml file:

additionalArguments: To avoid mixing Traefik access logs with application logs, we will change the default configuration to write logs to a specific file (/data/access.log) using the --accesslog.filepath parameter.

additionalContainers: Defines additional containers to be run along with the main Traefik container. This container will print access.log file to stdout but not missing it with the rest of logs.

volumeMounts: Specifies where the /data directory should be mounted inside the container. This is required because Traefik will write access logs to the /data/access.log file.

service: Assign a static IP from MetalLB pool

providers: Enable cross-namespaces references in IngressRoute resources

metrics: Create metric service, to enable access to Prometheus metrics

Install Traefik:

Basic authentication

We will later need to secure access to our kubernetes apps exposed by traefik (longhorn ui, prometheus).

The Traefik BasicAuth middleware can be utilized for basic HTTP authentication:

Back to blog