Kubernetes on Ubuntu 20.04
Install K8s
1# remove docker if installed via snap
2sudo snap remove docker
3
4sudo apt install -y docker.io
5sudo apt install -y apt-transport-https curl
6
7# switch to systemd
8sudo tee /etc/docker/daemon.json <<EOF
9{
10 "exec-opts": ["native.cgroupdriver=systemd"],
11 "log-driver": "json-file",
12 "log-opts": {
13 "max-size": "100m"
14 },
15 "storage-driver": "overlay2"
16}
17EOF
18
19## Enable Docker Service
20sudo systemctl enable docker.service
21
22sudo systemctl stop docker
23sudo systemctl start docker
24
25## Disable Swap
26sudo swapoff -a
27sudo sed -i '$ d' /etc/fstab
28
29
30## Add Kubernetes signing key, as of this writing xenial is latest
31curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add
32sudo apt-add-repository "deb http://apt.kubernetes.io/ kubernetes-xenial main"
33
34## install kubernetes
35sudo apt install -y kubeadm=1.19.5-00 kubelet=1.19.5-00 kubectl=1.19.5-00 kubernetes-cni=0.8.7-00
36sudo apt-mark hold kubeadm=1.19.5-00 kubelet=1.19.5-00 kubectl=1.19.5-00 kubernetes-cni=0.8.7-00
37sudo apt-mark showhold
38## only on master
39sudo kubeadm init --pod-network-cidr=10.244.0.0/16
40
41 ## -- or on worker -- ##
42
43## use join command as printed by init
44sudo kubeadm join 192.168.1.44:6443 --token 8n8r48.7n4wdkt42nw4j436 \
45 --discovery-token-ca-cert-hash sha256:89b472970d7a3332559b06a01ddbd1f341bc8e4261ad98aa07878dda3ba0e411
46
47###### stop here if this is a worker node
48
49## setup local kube config
50
51mkdir -p $HOME/.kube
52sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
53sudo chown $(id -u):$(id -g) $HOME/.kube/config
54
55## deploy pod networking
56kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml
57kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/k8s-manifests/kube-flannel-rbac.yml
58
59## install metric server
60kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/download/v0.3.7/components.yaml
POST Setup
install MetalLB
1kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.5/manifests/namespace.yaml
2kubectl apply -f https://raw.githubusercontent.com/metallb/metallb/v0.9.5/manifests/metallb.yaml
3# On first install only
4kubectl create secret generic -n metallb-system memberlist --from-literal=secretkey="$(openssl rand -base64 128)"
Create MetalLB config
1apiVersion: v1
2kind: ConfigMap
3metadata:
4 namespace: metallb-system
5 name: config
6data:
7 config: |
8 address-pools:
9 - name: default
10 protocol: layer2
11 addresses:
12 - 192.168.1.200-192.168.1.220
nfs provisioner
install nfs client on all nodes with
1sudo apt-get install -y nfs-common
Install Provisioner
1sudo snap install helm --classic
2helm repo add stable https://charts.helm.sh/stable
3
4helm repo update
5
6helm install nfs stable/nfs-client-provisioner --set nfs.server=192.168.1.87 --set nfs.path=/mnt/nfs_share --set storageClass.defaultClass=true
concourse
1helm repo add concourse https://concourse-charts.storage.googleapis.com/
2
3helm install concourse concourse/concourse
4
5kubectl expose deployment concourse-web --target-port=8080 --port=80 --type=LoadBalancer --name lb-concourse
6
7helm install concourse concourse/concourse --set web.service.api.type=LoadBalancer --set concourse.web.externalUrl=http://concourse.ellin.net --set concourse.web.bindPort=80 --set worker.persistence.enabled=false --set postgresql.persistence.enabled=false
ArgoCD
1kubectl create namespace argocd
2
3kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml
4
5kubectl patch svc argocd-server -n argocd -p '{"spec": {"type": "LoadBalancer"}}'
6
7# get the password
8kubectl get pods -n argocd -l app.kubernetes.io/name=argocd-server -o name | cut -d'/' -f 2
Misc Linux options
resize partition
You can’t resize a mounted filesystem with parted and resize2fs won’t resize the underlying partition. The workaround is a bit tricky and you have to be careful to keep the start cylinder the same when doing this. What you do is, in fdisk you delete the partition and recreate it with a larger size making sure you keep the start location (cylinder) the same. The example below illustrates this:
1[root@temeria ~] fdisk /dev/sda
expand the partition in fdisk by deleting the partition, create a new one using the same starting cylinder,
Pertinent information marked with <------
1WARNING: DOS-compatible mode is deprecated. It’s strongly recommended to switch off the mode (command ‘c’) and change display units to sectors (command ‘u’).
2Command (m for help): p
3Disk /dev/sda: 25.8 GB, 25769803776 bytes
4255 heads, 63 sectors/track, 3133 cylinders
5Units = cylinders of 16065 * 512 = 8225280 bytes
6Sector size (logical/physical): 512 bytes / 512 bytes
7I/O size (minimum/optimal): 512 bytes / 512 bytes
8Disk identifier: 0x00073409
9Device Boot Start End Blocks Id System
10/dev/sda1 * 1 39 307200 83 Linux <------ Starts at 39
11Partition 1 does not end on cylinder boundary.
12/dev/sda2 39 2097 16534528 83 Linux
13Command (m for help): d <------ delete the original partition
14Partition number (1-4): 2
15Command (m for help): n <------ new partition
16Command action
17e extended
18p primary partition (1-4)
19p <------ primary
20Partition number (1-4): 2 <------ usually 2
21First cylinder (39-3133, default 39):
22Using default value 39 <------ Starting positiono of or
23Last cylinder, +cylinders or +size{K,M,G} (39-3133, default 3133):
24Using default value 3133 <------ default is full disk
25Command (m for help): p
26Disk /dev/sda: 25.8 GB, 25769803776 bytes
27255 heads, 63 sectors/track, 3133 cylinders
28Units = cylinders of 16065 * 512 = 8225280 bytes
29Sector size (logical/physical): 512 bytes / 512 bytes
30I/O size (minimum/optimal): 512 bytes / 512 bytes
31Disk identifier: 0x00073409
32Device Boot Start End Blocks Id System
33/dev/sda1 * 1 39 307200 83 Linux
34Partition 1 does not end on cylinder boundary.
35/dev/sda2 39 3133 24857598+ 83 Linux
36Command (m for help): w <------ write changes
37The partition table has been altered! Calling ioctl() to re-read partition table. WARNING: Re-reading the partition table failed with error 16: Device or resource busy. The kernel still uses the old table. The new table will be used at the next reboot or after you run partprobe(8) or kpartx(8)
38Syncing disks.
Reboot the system to ensure the partition table is reread.
Resize the filesystem
This is perhaps the simplest step. Simply execute the resize2fs command with your partition as an argument.
1[root@temeria ~] resize2fs /dev/sda2
Reset IpTables after kubeadm reset
If you run kubeadm reset on the master you may need to reset iptables before running kubeadm again.
1iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X
Enable system trust on Ubuntu
1mkdir /usr/local/share/ca-certificates/k8s
2
3sudo cp file.crt /usr/local/share/ca-certificates/k8s
4
5sudo update-ca-certificates
file copeied to /etc/ssl/certs
Recreating the Join Command
To rebuild join command
1jeff@kube-0:~$ kubeadm token generate
2
3jeff@kube-0:~$ kubeadm token create <token> --print-join-command --ttl=0
retrieve cert
1openssl x509 -in /etc/kubernetes/pki/ca.crt -pubkey -noout |
2openssl pkey -pubin -outform DER |
3openssl dgst -sha256
nfs client
1sudo apt-get install nfs-common
upgrade K8s
Do on master, then repeat on each worker
- Upgrade Kubeadm
1apt-mark unhold kubeadm && \
2apt-get update && apt-get install -y kubeadm=1.19.3-00 && \
3apt-mark hold kubeadm
- Drain the Node
1kubectl drain kube-2 --ignore-daemonsets
- Upgrade the Node.
1sudo kubeadm upgrade node
- Upgrade the Kubelet
1apt-mark unhold kubelet kubectl && \
2apt-get update && apt-get install -y kubelet=1.19.3-00 kubectl=1.19.3-00 && \
3apt-mark hold kubelet kubectl
4
5sudo systemctl daemon-reload
6sudo systemctl restart kubelet
- Uncorden the node
1 kubectl uncordon kube-2