Kubernetes

Greek means pilot of the ship. Successor to Borg used inside Google.

Basics

Nodes (prev. Minions) - A machine: Physical or virtual. - Which has K8 installed. - Where containers will be launched by K8s.

Clusters - Set of nodes grouped together. - Disaster Recovery. - Resilience. - Load sharing.

[!IMPORTANT] Cluster also has a special node called Master that controls all other nodes.

Components

K8 installs following things with it:

API Server - kubectl is all we need. For cluster admin tasks, kubeadm utility. - Acts as frontend for K8. - CLI, UI. - Interaction.

etcd key-store - stores all the data used to manage the cluster. - key-value; distributed manner. - ensures there are no conflicts b/w the master.

Scheduler - responsible for distributing load across multiple clusters.

Controller - DR head. - Decides what and how when cluster goes down.

Container Runtime - Used for running container. - Our case -> Docker

kubelet - runs on each node in the cluster. - makes sure containers are running on nodes as expected.

[!NOTE] Namespaces: Think of them as folders that holds a set of objects. (--namespace)

Master vs Worker Nodes

Master has: - kube-apiserver - controller - etcd pair - scheduler

Worker has: - kubelet

[!IMPORTANT] kubectl is an important command for running k8. - kubectl run hello-minikube - kubectl cluster-info - kubectl get nodes - kubectl logs pod_name -c init-container get the name of the init-container by doing describe

Pods

Pods provide access to same storage and networking for the underlying containers. Instead of linking manually and maintaining the connection tables, all the containers inside a pod share same underlying networking.

• Minimal entity that can be managed by kubernetes.
• Typically, Pods are started only through a Deployment because "naked" pods are not rescheduled in case of a failure.
• Each Pod is assigned a unique IP address. Every container in the pod shares the same namespaces, IP and network ports and communicate with each other through localhost (or the configured name of the pod.

Single-Container Pods - Containers are encapsulated inside POD. - Single instance of an application. - One-to-one relationship. - Only has one container of same type. - If load increases, more pods are spun-up.

Multi-Container Pods (less used but still beneficial) - More than one containers inside a pod but of different types. - Like a python and .net (helper) container. - Eg. Container 1 is a web server and container 2 is a Git sync containers sharing the same pod.

[!TIP] In general, the right question to ask yourself when designing Pods is “Will these containers work correctly if they land on different machines?” If the answer is no, a Pod is the correct grouping for the containers. If the answer is yes, using multiple Pods is probably the correct solution.

Commands

• Run a pod using: kubectl run mynginx --image=nginx.
• Get pods: kubectl get pods
• Describe pods: kubectl describe pods
• To get the yaml: kubectl get pods mynginx -o yaml
• Label a pod: kubectl label pods name color=red

Setup

minikube

• MiniKube is scaled down but fully equivalent of K8 for dev purpose.
• First, install kubectl and then minikube.
• Check status of minikube by minikube status.
• Start minikube by: minikube start and then check for kubectl cluster-info and minikube status.

Commands

• minikube dashboard: opens k8 dashboard in browser
• minikube delete: delets a cluster
• minikube ip: shows the currently used IP address
• minikube start: start
• minikube stop: stop
• minikube status: status
• minikube version: shows version
• minikube service name or minikube service name --url: open the service in browser
• minikube pause: pause k8 without impacting deployed apps
• minikube unpause
• minikube delete --all: delete all minikubes clusters

[!NOTE] To log into minikube host, do minikube ssh.

kubectl

To get the nodes running - kubectl get nodes

[!TIP] To get the shortnames for the commands, run kubectl api-resources.

Example 1 (running nginx server)

1) kubectl run mynginx --image=nginx - --image should be a valid docker image - This creates a pod for the container.

2) kubectl get pods - Check the running pods. - The READY column shows the no. of containers in the ready state. - kubectl get pods -o wide: info about the node also

3) kubectl describe pod nginx - More info on the pod

YAML

• Used by k8 for initializing various objects like pods, replicas, services, etc.
• A k8 configuration always consists of atleast below 4 top-level fields:
  • apiVersion
  • kind: the type of object we are creating
  • metadata: dictionary. about object. cannot add props not supported by k8.
  • spec: about images and container
    • container: under this following can be defined:
      • name
      • image
      • command
      • args
      • env

[!TIP] Know about apis using kubectl explain pods / kubectl explain deploy / kubectl explain pods.spec | less or kubectl explain --recursive pods.spec [better layout]

• To get the YAML for existing service:

  • kubectl get pods <pod-name> -o yaml

• Try to generate YAML files rather than creating them.

  • kubectl run mynginx --image=nginx --dry-run=client -o yaml > mynginx.yaml

Commands

• kubectl create -f my.yaml
• kubectl apply -f my.yaml: create if doesn't exist and modify if exists.
• kubectl replace -f my.yaml: replace with new configuration
• kubectl delete -f my.yaml: remove everything specified in the file

[!TIP] To get into a running container using k8, we can do same things as docker: kubectl exec -it <pod-name> -- /bin/bash.

example.yml

apiVersion: v1
kind: Pod
metadata:
    name: myapp-prod
    labels:
        app: my-app
        type: front-end
spec:
    containers:
        - name: nginx-container
          image: nginx    

Then, start using kubectl create -f example.yml or kubectl apply -f example.yml and verify using kubectl describe pod nginx.

[!TIP] We can also generate a pre-built yml using: kubectl run redis --image=redis --dry-run=client -o yaml > redis.yaml

Example 2 (ssh into Ubuntu image)

1) First, generate a yaml for ubuntu image using: - kubectl run myubuntu --image=ubuntu --dry-run=client -o yaml > myubuntu.yaml

2) Now, because ubuntu is just a base image, if we just create the pod using the above, it will fall in a start-loop. So, we edit the myubuntu.yaml:

apiVersion: v1
kind: Pod
metadata:
    labels:
        run: ubuntu
    name: ubuntu
spec:
    containers:
    - image: ubuntu
      name: ubuntu
      command: ["sleep"]
      args: ["infinity"]
// and the default ones

3) Create Pod using: kubectl create -f myubuntu.yaml. 4) SSH into the image using: kubectl exec -it myubuntu -- /bin/bash.

Troubleshooting the errors

• Start by checking kubectl get pods.
• If some error, do kubectl describe pod <name>.
• If non-zero exit code, do kubectl logs <name>.

[!NOTE] There's no way to stop the pod in k8. Stopping is just like terminating so terminate using kubectl delete <pod>.

Health Checks

• When you we run our application as a container in Kubernetes, it is automatically kept alive for us using a process health check. This health check simply ensures that the main process of our application is always running. If it isn’t, Kubernetes restarts it.
• But there are application specific health checks to verify the functioning of the application.
• We define them in the Pod manifest.

apiVersion: v1
kind: Pod
metadata:
  name: kuard
spec:
  containers:
    - image: gcr.io/kuar-demo/kuard-amd64:blue
      name: kuard
      livenessProbe: # health check
        httpGet:
          path: /healthy
          port: 8080
        initialDelaySeconds: 5 # not called for initial 5 seconds
        timeoutSeconds: 1 # probe must respond within 1 sec timeout
        periodSeconds: 10 # call the probe every 10 sec
        failureThreshold: 3 # after 3 failures, restarts Pod
      ports:
        - containerPort: 8080
          name: http
          protocol: TCP

Port Forwarding

• The internal private IP of the pod can be exposed using a port mapping by doing:
  • kubectl port-forward nginx 8080:80 & (& makes this a background process).

[!NOTE] Check out for the advanced concept Security Context once familiar with basics.

Managing Jobs

• Pods are normally created to run forever.
• But sometimes we need a pod to terminate after the job completion. For this, we use Jobs.
• Jobs are useful for one-shot tasks like backup, calculation etc.
• Use spec.ttlSecondsAfterFinished to clean up completed Jobs automatically.

There are 3 types of Jobs specified by completions and parallelism:

1) Non-parallel Jobs: Default one. One pod is started, unless the pod fails. - completions: 1 - parallelism: 1

2) Parallel Jobs with fixed completion count: Job is complete after successfully running as many times as specified in jobs.spec.completions: - completion: n - parallelism: m

3) Parallel Jobs with a work queue: Multiple jobs are created, when one completes successfully, the Job is complete: - completion: 1 - parallelism: n

Example

1) Do this: kubectl create job myjob --image=busybox --dry-run=client -o yaml -- sleep 5 > myjob.yaml. 2) Then edit the yaml file:

C: 1 & P: 1

apiVersion: batch/v1
kind: Job
metadata:
  creationTimestamp: null
  name: myjob
spec:
  completions: 1
  parallelism: 1
  ttlSecondsAfterFinished: 10 # takedown after 10 seconds of completion
  template:
    metadata:
===

C: 2 & P: 4 Change the relevant vars

CronJobs

• Jobs that run on a schedule.

1) kubectl create cronjob mycronjob --image=busybox --schedule="*/2 * * * *" -- echo greetings from your cluster.

Now the cronjob will run on its specified schedule but for testing, to run it before, do:

kubectl create job mycronjob --from=cronjob/mycronjob

Schedule The schedule syntax is defined as:

* * * * *

1) minute (0-59)
2) hour (0-23)
3) day of the month (1-31)
4) month (1-12)
5) day of the week (0-6: sun to sat)    

Example

0 0 13 * 5: 13th of each Friday every week every month

[!NOTE] We can also set quotas or resource limits. Read it after basics.

Deleting

To delete everything: - kubectl delete all --all

[!IMPORTANT] Don't force delete. Care about your job :/

Deployment

• Standard for running applications in K8.
• Adds to scalability and reliability of the application.
• Protects Pods and auto restarts them in case of failure.

Create a deployment - kubectl create deployment name --image=<> or - kubectl create deploy name --image=nginx --replicas=3

Get deployments - kubectl get deployments or - kubectl get deploy name

Expose the service - kubectl expose deployment name --type=NodePort --port=8080

Delete deployment - kubectl delete deployment name or - kubectl delete deploy name

Replicas

• Helps in making sure that if one pod goes down, another one is brought up.
• Pod can be surrounded by replication controller.
• Makes sure that the specified no. of pods are always running as intended.
• Load Balancing & Scaling can be done with the help of replication controllers.
• They can span over multiple nodes.

Replication Controller or Replica Set? - Replica Set is the new version for Replication Controller. - Replica Set can control the pods created earlier also but controller only controls those created using it.

Replication Controller

apiVersion: v1
kind: ReplicationController
metadata:
    name: myapp-rc
    labels:
        app: myapp
        type: front-end # to tag pods of same category
spec:
    template:
        metadata:
            name: myapp-pod
            labels:
                app: myapp
                type: front-end
        spec:
            containers:
                - name: nginx-container
                  image: nginx
replicas: 3                

• Run by: kubectl create -f above.yml
• Info by: kubectl get replicationcontroller & get pods

ReplicaSet

apiVersion: apps/v1 # change here
kind: ReplicationSet # change here
metadata:
    name: myapp-rc
    labels:
        app: myapp
        type: front-end # to tag pods of same category; shoule be same as below
spec:
    template:
        metadata:
            name: myapp-pod
            labels:
                app: myapp
                type: front-end # same
        spec:
            containers:
                - name: nginx-container
                  image: nginx
replicas: 3
selector: # identify what pods fall under it
    matchLabels:
        type: front-end # all the pods; the ones not even created using replicaSet

• Info by: kubectl get replicaset
• Deployment uses ReplicaSet for scalability. Donot manage ReplicaSets directly, manage them through the deployment only.

Scale

We can scale by following ways: 1) Change replicas in yml file and run: kubectl replace -f above.yml. 2) Or do kubectl scale --replicas=6 -f above.yml. 3) Or run kubectl edit deployment name to edit the number of replicas manually.

[!NOTE] Both above scale as well as modify the replicas in the yml files.

4) Or do kubectl scale deployment name --replicas=6 or kubectl scale --replicas=6 replicaset <name-of-replica> -> doesnt modifies the yaml file.

Delete replicas

kubectl delete replicaset myapp-replicaset (also deletes underlying pods)

[!IMPORTANT] The names and labels of pods and under selector of replicaset should be the same.

[!NOTE] If we already have replicaset with a label, we cannot manually create a pod with the same label. ReplicaSet will automatically delete it.

Editing replicaset - Use kubectl edit replicaset <replicaset-name>. Changes on-the-spot: running configuration.

[!TIP] To delete all, use -all arg like: kubectl delete pod -all.

[!TIP] To see all the available api versions, do kubectl api-versions.

More about Deployment

• Controls the deployment aspects in k8s.
• Comes above the replica set
• Containers -> Pods -> Replica Set -> Deployment

apiVersion: apps/v1 
kind: Deployment # change here
metadata:
    name: myapp-rc
    labels:
        app: myapp
        type: front-end
spec:
    template:
        metadata:
            name: myapp-pod
            labels:
                app: myapp
                type: front-end 
        spec:
            containers:
                - name: nginx-container
                  image: nginx
replicas: 3
selector: 
    matchLabels:
        type: front-end

Then, do: - kubectl create -f deployment.yaml. - kubectl get deployments - kubectl get replicaset

[!NOTE] Deployment auto creates the replica set.

[!TIP] Or get everything using: kubectl get -all.

[!TIP] To view specific info, use --selector attribute: kubectl get all --selector app=redis.

Deployment short-hand kubectl create deployment <name> --image=nginx --replicas=3

Updates

• Updates are used to change diff parts of the deployment.

  • Image versions
  • any param that can be used with kubectl set command

• When an update is applied, a new ReplicaSet is created with new properties.

  • After successfull updates, the old replicaSets may be deleted.
  • The deployment.spec.revisionHistoryLimit is set to keep the 10 last ReplicaSets.

kubectl set - Check kubectl set -h. - For example, for changing the image, do: kubectl set image deploy mynginx nginx=nginx:1.17 => creates a new ReplicaSet. The pods are modified, old ones are deleted but the old replicaset still remains. Might come handy during rollbacks.

Labels

• key:value pairs
• Set in resources and use selectors to connect to related resources.
• Deployment finds its pods using a selector.
• Service finds its endpoint pods using a selector.
• Set a label: kubectl run hello --labels="env=prod,ver=1"
• To find all the resources matching a specific label, use --selector-key=value.

Get the labels using: kubectl get all --show-labels.

Or do: kubectl get deploy -L canary.

Annotations

• Extra info.
• key-pair.

Example

...
metadata:
    annotations:
        example.com/icon-url: "https://example.com/icon.png"
...

Rollouts & Versioning

• Types of deployment strategies:
  • Recreate (not recommended): Destroy all the old ones and then start the new ones.
  • Rolling Update: Destroy one old and then bring new one in its place.
    • maxUnavailable: determines the max no of pods that are upgraded at the same time. By default, Kubernetes terminates one pod at a time while creating new pods, ensuring that the desired replica count is maintained.
    • maxSurge: the no of pods that can run beyond the desired no of pods as specified in a replica to guarantee minimal availability. or maximum number of additional pods that can be created during a rolling update.

Rollout

kubectl rollout status deployment/myapp-deployment

Revisions or History of rollouts

• kubectl rollout history deployment/myapp-deployment

• kubectl rollout history deployment rolling-nginx --revision=2

Rollback

• kubectl rollout undo deployment/myapp-deployment

• kubectl rollout undo deployment rolling-nginx --to-revision=1

• Rollout creates a new replicaset and then starts stopping the pods in old repicaset and spinning-up a new pod in the new replicaset.

Example 3 (create and update nginx)

1) kubectl create deployment proxy --image=nginx:1.9 --replicas=5 --dry-run=client -o yaml > mynginx.yaml 2) kubectl create -f mynginx.yaml 3) kubectl set image deployment proxy nginx=nginx:latest

AutoScaling

• In real clusters, Pods are often automatically scaled based on resource usage properties that are collected by metrics server.

• The Horizontal Pod Autoscaler observes usage statistics and after passing a threshold, will add additional replicas.

• Autoscale using: kubectl autoscale deployment php-apache --cpu-percent=50 --min=1 --max=10 (means if the cpu usage > 50%, scale up as desired but upto 10 pods only)

• Get pods using: kubectl get hpa.

There are many addons of minikube, view them using: minikube addons list. Enable using minikube addons enable <add-on>.

Resource requests

• We can specify the resources limits in the k8.
• They are requested per containers, not per Pods. The final sum for Pod is the sum of all the resources requested for the containers.

Networking

• Each POD in a node is assigned its own IP.
• That IP is subject to change when PODs are recreated.
• K8 doesn't set up some networking fundamentals itself. It expects us to do the following:
  • All containers/PODs can communicate to one another without NAT.
  • All nodes can communicate with all containers and vice-versa without NAT.
• For setting that up, we can use external networking services like CISCO, VMWare NSX, cilium, flannel, etc.

Services (Service Discovery)

• Helps decouple groups and the connectivity between them.
• Eg. frontend attached to backend and backend attached to a db source.
• to expose a logical set of Pods.
• set of pods targeted by a service is determined by a selector (label)
• The kube-controller-manager will continuously scan for pods that match the selector and include these in the service
• kube-proxy agent on the nodes watches the Kubernetes API for new services and endpoints. It works in the background & is reponsible for redirecting the traffic to the desired Pod.
• We can use kubectl expose to create a service. After exposing, the service is assigned a virtual IP called a Cluster IP. This special IP address will load balance across all of the pods that are identified by the selector.
• Then to interact with service, we perform the --port-forward to one of the pods.

Service DNS

• K8 provides a DNS service exposed to Pods running in the cluster.
• Installed by default. Managed by k8.
• It provides DNS names for cluster IPs.
• Service object operates at Layer 4 (TCP/IP). Check Ingress for HTTP/HTTPS aware connections.
• Format is:

service-name.namespace.svc.cluster.local.

• cluster.local. : base domain name for the cluster

Types of Services

1) NodePort - For allowing new traffic in. - We have a pod inside a node. We can't directly access the pod as it is an internal IP. - We can expose the service on a port using Service. - It listens on exposed port and forwards the requests to pod back and forth. - allocates a specific node port on the node that forwards to the service cluster IP address.

service.yaml

apiVersion: v1
kind: Service
metadata:
    name: myapp-service
spec:
    type: NodePort
    ports:
        - targetPort: 80
          port: 80
          nodePort: 30008
    selector: # to which pod to associate service with
        app: myapp
        type: frontend

• Run with: kubectl create -f service.yaml
• Get services: kubectl get services

[!NOTE] Let's say we had 3 different pods with same labels. The above service would take all those as the endpoints and would balance the requests.

Algorithm: Random

SessionAffinity: Yes

Incase of multiple Nodes

Different IP for different nodes but with same port.

2) ClusterIP

• Communicate through Service (provides a uniform interface) Cluster IP.
• the default type exposes the service on an internal cluster IP address.

clusterip.yaml

apiVersion: v1
kind: Service 
metadata:
    name: back-end
spec:
    type: ClusterIP # default; if not specified
    ports:
        - targetPort: 80
          port: 80

    selector:
        app: myapp
        type: back-end

3) LoadBalancer

• This is cloud-platform specific.
• Just set kind to LoadBalancer.

4) ExternalName

• works on DNS names;
• redirection is happening at DNS Level.

How to service

• kubectl expose: for creating services; providing access to deployments, ReplicaSets, Pods or other services
• In most cases, kubectl expose exposes a Deployment, which allocates its Pods as the service endpoint.
• kubectl create service: alternative way.
• Specify --port to indicate the service port.

While working with services, diff ports are specified: - targetport: port on container - port: port on service which is accessible - nodeport: port exposed externally (large range)

Example 4 (service for nginx)

1) kubectl create deploy nginxsvc --image=nginx --replicas=3 2) kubectl expose deploy nginxsvc --port=80 3) kubectl describe svc nginxsvc # look for endpoints 4) kubectl get svc nginxsvc -o=yaml 5) kubectl get svc 6) kubectl get endpoints (endpoints are dynamically fetched)

• ClusterIP is not reachable. Can curl nginx by ssh'in into minikube: minikube ssh and then doing curl. ClusterIP is for internal access only.

To make it public, edit the service and change the following:

• type: NodePort

• ports:

  • targetPort: 80
  • nodePort: 32000

• Then check the minikube ip and :32000.

[!NOTE] So, the frontend can be expose as NodePort while the backend can be exposed as ClusterIP (remains private), hence more secure.

NetworkingPolicy

• By default, there are no restrictions to network traffic in k8s. Pods can always communicate even if they are in other NameSpace.
• To limit it, NetworkPolicies can be used.
• If in a policy there is no match, traffic will be denied.

Three different identifiers can be used: - Pods: (podSelector) - Namespaces: (namespaceSelector) to grant access tospecific namespaces. - IP blocks: (ipBlock)

Selector labels are used heavily. And NetworkPolicies donot conflict, they are additive.

Ingress

• basically, when we expose services, we connect using ports say http://192.19.10.9:3000. This is good for testing but in prod, we need something like http://hello-app. Then we redirect to internal service i.e. route to ports based on paths.
• provide external access to internal k8 cluster resources.
• uses a load balancer present on the external cluster.
• uses selector lables to connect to Pods that are used as service endpoints.
• to access resources in the cluster, DNS must be configured to resolve to the ingress load balancer IP.
• exposes HTTP/HTTPS routes from outside the cluster to services inside.
• Can do:
  • load balance traffic
  • terminate SSL/TLS
• Need ingress controllers:
  • nginx
  • hapropxy
  • kong

Ingress Controller

• Ingress requires ingress controller to be installed on the cluster to perform the rules evaluation.
• Entrypoint to the cluster.
• Eg. K8s Nginx Ingress Controller
• When we minikube addons enable ingress, it automatically starts the nginx implementation of Ingress Controller.
• Check with kubectl get pod -n kube-system

Setup in minikube

• minikube addons enable ingress: enable

[!IMPORTANT] Get all the namespaces in k8: kubectl get ns

1) This continues on Example 4. 2) kubectl get deployment 3) kubectl get svc nginxsvc 4) curl http://$(minikube ip):32000 5) kubectl create ingress nginxsvc-ingress --rule="/=nginxsvc:80" --rule="/hello=newdep:8080" (we can even create a rule for the image not yet created) 6) kubectl describe ingress nginxsvc-ingress 7) minikube ip (note this ip) 8) Open /etc/hosts and add the line: 192.168.49.2 nginxsvc.info 9) curl nginxsvc.info

Ingress Rules

Each rule contains the following: - An optional host. If no host is specified, the rule applies to all inbound HTTP traffic. - A list of paths and each path has its own backend. - Backend which consists of either a service or a resource. Should configure a default backend for traffic that doesnt matches a specific path.

[!NOTE] Resource backends and service backends are mutually exclusive.

Ingress Path Types

• pathType: how to deal with path requests.

  • Exact: exact match
  • Prefix: should start with

• single service: kubectl create ingress single --rule="/files=filesservice:80"

• simple fanout: kubectl create ingress single --rule="/files=filesservice:80" --rule="/db=dbservice:80"
• name-based (route requests based on the host header):
  • make sure that there is a DNS entry for each host header
  • kubectl create ingress multihost --rule="my.example.com/files*=filesservice:80" --rule="my.example.org/data*=dataservice:80"

[!IMPORTANT] About /etc/hosts

The /etc/hosts file in Linux is a plain text file that acts as a local directory for translating hostnames into IP addresses. It has following contents: 1) Lines: Each line typically consists of two parts separated by spaces or tabs. 2) IP Address: The first part is the IP address of the host. 3) Hostname(s): The second part is one or more hostnames (domain names) that you want to associate with that IP address.

For example, inside the file, the following

127.0.0.1 example.com

tells your system to direct "example.com" to the loopback address (127.0.0.1), effectively blocking it.

Example 5 (name-based virtual hosting ingress)

1) kubectl create deploy mars --image=nginx 2) kubectl create deploy saturn --image=httpd 3) kubectl expose deploy mars --port=80 4) kubectl expose deploy saturn --port=80 5) Add entries to /etc/hosts: - $(minikube ip) mars.example.com - $(minikube ip) saturn.example.com 6) kubectl create ingress multihost --rule="mars.example.com/=mars:80" --rule="saturn.example.com/=saturn:80" 7) kubectl edit ingress multihost and change pathType to Prefix. 8) curl saturn.example.com & curl mars.example.com

Sample Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: minimal-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx-example
  rules:
  - host: my-app.com # this corresponds to the dns of the main master node server (the entrypoint)
    http:
      paths:
      - path: /testpath
        pathType: Prefix
        backend:
          service:
            name: test
            port:
              number: 80

Default Backend

There is a default backend for all the requests that don't match. It is specified in the kubectl describe ingress.

We can configure it by running an internal service with the exact default-backend name and exposing it on the defined port in the ingress default backend.

Storage Options

• Files stored in containers are ephemeral.
• Pod Volumes are used to allocate storage that outlives a container and stays available during Pod lifetime.
• Pod Volumes can directly bind to a specific storage type.
• Pod Volumes can also use Persistent Volume Claims (PVC) to decouple the Pod from the requested site-specific storage.
• A persistent volume defines access to external storage available in a specific cluser.
• A wide range of PV types are available.
• Persistent Volume Claims (PVC) are used to connect to Persistent Volumes.
• When a PVC is created, it will search for an available PV that matches the storage request.
• If a perfect match doesnot exist, StorageClass can automatically create it.

Configuring Volumes

• To configure Pod Volumes, the volume is defined in pod.spec.volumes. This array defines all the volumes that may be accessed by containers in the Pod manifest.
• The volume points to a specific volume type:
  • For testing purposes,
    • emptyDir: temp storage
    • hostPath: persistent one
  • Other types are also available like s3 or other cloud providers.
• Then, the volume is mounted in the containers through pod.spec.containers.volumeMounts. This array defines the vols that are mounted into a particular container and the path where each volume should be mounted.

Set up Pod Local Storage

1) kubectl explain pod.spec.volumes 2) Create the following yaml:

apiVersion: v1
kind: Pod
metadata:
  name: "morevol2"
  labels:
    app: "morevol2"
spec:
  containers:
  - name: centos1
    image: centos:7
    command:
      - sleep
      - "3600"
    volumeMounts:
    - name: test
      mountPath: /centos1

  - name: centos2
    image: centos:7
    command:
      - sleep
      - "3600"
    volumeMounts:
      - name: test
        mountPath: /centos2
  volumes:
    - name: test
      emptyDir: {}
  restartPolicy: Always
---

3) kubectl create -f morevolumes.yaml 4) kubectl get pods morevol 5) kubectl describe pods morevol | less 6) kubectl exec -ti morevol2 -c centos1 --touch /centos1/test 7) kubectl exec -it morevol2 -c centos2 -- ls /centos2 8) kubectl exec -ti morevol2 -c centos2 --touch /centos2

Persistent Volumes

• An independent resource that connects to external storage.
  • Can point to all types of storages
  • Use PVC to connect to it.
  • The PVC talks to the available backend storage provider and dynamically uses volumes that are available on that storage type
• The PVC will bind to a PV according to the availability of the requeste volume accessModes and capacity.

[!NOTE] The above will create on the host. In our case, it's not gonna be our system but the minikube host. So, ssh into it to verify.

Setting up

1) Create the following yaml:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-volume
  labels:
    app: pv-volume
spec:
  # AKS: default,managed-premium
  # GKE: standard
  # EKS: gp2 (custom)
  # Rook: rook-ceph-block,rook-ceph-fs
  capacity:
    storage: 2Gi
  accessModes:
  - ReadWriteOnce
  hostPath:
    path: "/mydata"
---

2) Create it.

PVC Binds

• After connecting to a PV, the PVC will shows as bound.
• A PVC is bind exclusive; after successfully binding to a PV, the same PV cannot be used by other PVC anymore.
• Say you got a 100gb storage. If PVC only needs 1gb of it and is bound to it, the 99gb is a waste.
• If a perfect match is not available, StorageClass may kick in and dynamically create a PV that does exactly match the request.

Create a PVC

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pv-claim
  namespace: default
  labels:
    app: pv-claim
spec:
  # AKS: default,managed-premium
  # GKE: standard
  # EKS: gp2 (custom)
  # Rook: rook-ceph-block,rook-ceph-fs
  storageClassName: default
  accessModes:
  - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---

Using PVCs for Pods

Example

1) Create the following:

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nginx-pvc
spec:
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 2Gi
---
apiVersion: v1
kind: Pod
metadata:
  name: nginx-pvc-pod
  labels:
    app: "nginx-pvc-pod"
spec:
  volumes:
    - name: site-storage
      persistentVolumeClaim:
        claimName: nginx-pvc
  containers:
  - name: pv-container
    image: nginx
    ports:
    - containerPort: 80
      name:  webserver
    volumeMounts:
    - name: site-storage
      mountPath: "/usr/share/nginx/html"

2) kubectl get pvc,pv 3) kubectl describe pv pvc-xxx-yyy 4) kubectl exec nginx-pvc-pod -- touch /usr/share/nginx/html/testfile 5) minikube ssh 6) ls /tmp/hostpath-provisioner/default/nginx-pvc/

StorageClass

• Allows for auto provisioning of PVs when a PVC request for storage comes in
• Must be backed by a storage provisioner, which takes care of the volume configuration
• K8 comes with some internal provisioners, alternatively external provisioners can be installed using Operators

[!IMPORTANT] - Typical lab env is not backed by a k8 provisioner. - But minikube comes with a provisioner.

• A PVC will only bind to a PV that has the same StorageClass setting. If such a PV is not available, then it stays in pending state.

Know about the storage class by: kubectl get storageclass.

MORE ABOUT STORAGECLASS LATER ON

ConfigMaps

Providing Variables

• K8 doesnt offers a cmd option to provide vars while running a deployment with kubectl create deploy.

  • First create the deploy
  • Then, do kubectl set env deploy deploy_name VAR=VALUE

• While running a POD, we can provide env but we shouldnt run naked pods.

  • kubectl run mydb --image=mysql --env="ENV_VAR=VALUE"

Need for decoupling

• Better management
• portability
• Variables are site-specific & should not be provided in the deployment config.
• K8 provide ConfigMap.
• It is used to define vars and the deployment just points to ConfigMap. Site-specific info is included in the ConfigMap which allows to keep the deployment manifest files generic.
• ConfigMap is stored in etcd.
• They can be used for:
  • Vars
  • Config files
  • CMD args
• If an app refers to a ConfigMap, then the ConfigMap should exist in the cluster before running the application.

Providing Vars using ConfigMaps

1) Two ways using kubectl create cm mycm: - Using: --from-env-file=file-name (cant use multiple times) - Using: --from-literal=VAR=VALUE (can use multiple times)

2) Then, use kubectl set env --from=configmap/mycm deploy/myapp (create the said deployment)

3) Use --dry-run=client on kubectl create deploy & kubectl set env to generate yaml file using ConfigMap. - eg. kubectl get deployment.apps mydb -o yaml | less.

Providing ConfigMaps for ConfigFiles

• kubectl create cm mycm --from=file=/my/file.conf.
• If dir is specified, all the confs are included in the ConfigMap.
• To use the conf file in an app, ConfigMap must be mounted in the app.

Mounting a ConfigMap in an App - Generate the base YAML code, and add the ConfigMap mount to it later on. - In the app manifest, define a vol using ConfigMap type. - Mount this vol on a specific dir. - The ConfigMap file will appear inside the directory.

Example 6 (Using ConfigMap with a conf)

• echo "hello world" > index.html
• kubectl create cm myindex --from-file=index.html
• kubectl describe cm myindex
• kubectl create deploy myweb --image=nginx

• kubectl edit deploy myweb

  • spec.template.spec

    yaml volumes: - name: cmvol configMap: name: myindex

  • spec.template.spec.containers

    volumeMounts:
      - mountPath: /usr/share/nginx/html
        name: cmvol
    

Secrets

• senstive data
• secrets are not encrypted; they are base64 encoded.
• By default, k8 secrets are stored in plain text in the etcd storage for the cluster.
• 3 types of secrets offered:
  • docker-registry: used for conn to docker registry
  • TLS: used to store TLS key material
  • generic: creates a secret from local file, dir or literal value
• When defining secret, the type must be specified: kubectl create secret generic ...
• To access the K8 API, all k8 resources need access to TLS keys.
• These keys are provided by Secrets and used through ServiceAccounts.
• By using ServiceAccount, the application gets access to its Secret.

Demo

1) do kubectl get pods -n kube-system 2) do kubectl get pods -n kube-system coredns-<tab> -o yaml | less: we'll see serviceAccount: coredns 3) do kubectl get sa -n kube-system coredns -o yaml | less: we see the secret name 4) do kubectl get secret -n kube-system coredns-token-<tab> -o yaml: we see the secrets 5) for more: try to decode the namespace by: echo <namespace> | base64 -d. 6) Normally this is not easily accessible in real-world secure K8 cluster. It's the kubeadmin level.

Using Secrets in Apps

• To provide TLS keys: kubectl create secret tls name --cert=tls/my.crt --key=tls/my.key
• To provide security to passwords: kubectl create secret generic my-secret-pw --from-literal=password=verysecret
• To provide access to SSH private key: kubectl create secret generic my-ssh-key --from-file=ssh-private-key=.ssh/id_rsa
• To provide access to senstive file, which would be mounted in the app with root access only: kubectl create secret generic my-sec-file --from-file=/my/secretfile

[!IMPORTANT] - As a secret is an encode ConfigMap, it is used in a way similar to using ConfigMaps. - If it contains vars, use kubectl set env. - If it contains files, then the same mounting way like ConfigMap. - While mounting the Secret in the Pod spec, consider using defaultMode: 0400.

Demo

1) Create a secret: kubectl create secret generic dbpw --from-literal=ROOT_PASSWORD=password 2) kubectl describe secret dbpw 3) kubectl get secret dbpw -o yaml 4) kubectl create deploy mynewdb --image=mariadb 5) kubectl set env deploy mynewdb --from=secret/dbpw --prefix=MYSQL_

Configuring docker-registry seceret

kubectl create secret docker-registry my-docker-cred --docker-username=$NAME --docker-password=$PASS --docker-email=$EMAIL --docker-server=myregistry:3000

• Registry can be any cloud registry or local registry.

Example 7 (ConfigMaps & Secrets)

ToDo - Create index.html with "Hello world" - Create a ConfigMap that has the contents of the index.html - Create a Secret that contains the var definition MYPASSWORD=verysecret - Create a Manifest file that starts an Nginx Deployment that mounts the index.html file from the ConfigMap on /usr/share/nginx/html and sets the env var from the secret.

Way

1) Create index.html 2) kubectl create cm mycm --from-file=index.html 3) kubectl get cm mycm 4) kubectl get cm mycm -o yaml 5) kubectl create secret generic mysec --from-literal=MYPASSWORD=verysecret 6) kubectl get secret mysec -o yaml 7) echo <secret> | base64 -d 8) kubectl get cm,secret,deploy 9) kubectl create deploy mynginx --image=nginx 10) kubectl set env deploy mynginx --from=secret/mysec 11) kubectl edit deploy mynginx 12) Do the following edit:

under spec.template

volumes:
    name: myvol
    - configMap:
        name: mycm

under spec.template.spec.containers

volumeMounts:
    - name: myvol
      mountPath: /usr/share/nginx/html

13) kubectl get deploy mynginx -o yaml > mynginxconfig.yaml 14) kubectl delete deploy mynginx 15) kubectl create -f mynginxconfig.yaml 16) Check using: - kubectl exec mynginx-<pod-name> -- cat /usr/share/nginx/html/index.html - kubectl exec mynginx-<pod-name> -- env

kube-proxy

• K8 api's are RESTful means they can be accessed using direct HTTP traffic.
• kubectl uses curl behind the scenes. But in a secure way.
• To make our own API requests using curl, appropriate certs are required.
• Use kube-proxy to provide a secure interface for curl.
• We use curl to connect to kube-proxy which in turn connects to kubectl's .kube.config and uses TLS by deafult.

To access the API using curl, start the kube-proxy on the k8 user workstation: - kubectl proxy --port=8001& - curl http://localhost:8001 and many other apis.

[!IMPORTANT] With new k8 release, old API versions may be deleted and will be supported for a min of 2 more k8 releases.

Authentication & Authorization

Authentication

• Authentication is about where k8 users come from.
• By default, a local k8 admin account is used and auth is not required.
• We can create our own user accounts.
• The kubectl config specified to which cluster to authenticate.
  • Use kubectl config view: to see current settings
  • Config is read from ~/.kube/config

Authorization

• Authorization is what users can do.
• Behind authorization, there is Role Based Access Control (RBAC) to take care of diff options
• Use kubectl auth can-i get pods to find out about the authorization status
• Can also check for another user: kubectl auth can-i get pods --as arnav@example.com

Understanding RBAC

• It is used to provide access to API Resources.
• 3 elements are used:
  • Role: defines access permissions to specific resources
  • User or ServiceAccount: to work with the API
  • RoleBinding: connects a user or ServiceAccount to a specific Role

ServiceAccounts

• All actions in k8 cluster need to be authenticated and authorized
• ServiceAccounts are used for basic authentication from within the k8 cluster
• RBAC is used to connect SA to a specific Role
• Every Pod uses the default SA to contact the API server
• Each SA uses a secret to automount API creds

Like this: Pod -> SA -> Role-binding -> Role -> api-server -> etcd

• SA comes with a secret which contains API creds and is auto mounted.
• Get it with: kubectl get sa or kubectl get sa -o yaml.
• To get all the sa available: kubectl get sa -A

More RBAC

• A cluster admin can configure RBAC authorization to specify what exactly can be accessed by a SA or user
• Custom SA can be created and configured with additional permissions for enhanced access from Pods to cluster resources
• For additional permissions, Namespace or cluster scope Roles and RoleBindings are used
• ClusterRole and ClusterRoleBinding are used to configure access to cluster resources

Example 8 (Creating Role and its binding and the SA)

1) Role yaml:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: list-pods
  namespace: default
rules:
  - apiGroups:
    - ''
    resources:
    - pods
    verbs:
    - list

2) Role binding yaml:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: list-pods-mysa-binding
  namespace: default
roleRef:
  kind: Role
  name: list-pods
  apiGroup: rbac.authorization.k8s.io
subjects:
  - kind: ServiceAccount
    name: mysa
    namespace: default

3) But it requires the SA mysa. It is not created yet. Create it using: kubectl create sa mysa.

Managing ServiceAccounts

• Every namespace has a default SA called default
• Additional SA can be created to provide additional access to resources
• After creating additional SA, it needs to get permissions through RBAC
• SA can be created using the imperative way: kubectl create serviceaccoubt mysa
• While creating SA, a Secret is automatically created to have the SA connect to the API
• This secret is mounted in Pods using the SA and used as an access token
• After creating the SA, RBAC must be configured

Example 9 (Configuring & Using SA)

ToDo

• Create a Pod using Standard SA
• Try to access the pod using curl (https://kubernetes/api/v1 --insecure: will be forbidden)
• Try to access using the default SA token
• Try again but this time to list the pods
• Then create a new SA with new Role and RoleBinding that is able to list the pods

Way

DevOps Way

Helm

• To streamline installing and managing k8 apps
• Consists of the helm tool and a chart
• A chart is a helm package, which contains:
  • A desc of the package
  • One or more temps containing the k8 manifest files
• Charts can be stored locally or accessed from Helm repos.
• Helm also supports templating engine meaning we can have a dynamic yaml file (can replace the vars on the fly)
  • better suited for CI/CD pipeline where in our build we can replace the values on the fly
  • same apps in different environments

Installation

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh

Helm Charts

• Main site for finding them: https://artifacthub.io.
• Search the required and install like:
  • helm repo add kubernetes-dashboard https://kubernetes.github.io/dashboard
  • helm install kubernetes-dashboard kubernetes-dashboard/kubernetes-dashboard

[!NOTE] In minikube, use minikube addons enable dashboard.

We can: - helm repo add name url - helm repo list - helm search repo name - helm repo update - helm install <release_name> name - helm list - helm delete - helm upgrade <release_name> name - helm rollback <release_name> <revision_no>

Helm Demo (mysql helm chart)

1) See this: https://artifacthub.io/packages/helm/bitnami/mysql?modal=install. 2) kubectl get all 3) helm show chart bitnami/mysql 4) helm list 5) helm status <name>

But there's a way to customize before installing (RECOMMENDED)

• A Helm chart consists of templates to which specific values are applied.
• Stored in values.yaml file within the helm chart.
• So, to modify, begin with helm pull to fetch a local copy of the helm chart
• Then edit chartname/values.yaml to change the values

1) helm show values bitnami/nginx 2) helm pull bitnami/nginx: downloads the tar 3) tar xvf nginx-xxx 4) vim nginx/values.yaml 5) helm template --debug nginx (to check what will be applied; do in the directory where nginx is subdir or extracted) 6) helm install -f nginx/values.yaml my-nginx nginx/

Create Helm Chart

1) helm create <name> 2) helm list -a 3) helm install <release_name> <dir> 4) helm install <release_name> --debug --dry-run <dir>: Validate and verify the helm chart 5) helm template <dir>: for only validating the yamls without connecting to k8 api server 6) helm lint <dir> 7) Then, configure out the values.yaml, Charts.yaml and if required the templates files in the dir.

HelmFile: automate Helm

• pre-requisite: Helm Chart installed

Installation

• Download the binary release of Helm.
• Untar and chmod 777 helmfile.
• mv helmfile /usr/local/bin
• Ready: helmfile --version

Then, once we create our helm chart:

1) Create helmfile.yaml at the base of the helm chart folder. 2) Then write:

---
releases:

  - name: helloworld # name of the release
    chart: ./helloworld # name of the chart
    installed: true # install; false: uninstall

3) Then do start do: helmfile sync. 4) Also, we can specify helm chart from github:

---
repositories:
  - name: helloworld
    url: git+https://github.com/rahulwagh/helmchart@helloworld?ref=master&sparse=0

releases:

  - name: helloworld
    chart: helloworld/helloworld
    installed: false 

5) We can also manage multiple helm charts:

---
releases:

  - name: helloworld1
    chart: ./helloworld1
    installed: true

  - name: helloworld2
    chart: ./helloworld2
    installed: true

Helm Tests

• Under tests under templates.
• pre-requisite: helm chart must be installed.
• Then, do helm test <release-name>

Kustomize

• k8 feature (better decoupling)
• file name should be: kustomization.yaml to apply changes to a set of resoruces
• convenient for applying changes to input files that the user doesnt control himself and whose contents may change because of new versions appearing in Git
• Use kubectl apply -k ./ in the dir with the kustomization.yaml and the file it refers to apply changes
• Use kubectl delete -k ./ to undo

Kustomization Overlays

• To define a base configuration as well as multiple deployment scenarios (overlays) as in dev, staging and prod for instance.
• In such a config, the main kustomization.yaml defines the structure:

- base
  - deployment.yaml
  - service.yaml
  - kustomization.yaml

- overlays
  - dev
    - kustomization.yaml
  - staging
    - kustomization.yaml
  - prod
    - kustomization.yaml

• In each of the overlays/{dev,staging,prod}/kustomization.yaml, users would reference the base config in the resources field, and specify changes for that specific env.

kustomization.yaml

resources:
    - ../../base
      namePrefix: dev-
      namespace: development
      commonLabels:
        environment: development

• Then after applying: kubectl apply -k .
• We can also do: kubectl get all --selector environment=development

DIVE MORE INTO KUSTOMIZATION

Vids left: Implementing Canary & Blue-Green deployments in CKAD course. Do it.

Modules

• repeatble ones.
• For getting them, do: terraform get

Example 9 (AWS EKS Setup)

1) Set up all the images on ECR.

[!NOTE] To make the aws ecr cli take the iam sso credentials, do export=PROFILE_NAME beforehand.

2) Download eksctl. 3) For creating the cluster based on ec2, do:

eksctl create cluster \
--name tf-cluster \
--region us-east-1 \
--nodegroup-name tf-nodes \
--node-type t2.micro \
--nodes 2
--zones us-east-1a,us-east-1b

[!IMPORTANT] If the env var AWS_PROFILE is not working, prefix the command above with AWS_PROFILE=arnav <rest-code>

[!IMPORTANT] Cluster creation is not supported in us-east-1e. (atleast when i tested)

[!NOTE] For fargate type, do: --fargate after --region. And for fargate type, we need to define a fargate profile beforehand i.e.

eksctl create fargateprofile \
--cluster cluster-name
--region us-east-1
--name alb-sample-app
--namespace game-2048

4) Check the nodes with: kubectl get nodes 5) Now, we have everything i.e. our helm chart. The only thing to configure is the alb for our service's ClusterIP. For that the ingress annotations are:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: fastapi-ingress
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing # internet-facing for internet facing pod
    alb.ingress.kubernetes.io/target-type: ip # ip for cluster-ip; instance for nodeport
spec:
  ingressClassName: alb
  rules:
    #- host: detect.com # custom domain
    - http:
        paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: fastapi-service
              port:
                number: 8000

6) Now, update the kube-config, aws eks update-kubeconfig --name tf-cluster --region us-east-1 7) Then apply the manifests. 8) Now, if we do: kubectl get ingress, we dont get the address. For that we need to create the alb ingress controller. - In order for our cluster to communicate with AWS service, we need to configure IAM OIDC provider. Do: - eksctl utils associate-iam-oidc-provider --cluster <cluster-name> --approve - Now, we need our ingress controller to communicate with AWS ALB. For that we need to create an iam role and a policy, do: - curl -O https://raw.githubusercontent.com/kubernetes-sigs/aws-load-balancer-controller/main/docs/install/iam_policy.json - aws iam create-policy --policy-name <ANY_NAME> --policy-document file://<the_above_file> - Now we need to attach the policy to the service account of our pods so that they can access: - eksctl create iamserviceaccount --cluster=<your-cluster-name> --namespace=kube-system --name=aws-load-balancer-controller --role-name AmazonEKSLoadBalancerControllerRole --attach-policy-arn=arn:aws:iam::<your-aws-account-id>:policy/AWSLoadBalancerControllerIAMPolicy --approve - Now add the ingress controller: - helm repo add eks https://aws.github.io/eks-charts - helm repo update eks - helm install aws-load-balancer-controller eks/aws-load-balancer-controller -n kube-system --set clusterName=<your-cluster-name> --set serviceAccount.create=false --set serviceAccount.name=aws-load-balancer-controller --set region=<region> --set vpcId=<your-vpc-id> - Now check the ALB controller (they will be deployed in 2 AZ for redundancy): - kubectl get deploy -n kube-system 10) For deleting cluster, do: eksctl delete cluster --name <cluster-name>

[!NOTE] When pod is on aws eks, if need to ssh into pod, do: kubectl run -n YOUR_NAMESPACE troubleshoot -it --rm --image=amazonlinux -- /bin/bash . Afterwards access with: kubectl attach troubleshoot -c troubleshoot -i -t.

[!NOTE] In case of AWS Fargate, CoreDNS gets in pending state if Fargate profile is not created. Fargate Profile allows EKS to automatically create nodes for our application based on kubernetes namespace and optionally pod labels. Check: https://docs.aws.amazon.com/eks/latest/userguide/fargate-getting-started.html#fargate-gs-coredns for more.