Skip to main content

K3s Kubernetes

Useful Commands

What ?

Various commands I keep forgetting šŸ™‚

Deployments

Restart deployments / pods

I keep seeing this on various forums, and I did have to also look for way to restart pod. Since there is no "restart" command. Ok, ok, there are rolling restarts, but if you use that, it first creates new pod in deployment and then terminate the old one. Maybe that's what you need, but what if your deployment have one pod and that pod uses persistent storage (that can be mounted only to one pod) ? Well your rolling restart will be stuck on creating new pod because the storage is not available...

Recently my docker-registry start returning error that it can't reach storage, don't know what happened, but restart seemed like "go to" option.

# Check
root@control01:~# kubectl get pods -n docker-registry
NAME                        READY   STATUS    RESTARTS   AGE
registry-7895c5bf6d-nkhk2   1/1     Running   0          3h11m

root@control01:~# kubectl scale --replicas=0 deployment registry -n docker-registry
# wait till finish
root@control01:~# kubectl scale --replicas=1 deployment registry -n docker-registry

In essence, we told to Kubernetes to scale the pods of deployments to 0, and then back to 1. That's how you restart pod in Kubernetes.

How do I delete what I created ?

Simple if you did not change names / labels in the config you can do the same as kubectl create just with:

kubectl delete -f <file>

How do I change configuration ?

Change the YAML file (keep the labels the same, or it will create another instance) and do:

kubectl apply -f <filename>

This will apply the new settings, in most cases it can create a new instance of the service, wait till it's ready then kills the old instance...bit depends.

ods

How do I copy stuff inside and out of container ?

Well, it's quite easy, but keep in mind that if you copy data into a pod and the target path is not on persistent storage, the changes will disappear anytime the pod moves.

Copy files from container

On your client machine:

kubectl cp <pod_name>:/<file>/<location> <local_destination>
# For example
kubectl cp magento-poc-5699f5f968-fn8jm:/etc/nginx/nginx.conf /home/vlado/

Copy files to the container

Same as above, just in reverse

kubectl cp <local_file>  <pod_name>:/<file_destination>/
# For example:
kubectl cp /home/vlado/magento_www.gzip magento-poc-5699f5f968-fn8jm:/var/www/html/

Evicted pods

There is a mechanism inside Kubernetes that will kick pods if its RAM or CPU is reaching limits. You can see something like this in kubectl get pods

root@control01:~# kubectl get pods
NAME                                        READY   STATUS              RESTARTS   AGE
echo1-68949fd997-f9hxp                      1/1     Running             0          25h
echo1-68949fd997-qj2nv                      1/1     Running             0          25h
echo2-84fcb44c98-hjccw                      1/1     Running             0          25h
ingress-nginx-controller-569cfbd456-brfgd   1/1     Running             0          24h
magento-poc-6c8d8b8545-8vmtw                1/1     Running             0          25m
mysql-9fd744889-5vjc4                       0/1     ContainerCreating   0          3m16s
mysql-9fd744889-6hxgb                       0/1     Evicted             0          3m17s
mysql-9fd744889-d4n8f                       0/1     Evicted             0          5m47s
mysql-9fd744889-fdwzz                       0/1     Evicted             0          3m20s
mysql-9fd744889-fljzk                       0/1     Evicted             0          3m19s
mysql-9fd744889-kbcfl                       0/1     Evicted             0          3m16s
mysql-9fd744889-lcpb2                       0/1     Evicted             0          3m17s
mysql-9fd744889-lx6z6                       0/1     Evicted             0          3m20s
mysql-9fd744889-m242j                       0/1     Evicted             0          3m19s
mysql-9fd744889-m9dxr                       0/1     Evicted             0          3m20s
mysql-9fd744889-q2wkv                       0/1     Evicted             0          3m20s
mysql-9fd744889-tq9nv                       0/1     Evicted             0          3m18s
mysql-9fd744889-wmbrg                       0/1     Evicted             0          25m
mysql-client                                0/1     Completed           0          142m

In most cases it's a RAM issue, and the node kicked (evicted) the container, you can see something like this if you try to describe the evicted node.

root@control01:~# kubectl describe pods mysql-d64dbb968-jlc8n
Name:           mysql-d64dbb968-jlc8n
.
.
.
Events:
  Type     Reason            Age    From               Message
  ----     ------            ----   ----               -------
  Warning  FailedScheduling  3m37s  default-scheduler  0/2 nodes are available: 2 Insufficient memory.
  Warning  FailedScheduling  3m37s  default-scheduler  0/2 nodes are available: 2 Insufficient memory.

This will also taint the node where it happened (describe specific node) and prevents other pods to run there. Manual action is needed!

Clear taint

Removing the taint is easy, just do:

kubectl taint nodes <name_of_node> node.kubernetes.io/memory-pressure:NoSchedule-

This will allow running pods on the node again.

Remove evicted pods

kubectl get pod | grep Evicted | awk '{print $1}' | xargs kubectl delete pod