5 Kubernetes Shell Tricks

Hi! Andrew here—developer at Linode. In the process of building a managed Kubernetes solution, I’ve discovered and created several tricks that I use to make daily tasks with Kubernetes easier. I wanted to share these five here with you and I hope you find them as helpful as I have.

1. The Meta-Shell Trick

I first saw this trick in Olive Power’s talk on CKA Preparation at this year’s KubeCon Europe.

kubectl <command> --help | grep kubectl

Every kubectl command contains a number of examples in its help output, and this is a quick way to see them all at once. These examples can give you ideas about how to combine parameters and flags in useful ways that might not be obvious just from reading the documentation.

Currently, the kubectl run command renders the highest number of examples: 18. However, the command that I use most often is kubectl get. Let’s take a look at one of its examples:

kubectl get pod test-pod -o custom-columns=CONTAINER:.spec.containers[0].name,IMAGE:.spec.containers[0].image

This command transforms the kubectl get output, giving you just the container version deployed for this pod. With some slight changes, you can make the output much more useful. Notice that it’s requesting the name and image for just the first container (containers[0]). Using the JSONPath syntax supported by kubectl, you can list all versions of all containers in all pods.

kubectl get pod -o custom-columns=CONTAINER:.spec.containers[*].name,IMAGE:.spec.containers[*].image -A

This trick can save you from poking around individual pod resources to find a canary image or version skew in your cluster. With an alias of kgi (and the namespace selector -A removed), you can use find the version of the kube-apiserver running in your cluster. See below:

$ kgi -n kube-system -l component=component=kube-apiserver
CONTAINER        IMAGE
kube-apiserver   k8s.gcr.io/kube-apiserver:v1.14.5
kube-apiserver   k8s.gcr.io/kube-apiserver:v1.14.5
kube-apiserver   k8s.gcr.io/kube-apiserver:v1.14.5

The following command will print out an up-to-date list of all built-in examples. Pick one with some flags you’ve never seen before and experiment!

for c in $(kubectl --help | egrep "^\s+([a-z-]+)" -o); do echo -e "$(kubectl $c --help | egrep '^\s+kubectl')"; done

2. Get Memory Usage Per Namespace

Some of us know that awk is so much more than awk { print $1 }. Recently, I dug deeper and learned the basics of this incredibly powerful language which came to us from some of the titans of Unix history. Here’s an example of a very useful awk program that’s still short enough to alias as a one-liner. It sums and prints the number of MiB of memory used by all pods in each namespace, and uses awk’s printf for clean output, even when your cluster contains long namespace names. Consider using awk when you need an aggregate view of kubectl output.

kubectl top pod -A --no-headers=true | awk '{a[$1] += $4} END {for (c in a) printf "%-35s %s\n", c, a[c]}'

3. View Secrets in Plain Text

When you’re developing an app that relies on secrets, it can be a chore to find a way to pipe the kubectl output to base64. Maybe you’ve written a shell pipeline or a small script to decode them. I’ve seen a few different options with various drawbacks: not supporting multi-line output, adding or dropping newlines along the way, or losing the context of the field names. With Ashley Schuett’s secret decode plugin for kubectl, you can view secrets in a readable form within the kubectl get output and avoid all of these issues.

The project documentation recommends using curl to grab the binary from GitHub and add it to our path. Since this touches a variety of sensitive data, I would encourage you to compile it from source, and review the source first. Luckily, this plugin consists of a single well-formatted source file:

git clone [email protected]:ashleyschuett/kubernetes-secret-decode.git
cd kubernetes-secret-decode
GO111MODULE=off go build -o $GOPATH/bin/kubectl-ksd

The name of the binary built into the last step must begin with kubectl- for it to be recognized as a plugin. Make sure that $GOPATH/bin is on your $PATH, and then you can use it like this:

kubectl ksd get secret -n <namespace-name> <secret-name> -o yaml

4. Recent and Previous Logs

If you have a service that can reach weeks or months of uptime, the container logs can grow to a size that make typical kubectl log commands produce a ridiculous amount of output. You might feel like you’re testing the limits of your terminal’s ability to paint text, or amazed by the amount of network traffic incurred by what’s supposed to be human-readable. In these cases, there are multiple ways to cut down the output and get to the messages you want to see.

First is the --tail=N flag, which begins printing the most recent N lines of the file.

kubectl logs --tail=10 -f -n kube-system -l component=kube-apiserver

There is also --since=time which accepts intervals of time such as 30s, 1h, or 5m to refer to a number of seconds, hours, or minutes respectively.

kubectl logs --since=5m -f -n kube-system -l component=kube-apiserver

Another log flag that I’ve come to rely on is -p. This prints the logs of the previous run of the container; the logs of a container that has exited. This would have made my first few days with Kubernetes much easier, as I attempted to restart pods and “catch” their logs, with some quick mouse and keyboard action, before they crashed again.

kubectl logs -p -n kube-system -l component=etcd

5. Save Time While You Wait

Whenever you’re tempted to write a loop with a sleep command, there’s usually a better way to do it. No matter what interval you choose, you’re likely to waste one or more seconds. This can add up if it’s an automated process. This is where the kubectl wait command and its associated apimachinery package is for. This command allows you to pause your scripts for the specific amount of time needed for the condition to be met in your cluster. You can also use it to post air horns in Slack the moment a new release hits the cluster. Have fun with it.

Here are some examples:

kubectl wait --for=condition=Available deployment/metrics-server

kubectl wait --for=condition=Initialized pod/csi-linode-controller-0

The conditions available vary based on the resource type and can be discovered using kubectl describe under the “Conditions:”. A given resource may have many different conditions at the same time.

Bonus Tricks!

Sometimes you look at output and wish that some additional networking or other info was displayed, right? To help, add -owide to the command. This works for almost every resource type, and the columns displayed can be customized for Custom Resource Definitions (CRDs). Examples:

kubectl get pods -A -owide

kubectl get services -A -owide

Last but not least, if you haven’t noticed by now, -A was added as an alias for the flag --all-namespaces. I’d guess that feature alone has saved the world millions of keystrokes since it was introduced. Another great timesaver is to add alias k=kubectl to your shell rc.

This post mostly touched on built-in kubectl features. For even greater depth, there is the open source kubectl book, readable online. It focuses on the use of Kustomize, a built-in tool which allows you to organize and compose your Kubernetes resources. I highly recommend it, especially if you’ve been looking for Kubernetes project frameworks with a minimally vulnerable surface area. For a collection of other useful commands, check out Krew, the official package manager for kubectl plugins.