Kubesafe: Never run Kubernetes commands on the wrong cluster again

We’ve basically solved this where I work, with these steps:

- Each environment gets its own directory. We use kustomize to share config between environments.

- direnv “sets” the current context when you cd under a cluster’s directory (it sets an environment variable that a kubectl alias uses. Nobody calls kubectl directly; it wouldn’t work because we’ve banned the current context from the yaml files). You switch clusters by changing to that cluster’s directory.

- most of the time, the only command you run is ‘make’ which just does kubectl kustomize apply (or whatever). 100% of cluster config is checked into git (with git-crypt for secrets), so the worst that can happen is that you reapply something that’s already there.

I’ve also colored the command prompt according to the current cluster.

But anyway it’s essentially impossible to apply a config change to the wrong cluster. I haven’t worried about this in years.

I got burned by this recently and came to the conclusion that the concept of a current context is evil. Now I always specify —-context when running kubectl commands.

Don't keep anything in the default .kube/config. Set KUBECONFIG envar instead. Keep every cluster in separate config. Set an indicator in PS1. Helm et al follow the envar. Roast my zsh:

  k8x() {
    export env=$1;
  # exit if no param
    if [ -z $1 ]; then
      if [ -z ${KUBECONFIG+x} ]; then 
        echo "Need param of a k8s environment";
        return 1
      else 
        echo "Removing KUBECONFIG variable";
        PS1="$(echo "$PS1" | sed -e 's;^([^)]*) ;;')";
        unset KUBECONFIG;
        return 0
      fi
    fi;
  # exit if no file for param
    cfgPath="$HOME/.kube/config.${env}";
    if [ ! -f $cfgPath ]; then
      echo "A config does not exist";
      return 1
    fi;
    PS1="$(echo "$PS1" | sed -e 's;^([^)]*) ;;' -e 's;^;('$env') ;')";
    export KUBECONFIG="$cfgPath";
  }

In the early 1990s I ran a math department's 4 servers and 50 workstations and (with a few exceptions) only ever did administrative actions through scripts.

I've worked in lots of places since and the world's matured from scripts and rsync to ansible and puppet and similar.

Have we regressed to the point where we've turned big clusters of systems back into "oops I ran a command as superuser in the wrong directory" ?

Someone here showed me this cool technique with `fzf`:

    #!/usr/bin/env bash

    set -e

    context=$(kubectl config get-contexts | awk '{print $2;}' | grep -v NAME | fzf --preview 'kubectl config use-context {} && kubectl get namespaces')
    kubectl config use-context $context

Have been burnt by this, I have to deal with close to 8 clusters and it is very easy to make a mistake.

Would highly recommend kubie, it allows you to switch and shows you the name of the cluster in the prompt. It's probably a more visual way of solving the same problem.

https://github.com/sbstp/kubie

I like to print the k8s context and current namespace in the shell prompt.

It's still possible I could mess something up with kubectl, but it provides constant reminders of what I'm working with.

Hah! I accidentally deleted a production deployment the other day, because I thought it was mucking with my local Colima Kubernetes's cluster. I forgot that I had my context set to one of my AWS clusters. I had been meaning to write a command to wrap helm and kubectrl to prompt me with info before committing, so I will have to take a peek at this.

haha

i added the following to my bashrc a few days ago for similar reasons; this forces me to be explicit about the cluster; now i mess up the wrong namespace instead :)

    if [[ -e "/opt/homebrew/bin/kubectl" ]]; then
        /opt/homebrew/bin/kubectl config unset current-context >/dev/null
    fi

I am not trying to shit on this, sorry - but can't you achieve the same thing with rudimentary automation, and barring that, rudimentary scripting? This seems to just be adding y/n prompts to certain contexts. How's that different than a bash wrapper script that does something like this?

context=$(grep "current-context:" ~/.kube/config | grep "*prod*")

if [[ -z ${context} ]]

then # do the command

else # do a y/n prompt

fi

Am I missing something?

Can you use this with kubecolor? https://github.com/kubecolor/kubecolor

Incidentally: I have no idea why something like kubecolor isn't built in to kubectl itself.

I came up with a simpler solution that keeps kube contexts separated per terminal.

https://smlx.dev/posts/kubectl-global-state/

I handle this by never keeping production kubeconfigs on my local device. i pull them down on demand.

This seems good, but can it also be done via ACLs in vanilla Kubernetes?

Another option, just give prod's creds to CI only.