Cluster Introspection
kubectl get services # List all services
kubectl get pods # List all pods
kubectl get nodes -w # Watch nodes continuously
kubectl version # Get version information
kubectl cluster-info # Get cluster information
kubectl config view # Get the configuration
kubectl describe node <node> # Output information about a node
Pod and Container Introspection
kubectl get pods # List the current pods
kubectl describe pod <name> # Describe pod
kubectl get rc # List the replication controllers
kubectl get rc --namespace="" # List the replication controllers in
kubectl describe rc <name> # Describe replication controller
kubectl get svc # List the services
kubectl describe svc <name> # Describe service
Interacting with Pods
kubectl run <name> --image=<image-name>
# Launch a pod called
# using image
kubectl create -f <manifest.yaml>
# Create a service described
# in
kubectl scale --replicas=<count> rc <name>
# Scale replication controller
# to instances
kubectl expose rc <name> --port=<external> --target-port=<internal>
# Map port to
# port on replication
# controller
Stopping Kubernetes
kubectl delete pod <name> # Delete pod
kubectl delete rc <name> # Delete replication controller
kubectl delete svc <name> # Delete service
kubectl drain <n> --delete-local-data --force --ignore-daemonsets # Stop all pods on
kubectl delete node <name> # Remove from the cluster
Debugging
kubectl exec <service> <command> [-c <$container>]
# execute on , optionally
# selecting container
kubectl logs -f <name> [-c <$container>]
# Get logs from service , optionally
# selecting container
watch -n 2 cat /var/log/kublet.log
# Watch the Kublet logs
kubectl top node
# Show metrics for nodes
kubectl top pod
# Show metrics for pods
Administration
kubeadm init
# Initialize your master node
kubeadm join --token <token> <master-ip>:<master-port>
# Join a node to your Kubernetes cluster
kubectl create namespace <namespace>
# Create namespace
kubectl taint nodes --all node-role.kubernetes.io/master-
# Allow Kubernetes master nodes to run pods
kubeadm reset
# Reset current state
kubectl get secrets
# List all secrets
Troubleshooting Networking Issues
IP forwarding
allows forwarding of the traffic coming from one interface to be routed to another interface.
is necessary for Linux kernel to route traffic from containers to the outside world.
otherwise Pod to service connection times out, tcpdump could show that lots of repeated SYN packets are sent, but no ACK is received.
# check that ipv4 forwarding is enabled
sysctl net.ipv4.ip_forward
# 0 means that forwarding is disabled
net.ipv4.ip_forward = 0
# this will turn things back on a live server
sysctl -w net.ipv4.ip_forward=1
# on Centos this will make the setting apply after reboot
echo net.ipv4.ip_forward=1 >> /etc/sysconf.d/10-ipv4-forwarding-on.conf
Bridge-netfilter
enables iptables rules to work on Linux bridges just like the ones set up by Docker and Kubernetes.
is necessary for the Linux kernel to be able to perform address translation in packets going to and from hosted containers.
otherwise network requests to services outside the Pod network will start timing out with destination host unreachable or connection refused errors.
# check that bridge netfilter is enabled
sysctl net.bridge.bridge-nf-call-iptables
# 0 means that bridging is disabled
net.bridge.bridge-nf-call-iptables = 0
# Note some distributions may have this compiled with kernel,
# check with cat /lib/modules/$(uname -r)/modules.builtin | grep netfilter
modprobe br_netfilter
# turn the iptables setting on
sysctl -w net.bridge.bridge-nf-call-iptables=1
echo net.bridge.bridge-nf-call-iptables=1 >> /etc/sysconf.d/10-bridge-nf-call-iptables.conf
Firewall rules block overlay network traffic
one of most common on-premise Kubernetes networking setups leverages a VxLAN overlay network, where IP packets are encapsulated in UDP and sent over port 8472.
there is 100% packet loss between pod IPs either with lost packets or destination host unreachable.
better to use the same protocol to transfer the data, as firewall rules can be protocol specific, e.g. could be blocking UDP traffic.
# on the server side
iperf -s -p 8472 -u
# on the client side
iperf -c 172.28.128.103 -u -p 8472 -b 1K
# to fix, update the firewall rule to stop blocking the traffic
AWS source/destination check
AWS performs source destination check by default. This means that AWS checks if the packets going to the instance have the target address as one of the instance IPs.
many Kubernetes networking backends use target and source IP addresses that are different from the instance IP addresses to create Pod overlay networks.
when problem arise, pod to service connection times out, tcpdump could show that lots of repeated SYN packets are sent, without a corresponding ACK anywhere in sight.
Pod CIDR conflicts
Kubernetes sets up special overlay network for container to container communication.
with isolated pod network, containers can get unique IPs and avoid port conflicts on a cluster.
problem arise when Pod network subnets start conflicting with host networks, pod to pod communication is disrupted with routing problems.
# start with a quick look at the allocated pod IP addresses
kubectl get pods -o wide
# compare host IP range with the kubernetes subnets specified in the apiserver
ip addr list
Kubernetes imposes the following fundamental requirements on any networking implementation (barring any intentional network segmentation policies):
- all containers can communicate with all other containers without NAT
- all nodes can communicate with all containers (and vice-versa) without NAT
- the IP that a container sees itself as is the same IP that others see it as