Hands-on: Tilt files, External Secrets Operator examples and a deliberately broken microservices app in sharifndlovu123/devops-directive-kubernetes-course. Resources are covered in Kubernetes Resource Types.

Kubernetes is only pleasant if the daily loop is fast, secrets are handled safely and you know how to find faults. This note covers all three.

1. The Inner Development Loop

Default flow: edit code, rebuild the image, push, redeploy, wait. Too slow. Tools that shorten it:

# Tiltfile (fragment)
docker_build('api', './api', live_update=[sync('./api/src', '/app/src')])
k8s_yaml('k8s/api.yaml')
k8s_resource('api', port_forwards=8080)

Tips: keep images small and layers cache-friendly (3. Container Images and Build Tools), use hot-reload tools (air for Go, nodemon for Node) inside the dev image, and keep the dev environment as close to production as you can.

2. Secrets Management

Configuration belongs in Git. Secrets cannot, so the question is how they reach the cluster.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata: { name: db-creds }
spec:
  refreshInterval: 1h
  secretStoreRef: { kind: ClusterSecretStore, name: cloud-secrets }
  target: { name: db-creds }
  data:
    - secretKey: password
      remoteRef: { key: prod/app/db-password }

Rules that always hold: - Encrypt Secrets at rest in etcd and restrict who can get secrets (RBAC) - Use workload identity (IAM roles for service accounts, Azure workload identity, GKE Workload Identity) so pods authenticate without stored keys - Rotate secrets; one secret per purpose and environment - More general advice: 5. Secure Coding and Supply Chain

3. Debugging Kubernetes

Real clusters misbehave in repeatable ways. Practise on a broken app (the course uses a microservices demo with intentional faults) before an outage forces you to.

The investigation flow

flowchart TD
    A[Something is wrong] --> B{Pod exists?}
    B -- no --> C[Check Deployment / ReplicaSet events]
    B -- yes --> D{Pod status}
    D -- Pending --> E[Scheduling: resources, taints, PVC, quota]
    D -- "ImagePullBackOff" --> F[Image name, tag, registry credentials]
    D -- CrashLoopBackOff --> G[kubectl logs --previous; config, secrets, probes]
    D -- Running but not Ready --> H[Readiness probe failing]
    D -- "Running and Ready" --> I{Reachable?}
    I -- no --> J[Service selector and endpoints, port, NetworkPolicy, Ingress]
    I -- yes --> K[Application logic: logs, metrics, traces]

Commands in order

$ kubectl get pods -A                          # what is not Running?
$ kubectl describe pod <name>                  # events at the bottom explain most problems
$ kubectl logs <pod> [-c container] --previous # logs, including the crashed container
$ kubectl get events --sort-by=.lastTimestamp
$ kubectl get endpoints <service>              # empty means the selector matches no ready pods
$ kubectl exec -it <pod> -- sh                 # look from the inside
$ kubectl port-forward svc/<service> 8080:80   # test a Service from your laptop
$ kubectl debug -it <pod> --image=busybox --target=<container>   # ephemeral debug container
$ kubectl top pods                             # CPU and memory (needs metrics-server)

Common faults and where they hide

Reading order that works: events, then logs, then configuration, then network. The Learnk8s "troubleshooting deployments" flow chart is a good reference to keep nearby.

Tools That Help

Cheat sheet of commands: 3. kubectl Cheat Sheet.