MySQL container uses readiness probe by running mysql -h 127.0.0.1 -e ‘SELECT 1’ on the server to make sure MySQL server is still active. Open a new terminal and simulate MySQL as being unresponsive by following command.
kubectl -n mysql exec mysql-1 -c mysql -- mv /usr/bin/mysql /usr/bin/mysql.off
This command renames the /usr/bin/mysql command so that readiness probe can’t find it. During the next health check, the pod should report one of it’s containers is not healthy. This can be verified by following command.
kubectl -n mysql get pod mysql-1
mysql-read load balancer detects failures and takes action by not sending traffic to the failed container, @@server_id 101. You can check this by the loop running in the separate window from previous section. The loop shows the following output.
Revert back to its initial state at the previous terminal.
kubectl -n mysql exec mysql-1 -c mysql -- mv /usr/bin/mysql.off /usr/bin/mysql
Check the status again to see that both containers are running and healthy
kubectl -n mysql get pod mysql-1
The loop in another terminal is now showing @@server_id 101 is back and all three servers are running.
Press Ctrl+C to stop watching.
To simulate a failed pod, delete mysql-1 pod by following command.
kubectl -n mysql delete pod mysql-1
StatefulSet controller recognizes failed pod and creates a new one to maintain the number of replicas with the same name and link to the same PersistentVolumeClaim.
kubectl -n mysql get pod mysql-1 -w
Press Ctrl+C to stop watching.