Able to insert data against node3 after shutting down node1 and node2
I have a 3 node cluster. I shutdown mysql (/etc/init.d/mysql stop) on node1 and node2. To my surprise I am able to login to mysql on node3 and perform inserts, create tables and databases. I was expecting an error. Please advice.
Yes, graceful stop of a node causes reconfiguration of the cluster and the stopped node is not considered as lost, but as intentionally taken out of cluster.
Even on a two node cluster - if you stop gracefully one of them, so it has a chance to say to peer "I want to leave, goodbye" - the remaining node forms a new cluster and is still primary. Node crash, network split or data inconsistency are the possible reasons when cluster decides whether to keep operating or not, based on if it still has the quorum.