Galera has been promoted as a WAN Cluster solution for many years. Hopeful sysadmins have deployed it, envisioning a future where their database can survive an entire site failure. Seasoned database administrators say Galera isn't reliable outside a single data center or extremely reliable metro area.
The reason your entire Galera WAN cluster went Non-Primary is packet loss.
"But I deployed three nodes, and only one experienced packet loss! I still have a two-node quorum."
Wrong. For Galera to work, you not only need a quorum of nodes online, you also need every node to agree unanimously on which nodes are functioning parts of the cluster.
If your working nodes don't agree unanimously, the good news is that they will try three times! The bad news is you may still get called in the middle of the night to fix your "high availability" WAN cluster.
Let's calculate your probable fate!