⏱️ Estimated Reading Time: 11-12 minutes
Patroni Test Lab Setup Guide
At 1:20 AM, your primary PostgreSQL node goes down. Applications freeze, connection pools exhaust, and failover doesn’t happen. The problem is not PostgreSQL — it’s the lack of a tested HA setup.
In production, PostgreSQL without a proven failover mechanism becomes a single point of failure. Downtime leads to transaction loss, SLA breaches, and emergency firefighting during peak hours.
This guide walks you through building a Patroni-based PostgreSQL HA test lab that behaves like production — allowing you to test leader election, failover, and recovery safely before going live.
Table of Contents
- Why You Must Monitor Patroni Clusters Daily
- Production-Ready Patroni Test Lab Setup
- Script Output & Analysis Explained
- Critical Components: Patroni Architecture Concepts
- Troubleshooting Common Patroni Issues
- How to Automate This Monitoring
- Interview Questions: Patroni Troubleshooting
- Final Summary
- FAQ
- About the Author
1. Why You Must Monitor Patroni Clusters Daily
- Leader Election Failure: No primary available, writes blocked.
- Replication Lag: Standby lag exceeds 5–10 seconds under load.
- Split Brain Risk: Two primaries due to DCS inconsistency.
- Application Impact: P99 latency spikes from 40ms to 5+ seconds.
2. Production-Ready Patroni Test Lab Setup
- 3 Linux VMs (Primary + 2 Replicas)
- PostgreSQL 14 or higher
- etcd or Consul as DCS
- Passwordless SSH between nodes
scope: pg-ha-lab
name: node1
restapi:
listen: 0.0.0.0:8008
connect_address: 10.0.0.1:8008
etcd:
host: 10.0.0.10:2379
bootstrap:
dcs:
ttl: 30
loop_wait: 10
retry_timeout: 10
maximum_lag_on_failover: 1048576
initdb:
- encoding: UTF8
- data-checksums
postgresql:
listen: 0.0.0.0:5432
connect_address: 10.0.0.1:5432
data_dir: /var/lib/postgresql/data
bin_dir: /usr/pgsql-14/bin
authentication:
replication:
username: replicator
password: repl_pass
superuser:
username: postgres
password: pg_pass
3. Script Output & Analysis Explained
| Check Component | Healthy State | Red Flags |
|---|---|---|
| Leader Status | Single primary | No leader / multiple leaders |
| Replication Lag | < 1 second | > 10 seconds |
| Failover Time | < 10 seconds | > 30 seconds |
4. Critical Components: Patroni Architecture Concepts
Distributed Configuration Store (DCS)
DCS (etcd/Consul) stores cluster state. If DCS is unhealthy, leader election fails.
Leader Election
Patroni ensures only one writable primary. Broken fencing leads to split-brain scenarios.
Replication Slots
Prevent WAL loss but can cause disk bloat if lag grows.
5. Troubleshooting Common Patroni Issues
Issue: No Primary After Restart
Symptom: All nodes in replica mode.
Root Cause: DCS unreachable.
Resolution:
- Check etcd health:
etcdctl endpoint health - Restart Patroni service
6. How to Automate This Monitoring
Method 1: Cron-Based Health Check
#!/bin/bash
curl -s http://localhost:8008/health | jq .
Method 2: Cloud Monitoring
Export Patroni metrics to Prometheus or CloudWatch.
Method 3: Third-Party Tools
Use Grafana dashboards for replication and failover visibility.
7. Interview Questions: Patroni Troubleshooting
A: By using a distributed configuration store and strict leader locks.
A: Patroni freezes leader changes to avoid data corruption.
A: Stop PostgreSQL on primary and observe leader promotion.
A: No. Patroni requires OS-level PostgreSQL access.
A: REST API, Prometheus exporters, and logs.
8. Final Summary
A Patroni test lab is mandatory before production rollout. It exposes real-world failure modes safely.
With proper monitoring and automation, Patroni delivers predictable PostgreSQL high availability.
- Always test failover
- Monitor DCS health
- Track replication lag
- Automate health checks
9. FAQ
A: Minimal overhead, mostly control-plane traffic.
A: Yes, widely used at scale.
A: Yes, with StatefulSets.
A: Weak fencing and no DCS monitoring.
A: Patroni is more automation-focused.
10. About the Author
x