Multi-cluster OpenShift environments introduce a configuration management problem that single-cluster deployments do not have: how do you ensure that your desired state, defined once, is consistently applied and maintained across every cluster in the fleet?
GitOps is the answer most practitioners reach for — and for good reason. But a GitOps implementation that works for one cluster does not automatically scale cleanly to ten or twenty.
The Drift Problem at Scale
Configuration drift occurs when the actual state of a cluster diverges from its declared desired state. In a single cluster, drift is detectable and recoverable. In a large fleet, drift can propagate silently — particularly when manual interventions, hotfixes or environment-specific changes are applied directly to clusters rather than through the GitOps pipeline.
Drift is not a sign of a bad team. It is a sign of a GitOps architecture that does not make the right path easy enough.
Sync Policies and Automated Remediation
ArgoCD's self-heal option enables automated synchronisation when drift is detected. This works well for stateless configuration but requires careful consideration for stateful workloads, network policies and persistent volume claims.
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
- PruneLast=trueSecret Management Across Clusters
Secret management is the most common failure point in multi-cluster GitOps architectures. Storing secrets in Git is not acceptable. Distributing secrets manually is not scalable. The recommended approach is to combine a secrets manager — HashiCorp Vault or AWS Secrets Manager — with an operator pattern that synchronises secrets into clusters on demand.
Rollback Architecture
A GitOps rollback is a Git operation: revert the commit, push the change, observe the sync. In practice, rollbacks in large fleet environments need to be selective — affecting one cluster or one application without disrupting others — and they need to account for dependencies between applications.