KUBERNETES
AND GITOPS

Problem

Enterprise teams adopting Kubernetes commonly reach a point where the cluster itself is no longer the hard part. The infrastructure exists, but getting a change from a developer's laptop into production safely, repeatedly, and with a clear audit trail is not solved by kubectl apply. Manual deployments create drift between what is running and what is declared in source control, rollbacks depend on someone remembering the last-known-good configuration, and basic questions like "who changed this, and when" are hard to answer once enough services and teams share a cluster.

Approach

The recurring approach across these engagements has been to treat Git as the single source of truth for both infrastructure and application state, and to let a GitOps controller reconcile the cluster to match it rather than pushing changes imperatively:

• Infrastructure as code. Terraform provisions the Kubernetes cluster on AWS along with its supporting networking and IAM resources, so cluster-level changes go through the same review and plan/apply discipline as application code.
• GitOps delivery. Argo CD watches Git repositories containing Helm charts and Kubernetes manifests and continuously reconciles the live cluster state to match them. A deployment is a merged commit, not a manual step, and the running state of the cluster is always diffable against Git.
• Access control at the cluster layer. Namespace-scoped RBAC and Kubernetes-native access controls determine who can act on which workloads, so cluster access follows the same reviewable configuration model as everything else deployed through GitOps.
• CI as a gate, not a bypass. CI pipelines build, test, and package changes into container images and chart versions, but do not deploy directly. Deployment authority stays with the GitOps controller reading from Git, which keeps the pipeline's job limited to producing a reviewable, versioned artifact.

Architecture

Developer PR --> CI (build, test, package image + chart)
                     |
                     v
              Merge to Git (source of truth)
                     |
                     v
              Argo CD reconciliation loop
                     |
        +------------+-------------+
        |                          |
        v                          v
  Kubernetes cluster (Terraform-  Helm-templated
  provisioned on AWS: networking,   workloads,
  IAM, node groups)                 ingress, RBAC
        |                          |
        v                          v
  Namespace-scoped RBAC gates   Observability
  cluster + Argo CD access      (metrics, logs)
                                 feed rollout health

Deployment Safety and Tradeoffs

Because Argo CD continuously reconciles against Git, a manual kubectl edit against a live resource is corrected back to the Git-declared state on the next sync rather than silently persisting as drift. That property is valuable for auditability but has a real cost: it removes the option of a quick manual hotfix under pressure, and teams have to be deliberate about sync windows, health checks, and automated pruning so that a broken commit does not get reconciled into production before anyone notices. Rollback becomes a Git revert rather than a bespoke recovery procedure, which is simpler to reason about but only as reliable as the health checks gating promotion; a rollout that passes a shallow readiness probe but is functionally broken will still get promoted.

Centralizing deployment authority in a GitOps controller also concentrates risk: the reconciliation loop's scope and permissions have to be deliberately bounded, since a controller with cluster-wide write access is a single point of failure for both good and bad changes. Observability has to cover the reconciliation process itself, not just the workloads it deploys, so that a stuck or misbehaving sync is visible before it is mistaken for a healthy, idle state.

Outcome and Lessons

The consistent lesson across these engagements is that GitOps and infrastructure-as-code do not remove operational judgment, they relocate it. Instead of deciding whether a deployment looks safe at the moment of kubectl apply, the judgment goes into what the CI pipeline verifies before merge, what the readiness and liveness checks actually test, and how tightly the reconciliation loop's scope and permissions are bounded. Getting that judgment right, and documenting it so the next engineer does not have to rediscover it by reading Argo CD sync logs at 2 a.m., is the actual work behind "we use GitOps."