Explanation: Understanding cluster-manager (EKS) Infrastructure
Introduction
This document explains how ops/cluster-manager — the repository that provisions and operates W.W. Norton's Amazon EKS platform — is put together. It is written to answer one question that comes up constantly when you are debugging a certificate, a DNS record, or an egress path: "who owns this resource, and where do I change it?"
The platform is assembled from three sources: a Terraform layer, a GitOps (Flux) layer, and resources the Platform team manages outside it — in the ops/infrastructure repo or directly in the console. At every step, this document draws the line between what cluster-manager owns and what it merely consumes.
This is an explanation, not a runbook. For the commands to plan and apply an environment, see the Readme.md and the per-module readmes under modules/.
The Problem: One Repository, Three Sources of Truth
cluster-manager looks like a single repository, but the platform that actually runs is stitched together from three different sources of truth:
The catch is that nothing in the code makes that boundary obvious. The VPC, the Transit Gateways, the ACM certificates, and the Route53 records are all referenced by ID or ARN through Terraform data sources and tfvars — there is no terraform_remote_state link to follow back to the repository that owns them. An engineer who assumes "it's all in cluster-manager" can spend an afternoon looking for a resource that was never defined here.
The diagrams and ownership tables in this document exist to close that gap. A consistent color marks each resource's source of truth, in every diagram and table:
- 🟩 green — cluster-manager (Terraform): managed here, in this repo's Terraform.
- 🟪 violet — runtime (Flux / controllers): created in-cluster by Flux or by a controller (for example, the ALB controller), never written as Terraform.
- 🟦 blue — ops/infrastructure: managed in the sibling shared-infrastructure Terraform repo.
- 🟧 amber — manual / foundational: managed outside Terraform today (foundational / console), referenced by ID/ARN.
One team, four management surfaces. All of the AWS infrastructure here is owned by the Platform team — the colors distinguish how a resource is managed (this repo's Terraform, this repo's Flux, the ops/infrastructure repo, or manually via the console), not who owns it. The Platform team built even the amber resources that do not live in Terraform yet. The single exception is Entra ID (Azure AD), managed by the IT team; once identities reach AWS IAM Identity Center, they are in Platform's domain like everything else.
Architecture Overview
At the highest level, cluster-manager runs five EKS clusters across two AWS accounts, all in us-east-1, and feeds them through two delivery layers. The applications themselves live in separate wwnorton/app, wwnorton/lab, and wwnorton/media repositories and are delivered by Flux; the shared AWS services they depend on (DNS, certificates, databases, object storage) are managed by the Platform team in the ops/infrastructure repo.
The Two-Layer Delivery Model
Every cluster is built and kept alive by two cooperating layers, and where you make a change depends on which one owns it:
- Layer 1 — Terraform is applied by manual GitLab CI pipelines that authenticate to AWS via GitLab OIDC → IAM role assumption (no static keys). The two root modules are
deploy/<tier>/deploy_eks(cluster, network, IAM, add-ons) anddeploy/<tier>/deploy_k8s(in-cluster platform Helm + RBAC + Flux bootstrap). These call the reusablemodules/eks/*andmodules/k8s/*. State is stored as GitLab-managed Terraform state, one state per<env>-<component>. - Layer 2 — Flux GitOps is bootstrapped by Terraform and then continuously reconciles
clusters/tf-<env>-cluster/from this repository. Each application is a FluxGitRepositorysource pointing at awwnorton/app,wwnorton/lab, orwwnorton/mediadeploy repository, plus aHelmRelease.
The <tier> is one of lower (parameterized to serve dev, qa, and stg from one module), ops (serves ops), or prod.
The dividing line that keeps you safe. Application workloads are GitOps-managed, never Terraform-applied. That separation means a noisy, frequent application deploy can never reach into the cluster's IAM or networking — those live one layer down, behind a reviewed Terraform plan and a manual apply.
How a Change Reaches a Running Pod
The two layers hand off in time, not just in space. A platform change and an application change travel different paths to the same cluster:
Terraform runs once per change, on demand. Flux runs forever, on a loop — which is why drift in an application is corrected automatically, while drift in the platform substrate waits for the next intentional pipeline run.
The AWS Network and EKS Layer
Each cluster lives inside a pre-existing VPC — referenced, never created — and layers its own resources on top: subnets, egress, private endpoints, the EKS control plane, a node group, and IAM. The lower environments and production are built by different modules and have meaningfully different network shapes, so they are drawn separately below.
In every environment, what cluster-manager creates inside the VPC is the EKS-tier subnets and their route tables, the VPC endpoints (ec2, ecr.api, ecr.dkr, logs, s3), the node security groups, the EKS cluster (private-only API, all five control-plane log types), the managed node group on a custom launch template, the EKS IAM roles, and the IRSA OIDC provider. What it does not create is the VPC itself, the Internet Gateway, the Transit Gateways, and the NAT Elastic IPs — those are referenced by ID, tag, or IP.
Lower environments (dev / qa / stg)
A single module (modules/eks/lower) builds all three lower clusters identically: three availability zones, a NAT gateway for egress, and a private-only EKS API. The three private /22 and three public /24 subnets are tagged for EKS and ELB so the load-balancer controller can discover them.
Ops (the observability hub)
The ops cluster (modules/eks/ops) mirrors the lower network shape — three availability zones with NAT egress — but plays a special role: it is the central observability hub, aggregating logs, metrics, and traces from every other environment and running the Loki/Tempo/Thanos/Grafana backends (covered in Observability below).
An easy-to-miss quirk: ops has no subnet tags. Unlike every other environment, the ops cluster's private subnets carry no kubernetes.io/* tags. Subnet auto-discovery therefore does not work there, so ingress objects must name their subnets or target group explicitly. If an ALB fails to provision in ops with a "no subnets found" error, this is why.
Production
Production (modules/eks/prod) is the odd one out by design. It spans five availability zones, has no NAT gateway at all, and fronts traffic with both an internal and an internet-facing Application Load Balancer.
Two shapes of egress
The single biggest structural difference between the environments is how a private worker node reaches the internet. The lower and ops clusters take the familiar NAT path; production routes everything through the shared Transit Gateway instead:
lower / ops — NAT egress
prod — Transit Gateway egress
No NAT gateway — and no NAT Elastic IP — exists in the production account. Its private route table's default 0.0.0.0/0 route points at the Transit Gateway, so all egress is centralized. Looking for a NAT resource in production is a dead end.
GuardDuty is not managed here. Every cluster runs an aws-guardduty-agent EKS add-on and a guardduty-data VPC endpoint that this repository does not create. GuardDuty EKS Runtime Monitoring is enabled at the account/organization level, outside this repo's Terraform. Treat it as managed elsewhere when you reason about add-ons, and do not try to manage it from cluster-manager.
How traffic gets in
Ingress is the mirror image of egress, and the two load-balancer types serve genuinely different audiences:
- Internal ALBs sit in the private subnets and front the apps for AWS Client VPN users, who resolve the private Route53 zone (
wwnorton.netin lower,wwnorton.comin prod). This is the path for internal tooling and for engineers reaching internal services. - Internet-facing ALBs sit in the public subnets. In production they serve public customer traffic; in the lower environments they exist so that third-party providers (in the
external-servicesnamespace) can reach specific internal apps from outside the VPN when a business integration requires it.
Environment Differences
The values below are the live state confirmed against AWS (June 2026); node instance types and counts come from the launch templates and tfvars.
| Aspect | dev / qa / stg (lower) | ops | prod |
|---|---|---|---|
| AWS account | 637244866643 | 637244866643 | 100478842646 |
| EKS version (live) | 1.35 | 1.35 | 1.34 |
| Node instance | m5.2xlarge (stg m5.4xlarge) | m5.2xlarge | m5.4xlarge |
| Node scaling (min/max) | 2 / 6 (stg max 16) | 0 / 6 | 12 / 16 |
| Availability zones | 3 (1a/1b/1c) | 3 | 5 (1a/1b/1c/1d/1f) |
| NAT gateway | 1 (created here) | 1 (created here) | none — egress via Transit Gateway |
| Application Flux releases | ~93–105 | platform tooling only | ~94 |
| Extra add-ons | dev: adot | metrics-server | — |
| DNS domain | <svc>.<env>.wwnorton.net | <svc>.ops.wwnorton.net | <svc>.wwnorton.com / <svc>.prod.wwnorton.com |
GitOps Delivery of Applications
Once Terraform has bootstrapped Flux, Flux CD owns application delivery: it continuously reconciles clusters/tf-<env>-cluster/ from this repository. Each application is a GitRepository source pointing at a wwnorton/app, wwnorton/lab, or wwnorton/media repository, plus a HelmRelease; an application cluster runs roughly 90–105 of these in per-team namespaces. The numbered path below traces one change from commit to running pod.
The In-cluster Platform Stack
The deploy_k8s layer (modules/k8s/*) installs the platform Helm releases and the Kubernetes governance objects that every application depends on. The diagram groups them by function and shows the AWS resources each one reaches through IRSA.
- Ingress & TLS —
aws-load-balancer-controllermaterializes ALBs at runtime fromIngressobjects (which is why they are violet, not green); they are fronted by ACM certificates and reached through Route53 controlled subdomains, both managed inops/infrastructure. - Autoscaling —
cluster-autoscalerscales the managed node group, andkedadrives event-based workload autoscaling. - Secrets —
secrets-store-csi-driver(with the AWS provider) mounts AWS Secrets Manager values into pods via IRSA. - Observability agents —
kube-prometheus-stackand the OpenTelemetry Collector forward telemetry to the ops hub (next section). - Operations tooling —
gitlab-agent(KAS),kubecost/opencost, andreloader. - Governance — namespaces, per-namespace
LimitRangepolicies, ClusterRoles/RBAC, and IRSA-annotated ServiceAccounts.
Observability: A Hub-and-Spoke Model
Observability follows a hub-and-spoke pattern. Each application cluster is a spoke that ships metrics, logs, and traces to the central tf-ops-cluster hub, which stores them in S3 and surfaces them through Grafana.
- Metrics — each cluster's
kube-prometheus-stackships blocks via its Thanos sidecar to S3; the ops cluster's Thanos store/query then federates acrossprometheus.<env>.wwnorton.net/…wwnorton.com. - Logs & traces — OpenTelemetry collectors send OTLP to the ops cluster's Loki (logs) and Tempo (traces), which persist to S3.
- Dashboards — Grafana stores its own dashboards and users in the external RDS
poc-grafanadatabase and reads Thanos, Loki, Tempo, and CloudWatch as datasources. The ops cluster additionally runs DevLake, SonarQube, and Kubeshark. - MongoDB — application MongoDB Atlas databases are scraped by the
mongodb-exporter(a Prometheus exporter); the same Atlas databases are also consumed directly by application workloads.
There is no Grafana in production. Grafana runs only in the ops cluster, in the development account (637244866643). Production ships its metrics, logs, and traces to the ops hub and has no Grafana of its own — grafana_prod is simply a database inside the shared poc-grafana RDS instance, not a second Grafana deployment. Looking for a Grafana UI in the production account is another dead end.
The S3 buckets and the RDS instance are external to this repository — cluster-manager only grants IRSA access to them; ops/infrastructure manages them.
Identity and Trust
Two OIDC trust paths govern access to the platform: one for the pipeline that builds everything, and one for the workloads that run on it. They are worth understanding separately because they fail in different ways.
- Pipeline OIDC — the GitLab CI job presents an
id_token(aud=https://gitlab.com) and assumescluster-manager-pipeline-{dev,prod}, an IAM role defined inops/infrastructure(aws/iam/gitlab_oidc_extra_roles.tf, PLAT-1039). It is mapped tosystem:mastersin each cluster'saws-auth, so it is both the AWS-API identity and the EKS cluster-admin identity. - IRSA — each cluster's own OIDC provider lets IRSA-annotated ServiceAccounts assume scoped roles (the ALB controller, cluster-autoscaler, per-app secret access, and the ops S3 roles), reading only the
env/ns/apppaths they are granted. - Human access — engineers reach the clusters through AWS IAM Identity Center (SSO) and, in dev, the Azure AD (Entra ID) OIDC provider, mapped to the
developer/reporter/superadminRBAC groups. Entra ID is managed by the IT team; the AWS IAM Identity Center setup and the cluster RBAC mapping are Platform's.
The pipeline role is the most powerful identity in the platform. Because cluster-manager-pipeline-{dev,prod} is both an AWS-API principal and system:masters inside every cluster, anything that can run the pipeline can administer the clusters. That is why the role is defined and trust-scoped in ops/infrastructure rather than here, and why it deserves attention in any access review. IRSA roles, by contrast, are deliberately narrow.
Source of Truth: Where Each Resource Is Managed
This is the canonical answer to "where do I change this resource?" Every component the platform depends on is listed below, grouped by management surface and color-coded to match the diagrams. As before, the groups describe how each resource is managed, not who — it is all Platform team, except Entra ID.
🟩 Managed here — cluster-manager (Terraform)
| Resource group | Where in the repo |
|---|---|
| EKS clusters (6 defined, 5 deployed) | modules/eks/* |
| Managed node groups + launch templates | modules/eks/* |
| EKS-tier subnets (private/public) + route tables | modules/eks/* |
| NAT gateway (lower/ops only) | modules/eks/{lower,ops} |
VPC endpoints (ec2, ecr.api, ecr.dkr, logs, s3) | modules/eks/* |
| Node security groups | modules/eks/* |
| EKS IAM (cluster/worker/EBS-CSI roles), IRSA OIDC provider, per-app IRSA roles | modules/eks/*, modules/k8s/* |
EKS managed add-ons (ebs-csi, coredns, kube-proxy, cloudwatch-observability; ops metrics-server) | modules/eks/* |
| Platform Helm stack (ALB controller, cluster-autoscaler, gitlab-agent, kube-prometheus-stack, OpenTelemetry, secrets-store-csi, reloader, keda, kubecost, opencost; ops: Loki/Tempo/Thanos/DevLake/SonarQube/Kubeshark) | modules/k8s/* |
Namespaces, LimitRange, ClusterRoles/RBAC, IRSA ServiceAccounts, pull secrets | modules/k8s/* |
| Flux bootstrap | modules/k8s/* |
🟪 Created at runtime (Flux / controllers)
| Resource group | Created by |
|---|---|
| Application workloads (~90–105 HelmReleases per app cluster) | Flux, from clusters/tf-*-cluster/ + wwnorton/{app,lab,media} |
Runtime ALBs (k8s-tf*alb, internal & internet-facing) | aws-load-balancer-controller, from Ingress |
GuardDuty agent (DaemonSet) + guardduty-data endpoint | GuardDuty EKS Runtime Monitoring (account/Org) |
🟦 Managed in ops/infrastructure (Platform team)
| Resource group | Where (infrastructure repo) |
|---|---|
| Route53 hosted zones + records (incl. controlled subdomains) | aws/route53/ |
| ACM / ACM-PCA certificates | aws/acm/ |
Shared IAM — GitLab OIDC pipeline roles (cluster-manager-pipeline-{dev,prod}) | aws/iam/ |
| S3 buckets (Loki / Tempo / Thanos object storage) | aws/s3/ (IRSA access granted in cluster-manager) |
RDS poc-grafana (Grafana backend DB) | aws/rds/ |
Secrets Manager entries (read via CSI / data sources) | aws/secretsmanager/ |
| MongoDB Atlas projects | mongodb/projects/ |
🟧 Platform-managed outside Terraform (referenced by ID/ARN)
Built and owned by the Platform team, but not (yet) under this repo's Terraform — referenced by ID/ARN.
| Resource group | Notes |
|---|---|
VPCs (e.g. vpc-…rancher-lower, prod vpc-0e9a40e45f62a82ab) | Pre-existing (Rancher-era); referenced by data source |
Transit Gateways (tgw-0b15…c6655, tgw-077f…da8f) | Shared network backbone |
| Internet Gateways | Referenced by ID/tag |
| NAT Elastic IPs (lower/ops) | Referenced by public IP |
| Azure AD / Entra ID OIDC IdP | Managed by the IT team — the one dependency outside Platform; feeds human cluster auth |
| ECR repositories | Images pulled privately via the ECR VPC endpoints |
| GitLab-managed Terraform state backend | One state per <env>-<component> |
Account and Environment Reference
- Region:
us-east-1(all clusters). - Accounts:
637244866643(dev, qa, stg, ops) and100478842646(prod). - Clusters:
tf-dev-cluster,tf-qa-cluster,tf-stg-cluster,tf-ops-cluster,tf-prod-cluster. - Authentication: GitLab CI assumes AWS IAM roles via OIDC (
AWS_ROLE_ARN_<env>); human access via SSO (AWSReservedSSO_*) and the Azure AD OIDC provider mapped to RBAC groups.
Design Rationale
This section explains why the platform is shaped the way it is — the reasoning behind decisions you will run into, not advice on how to change them.
Why two layers instead of one
Terraform suits the slow-moving substrate — clusters, IAM, network, add-ons — where a reviewed plan and a manual apply are the right safety model. Application delivery moves far faster and wants the continuous reconciliation and drift correction Flux provides. Matching each tool to its tempo is the whole reason for the split.
Why data sources instead of terraform_remote_state
The VPCs, Transit Gateways, and gateways predate this repository (several are Rancher-era) and follow different lifecycles. Referencing them by ID/tag through data sources avoids coupling cluster-manager's state to a network repository's state — at the cost of making the boundary implicit, which is the gap this document closes.
Why private-only API endpoints
Every cluster's Kubernetes API is private-only, with all five control-plane log types enabled. Access is via the GitLab agent and in-VPC tooling, so there is no public API surface to harden in the first place.
References
Internal (Norton)
- Repository:
ops/cluster-manager·Readme.md - Diagram sources (D2
.d2+ rendered SVG):docs/architecture/ - Shared infrastructure:
ops/infrastructure— see Understanding Route53 Infrastructure and Understanding RDS Infrastructure - Observability detail:
docs/loki/README.md
External (AWS, Flux, CNCF)
- Amazon EKS User Guide
- IAM Roles for Service Accounts (IRSA)
- Flux Documentation
- AWS Load Balancer Controller