Developer deployment guide

Introduction

This guide provides detailed instructions for deploying applications to W.W. Norton's lower-environment clusters. It covers the complete deployment workflow, from setting up your deployment repository to monitoring your application in the cluster.

Prerequisites

Before starting the deployment process, ensure you have:

• A working application with containerized build process
• Access to W.W. Norton's GitLab organization
• Your application's GitLab CI pipeline configured and working
• Required access to the cluster-manager repository
• Basic understanding of Kubernetes, Helm, and FluxCD concepts

Deployment Repository Structure

Your application requires a dedicated deployment repository (-deploy) with the following structure:

<appname>-deploy/
├── CODEOWNERS                 # Define repository maintainers
├── README.md                  # Repository documentation
├── chart/                    # Helm chart directory
│   ├── Chart.yaml            # Chart metadata and version info
│   ├── templates/            # Kubernetes manifest templates
│   │   ├── NOTES.txt        # Usage notes
│   │   ├── _helpers.tpl     # Template helpers
│   │   ├── deployment.yaml  # Pod deployment configuration
│   │   ├── ingress.yaml    # Ingress configuration
│   │   ├── pvc.yaml        # Persistent Volume Claims
│   │   ├── service.yaml    # Service configuration
│   │   └── tests/          # Helm tests
│   └── values.yaml          # Default values for templates
├── cluster-config/          # Optional cluster-specific configs
│   ├── pv-prod.yaml        # Production persistent volumes
│   └── pv.yaml             # Development persistent volumes
└── flux/                   # Environment-specific values
    ├── dev/
    │   └── values.yaml    # Development environment values
    ├── prod/
    │   └── values.yaml    # Production environment values
    ├── qa/
    │   └── values.yaml    # QA environment values
    └── stg/
        └── values.yaml    # Staging environment values

Key Components

1. CODEOWNERS
   • Defines who can approve changes to the repository
   • Example format:

     * @platform-team
     /chart/ @platform-team @app-team

1. Chart Directory
   • Contains your Helm chart configuration
   • Defines how your application will be deployed
   • Includes all necessary Kubernetes manifests as templates
2. Cluster Config Directory (optional)
   • Optional directory for persistent volume and other specific configurations for the application.
   • Required only if your application needs persistent storage or has extra special needs
   • Contains environment-specific extra configurations that needs to be present in the cluster
3. Flux Directory
   • Contains environment-specific values
   • Each subdirectory represents a deployment environment
   • Values files override default chart values for each environment

Version Control Practices

1. Branch Strategy
   • Main branch is protected
     1. By default, "Allowed to push and merge" in Deploy Project Settings Branch Rules' Protect Branch only applies to "Maintainers". This needs to be updated to "Developers and Maintainers" to allow Developers to trigger pipelines in Deploy projects.
   • All changes must go through Merge Requests
   • Environment-specific changes should use feature branches
2. Change Management
   • All changes must be reviewed by designated approvers
   • Changes affecting platform configurations require platform team approval
   • Documentation updates should accompany configuration changes

Helm Chart Configuration

Chart.yaml Configuration

The Chart.yaml file defines your application's metadata and dependencies:

apiVersion: v2
name: <application-name>
description: <application-description>
type: application
version: 0.1.0          # Chart version
appVersion: "1.16.0"    # Application version

Version Management

• version: Increment when changing chart configuration
• appVersion: Should match your application's version
• Follow semantic versioning (MAJOR.MINOR.PATCH)

Deployment Template

The deployment.yaml template defines how your application runs in the cluster. Here's the complete template structure:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: <application-name>
  labels:
    {{- include "chart.labels" . | nindent 4 }}
spec:
  {{- if not .Values.autoscaling.enabled }}
  replicas: {{ .Values.replicaCount }}
  {{- end }}
  selector:
    matchLabels:
      {{- include "chart.selectorLabels" . | nindent 6 }}
  template:
    metadata:
      {{- with .Values.podAnnotations }}
      annotations:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      labels:
        {{- include "chart.selectorLabels" . | nindent 8 }}
    spec:
      imagePullSecrets: 
        - name: "{{ .Values.app.image.pullSecret }}"
      volumes:
        - name: env-creds
          secret:
            secretName: <application-name>-creds
        - name: <other-secret>
          secret:
            secretName: <other-secret-name>
            defaultMode: 420
        - name: persistent-storage
          persistentVolumeClaim:
            claimName: <name-of-pvc>

      containers:
        - name: {{ .Chart.Name }}
          image: "{{ .Values.app.image.name }}"
          imagePullPolicy: "{{ .Values.app.image.pullPolicy }}"
          volumeMounts:
            - name: env-creds
              mountPath: "/mnt/creds"
              readOnly: true
            - name: <other-secret>
              mountPath: "/mnt/certs"
              readOnly: true
            - name: persistent-storage
              mountPath: /var/www/html/<path>
              subPath: <subpath>
          ports:
            - name: http
              containerPort: {{ .Values.service.port }}
              protocol: TCP
          readinessProbe:
            {{- toYaml .Values.readinessProbe | nindent 12 }}
          env:
              - name: DEPLOY_ENV
                value: "{{ .Values.app.env }}"
              - name: NODE_ENV
                value: "{{ .Values.app.env }}"
              - name: CREDS_PATH
                value: /mnt/creds
              - name: GRAYLOG_HOST
                value: logs.wwnorton.com
              - name: GRAYLOG_PORT
                value: '12201'
          resources:
            {{- toYaml .Values.resources | nindent 12 }}
      {{- with .Values.nodeSelector }}
      nodeSelector:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      {{- with .Values.affinity }}
      affinity:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      {{- with .Values.tolerations }}
      tolerations:
        {{- toYaml . | nindent 8 }}
      {{- end }}

Key Values Configuration

The corresponding values.yaml configurations needed for this deployment template:

1. Application Configuration

app:
  replicaCount: <number>
  image:
    name: <registry-path>/<image-name>:<tag>
    pullPolicy: <Always|IfNotPresent|Never>
    pullSecret: image-pull-secret
    rollback: <previous-image-tag-for-rollbacks>
  env: <environment-name>

1. Health Check Configuration

readinessProbe:
  httpGet:
    path: <health-check-endpoint>
    port: http
  initialDelaySeconds: <delay>
  periodSeconds: <interval>
  timeoutSeconds: <timeout>
  successThreshold: <success-count>
  failureThreshold: <failure-count>

1. Service Configuration

service:
  type: <ServiceType>
  port: <port-number>

1. Resource Management

resources:
  limits:
    cpu: <cpu-limit>
    memory: <memory-limit>
  requests:
    cpu: <cpu-request>
    memory: <memory-request>

1. Autoscaling Configuration

autoscaling:
  enabled: <true|false>
  minReplicas: <min-pods>
  maxReplicas: <max-pods>
  targetCPUUtilizationPercentage: <target-cpu>

Important Notes:

1. Secret Requirements
   • <application-name>-creds: Application-specific credentials
   • image-pull-secret: Container registry authentication
   • All secrets must exist in cluster before deployment
2. Storage Configuration (optional)
   • Uses EFS for persistent storage
   • Requires a PVC created in the cluster
   • Configure appropriate mount paths and sub-paths
3. Health Checks
   • Configure readinessProbe based on application health endpoint
   • Adjust timing parameters based on application startup requirements
   • Consider initialization time when setting delays
4. Environment Variables
   • Set appropriate environment values
   • Configure external service endpoints
   • Enable/disable features as needed
5. Resource Allocation
   • Set appropriate CPU and memory limits
   • Consider application requirements
   • Plan for scalability
6. Image Management
   • Will be handled by the CI pipeline
   • Make sure to copy and paste the .gitlab-ci.yml file from any other -deploy repository into yours.

Service Template

The service.yaml template defines how your application is exposed within the cluster. Here's the standard template structure:

apiVersion: v1
kind: Service
metadata:
  name: {{ .Values.service.name | default "<application-name>" }}
  labels:
    run: {{ .Values.service.name | default "<application-name>" }}
spec:
  type: {{ .Values.service.type }}
  ports:
    - port: {{ .Values.service.port }}
  selector:
    run: {{ .Values.service.name | default "<application-name>" }}

Key Values Configuration

1. Service Configuration

service:
  name: <application-name>  # Optional, defaults to application name
  type: ClusterIP          # Service type
  port: <port-number>      # Application port number

Important Notes:

1. Naming Convention
   • Service name should match your application name
   • Consistent naming across labels and selectors
   • Default fallback if name not specified
2. Service Type
   • ClusterIP: Internal cluster access (default)
   • NodePort: Static port on each node (optional)
   • LoadBalancer: External cloud provider load balancer (optional)
3. Port Configuration
   • Must match container port in deployment
   • Consider standard ports for your application type

Ingress Template

The ingress.yaml template configures external access to your application. Here's the standard template structure:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  labels:
    {{- include "chart.labels" . | nindent 4 }}
  annotations:
    {{- toYaml .Values.ingress.annotations | nindent 4 }}
  name: "{{ default .Chart.Name $.Values.ingress.name }}-ingress"
spec:
  ingressClassName: {{ .Values.ingress.className }}
  rules:
  - host: {{ .Values.ingress.host }}
    http:
      paths:
        {{- range .Values.ingress.paths }}
        - path: {{ .path }}
          pathType: {{ .pathType }}
          backend:
            service:
              name: "{{ default $.Chart.Name $.Values.service.name }}"
              port:
                number: {{ $.Values.app.port }}
        {{- end }}
  tls:
    {{- toYaml .Values.ingress.tls | nindent 4 }}

Key Values Configuration

1. Basic Ingress Configuration

ingress:
  enabled: true
  className: "alb"
  name: <application-name>  # Optional, defaults to Chart.Name
  host: <environment-domain>
  paths:
    - path: /<application-path>
      pathType: Prefix

1. AWS ALB Annotations

  annotations:
    alb.ingress.kubernetes.io/scheme: <internal|external>
    alb.ingress.kubernetes.io/group.name: <alb-group-name>
    alb.ingress.kubernetes.io/subnets: <subnet-1>, <subnet-2>, <subnet-3>
    alb.ingress.kubernetes.io/security-groups: <security-group-id>
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/certificate-arn: <certificate-arn>
    alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS13-1-2-2021-06
    alb.ingress.kubernetes.io/healthcheck-path: /<application-path>/healthcheck

Important Notes:

1. Naming and Defaults
   • Ingress name uses application name with -ingress suffix
   • Service name defaults to Chart.Name
   • Port number from application configuration
2. Path Configuration
   • Use Prefix type for most applications
   • Path should match application context
   • Health check path follows service path
3. ALB Configuration
   • Internal scheme recommended
   • Group name based on environment
   • Subnets must be in cluster VPC
   • Security groups control access
4. SSL/TLS
   • HTTPS required by default
   • Certificate from AWS Certificate Manager
   • Modern TLS policy enforced
5. Health Checks
   • Custom path per application
   • Must return HTTP 200
   • Follows application path structure

FluxCD Configuration

When your Helm chart is ready for deployment, you'll need to configure FluxCD to manage your application in the cluster. This requires two main components: a GitRepository and a HelmRelease.

GitRepository Configuration

The GitRepository resource tells Flux where to find your deployment repository.

apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
  name: <application-name>-deploy
  namespace: flux-system
spec:
  interval: 1m
  url: <gitlab-repository-url>
  ref:
    branch: <branch-name>
  secretRef:
    name: flux-git-auth

HelmRelease Configuration

The HelmRelease resource defines how Flux should deploy your Helm chart.

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: <application-name>
  namespace: <target-namespace>
spec:
  interval: 1m
  chart:
    spec:
      chart: "./chart"
      reconcileStrategy: Revision
      sourceRef:
        kind: GitRepository
        name: <application-name>-deploy
        namespace: flux-system
      valuesFiles:
        - "./flux/<environment>/values.yaml"

Key Configuration Elements

1. GitRepository Settings

spec:
  interval: <sync-interval>
  url: https://gitlab.com/wwnorton/app/<your-path>/<repo-name>
  ref:
    branch: <branch-name>  # typically 'main'

1. HelmRelease Settings

spec:
  chart:
    spec:
      chart: "./chart"  # Path to chart in repository
      valuesFiles:
        - "./flux/<environment>/values.yaml"

Important Notes:

1. File Placement
   • Place configurations in:

     ./clusters/<cluster-name>/flux-config/

• Use descriptive filenames:

  • <application-name>-deploy-repo.yaml
  • <application-name>-deploy-release.yaml

• Repository Access

  • Requires GitLab repository access
  • Uses flux-git-auth secret (do not modify)
  • Ensure proper permissions

• Synchronization

  • Default 1-minute sync interval
  • Adjustable based on needs
  • Consider resource implications

• Environment Handling

  • Separate values per environment
  • Path structure must match
  • Consider environment variables

• Namespace Management

  • Use appropriate target namespace
  • Ensure namespace exists
  • Consider resource quotas

• Deployment Strategy

  • Revision strategy for GitOps
  • Automatic reconciliation
  • Rollback capabilities

• Monitoring Setup

  • Watch reconciliation status
  • Monitor Flux logs
  • Set up alerts for failures

• Security Considerations

  • Use specific branches
  • Implement access controls
  • Secure sensitive values

Conclusion - Deploying Your Application

Throughout this guide, we've built a comprehensive values configuration for your application. All these configurations come together in environment-specific values files located at ./flux/<environment>/values.yaml. Here's how to bring it all together:

Values File Structure

Your final values.yaml file combines all configurations we've covered:

# Application Configuration
app:
  replicaCount: <number>
  revisionHistoryLimit: <number>
  image:
    name: <registry-path>/<image-name>:<tag>
    pullPolicy: <Always|IfNotPresent|Never>
    pullSecret: image-pull-secret
    rollback: <previous-image-tag-for-rollbacks>
  env: <environment-name>

readinessProbe:
  httpGet:
    path: <health-check-endpoint>
    port: http
  initialDelaySeconds: <delay>
  periodSeconds: <interval>
  timeoutSeconds: <timeout>
  successThreshold: <success-count>
  failureThreshold: <failure-count>

resources:
  limits:
    cpu: <cpu-limit>
    memory: <memory-limit>
  requests:
    cpu: <cpu-request>
    memory: <memory-request>

autoscaling:
  enabled: <true|false>
  minReplicas: <min-pods>
  maxReplicas: <max-pods>
  targetCPUUtilizationPercentage: <target-cpu>

# Service Configuration
service:
  type: ClusterIP
  port: <port-number>
  name: <service-name>

# Ingress Configuration
ingress:
  enabled: true
  className: "alb"
  host: <environment-domain>
  paths:
    - path: /<service-path>
      pathType: Prefix
  annotations:
    alb.ingress.kubernetes.io/scheme: internal
    alb.ingress.kubernetes.io/group.name: tf-<env>-alb
    alb.ingress.kubernetes.io/subnets: <subnet-list>
    alb.ingress.kubernetes.io/security-groups: <security-group>
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
    alb.ingress.kubernetes.io/target-type: ip
    alb.ingress.kubernetes.io/certificate-arn: <cert-arn>
    alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS13-1-2-2021-06
    alb.ingress.kubernetes.io/healthcheck-path: /<service-path>/healthcheck

Adding Your Application to the Cluster

1. Example Repository
   • Reference Implementation: Testmaker Api Gateway Deploy
   • Use this as a template for your deployment structure
2. Create Flux Configuration
   • Submit a Merge Request to Cluster Manager Repository and notify a platform team member for approval.
   • Add two files to clusters/tf-<env>-cluster/flux-config/:

     ├── <your-app>-deploy-repo.yaml
     └── <your-app>-deploy-release.yaml     

1. Monitor Deployment
   • Watch Flux logs for reconciliation (using Flux-CLI)
   • Check application pods and services
   • Verify ingress configuration
   • Test application endpoints

Next Steps

1. Ensure all required secrets exist in the target namespace

2. Monitor the deployment progress

3. Test your application endpoints

4. For day-to-day configuration changes after the initial deployment, see How-To: Update Application Values in Your -deploy Repository

For additional support or questions, please reach out to the Platform Team