Skip to main content

How-To: Managing Labs Projects

Introduction​

This guide provides step-by-step procedures for every transition in the Labs project lifecycle — creating a Labs project from any of its three origins (fork, net-new, imported), tracking it through its time-box, graduating it to its production home, and sunsetting it when the evaluation ends.

Read the explanation guide first if you need the conceptual model — the why behind the gates, prohibitions, and approval paths. This document assumes you already know what state your project is in and what transition you need to make.

What this guide covers: the manual procedures that today connect a project's lifecycle states. The Fork Pipeline automates the creation step for fork-origin projects; everything else is a checklist executed by the owner with named approvers along the way.

Prerequisites​

Before performing any Labs lifecycle operation, make sure you have:

  • GitLab access: Maintainer (or higher) permissions on the relevant wwnorton/lab/<initiative> subgroup so you can create projects, configure CI/CD, and (later) initiate a project transfer. If you don't have it, contact the Platform team via the @platform group tag in any public Digital Product Group MS Teams channel.
  • AWS access (for verification): SSO access to the target AWS account via AWS Identity Center so you can verify secrets, IAM roles, and CloudWatch logs during deployment and teardown. See Managing Application Secrets for Console access details.
  • Fork Pipeline familiarity: If your project is fork-origin, read Using the Fork Pipeline before proceeding. This guide picks up where the Fork Pipeline leaves off.
  • kubectl + flux CLI: Required for verification commands during graduation and sunset. Install via your normal tooling channel.
  • Tracking-tool access: Either Jira (today) or the GitLab wwnorton/lab/labs-registry project (going forward) — see Step: Register the project. During the migration window, both.

You will need both Jira and GitLab access during the migration window. Until the GitLab Issues + Topics tracking model is fully rolled out, every state transition must be reflected in both surfaces. The procedures below show both — do not skip either tab.

Creating a Labs Project​

The creation procedure depends on origin. Pick the tab that matches yours.

A fork is a copy of an existing wwnorton/app/* repository for parallel development. The Fork Pipeline does almost all of the mechanical work — your job here is to wrap it in the lifecycle ceremony (tracking record, time-box, success criteria).

Step 1: Get EM/Tech-lead sign-off​

Before running any pipeline, get written sign-off from an EM or tech lead other than you. The sign-off confirms the work is real and the Labs slot is justified.

  • Open a discussion in your team's preferred channel (MS Teams thread, Slack DM, GitLab Issue draft).
  • Capture: project name, source upstream repo, initiative subgroup, your declared time-box (target end date), and written success criteria.
  • The EM/Lead replies with explicit approval. Save the reference (link, screenshot, or quoted text) — it goes into the tracking record in Step 5.

No template required. The sign-off can live in a Jira comment, a Teams message, or a GitLab Issue draft — whatever surface your team already uses. What matters is that the record exists and can be linked to.

Step 2: Run the Fork Pipeline​

Follow the full procedure in Using the Fork Pipeline. Key inputs for a Labs fork:

InputValue for Labs forks
source_repoThe upstream repo, e.g. wwnorton/app/ec/order-service
dest_groupwwnorton/lab/<initiative> — must exist; create the initiative subgroup first
fork_nameLowercase-hyphens, with a -fork suffix, e.g. order-service-cu-fork
namespacelabs (default for the Labs group)

Run with dry_run first, review the preview, then run for real. The pipeline produces three merge requests — note the URLs from the job summary.

Step 3: Open the Platform review ticket​

All three MRs produced by the pipeline require Platform team review and approval before they can be merged. If you are not a Platform team member, you must open a review ticket immediately after the pipeline completes — before doing anything else.

Open the ticket following the procedure in Platform Team Review in the Fork Pipeline guide. The ticket must contain the two repo URLs, the three MR URLs in merge order, the source upstream, the destination Labs subgroup, and the namespace.

Platform team members forking for their own work can skip the ticket but must still go through GitLab's normal MR approval flow on each MR.

Do not start merging without approvals. The Deploy Repo MR sets up the Helm chart structure that the IAM and FluxCD MRs depend on; merging any of the three without Platform approval can leave the chart, terraform state, or cluster config in an inconsistent state.

Step 4: Merge the three pipeline MRs​

Once Platform approval is recorded on each MR, follow the post-pipeline steps in the Fork Pipeline guide, paying close attention to:

  1. Update AWS secrets with new values before merging the Deploy Repo MR — the cloned secrets still point at the upstream's database and API keys.
  2. Merge order is enforced: Deploy Repo MR (after approval) → IAM Config MR (after approval, wait for terraform apply) → FluxCD Config MR (after approval).
  3. Verify the deployment in the labs namespace before moving on.

Step 5: Register the project in the tracker​

See Register the project in the tracker below for the dual-surface procedure. Be sure to capture:

  • The EM/Lead sign-off reference from Step 1
  • The two GitLab project URLs (code + -deploy) produced by the Fork Pipeline
  • The upstream repo URL — this is where the eventual graduation MR will be opened
  • Origin: Fork

Once the tracker is updated, the fork is in the Incubating state and you're done with creation.

Step: Register the project in the tracker​

Every Labs project lives in the tracker from the moment it moves from Proposed to Incubating. Maintain both Jira and GitLab during the migration window — these are not alternatives, they are parallel obligations.

  1. In your team's Jira project, create a new Epic.

  2. Title: [Labs] <project-name> (the [Labs] prefix is required — cross-project filtering depends on it).

  3. Assignee: the owner (you).

  4. Description: fill in all of the following:

    ## Project Metadata
    - **Origin**: Fork | New | Imported
    - **Initiative**: <initiative subgroup name>
    - **EM/Lead approver**: <name> — sign-off ref: <link or quote>
    - **Entry date**: <YYYY-MM-DD>
    - **Target end date**: <YYYY-MM-DD>
    - **Co-owners**: <names, if any>

    ## GitLab Projects
    - **Code repo**: <URL>
    - **Deploy repo**: <URL>
    - **Upstream repo** (Fork-origin only): <URL>
    - **Provenance attestation** (Imported-origin only): <URL to THIRD_PARTY_PROVENANCE.md>

    ## Success Criteria
    <Numbered list of testable, measurable criteria. Written before the Epic moves to In Progress.>
    1. ...
    2. ...

    ## Lifecycle State Log
    <One line per transition, newest at top.>
    - <YYYY-MM-DD> Incubating (entry approved by <approver>)

  5. Label / custom field: set the lifecycle state to incubating.

  6. Save the Epic. Link to it from any related work items in the team's normal way.

Tracking & Time-Box Reviews​

The owner is responsible for keeping the tracking surfaces current. Two events drive most updates: any lifecycle state transition, and arrival at the declared target end date.

Updating state on any transition​

Whenever the project moves between states (incubating → graduating, graduating → migrated, incubating → sunsetting, etc.):

  1. In Jira: update the lifecycle-state label/custom field, and append a new line at the top of the Lifecycle State Log in the Epic description with the date, new state, and approver (if any).
  2. In GitLab: update Topics on the project(s) (labs-incubating → labs-graduating, etc.) and the state::* label on the registry Issue. Add a comment on the registry Issue explaining the transition.

Do both on the same day. A mismatch between Jira and GitLab is treated as the project being out of compliance.

Time-box review at the declared end date​

When the Target end date in the tracking record arrives:

  1. Decide on an action with input from the EM/Lead (if helpful) and any co-owners:
  2. Record the decision in the tracking record:
    • If extending: update the Target end date field and add a comment with the justification (one or two sentences explaining why the original window wasn't sufficient and what changed)
    • If graduating: the state transition to graduating is the record
    • If sunsetting: the state transition to sunsetting is the record
  3. Notify the EM/Lead of the action taken (a comment-link or a brief Teams message — no formal review required for extensions).

An expired Target end date with no action is the failure mode. At the quarterly portfolio review, any incubating project whose Target end date has passed without an extension comment is flagged. Don't be that project.

Quarterly portfolio review​

The Platform team (or a designated delegate) runs a quarterly review across both tracking surfaces:

  • Jira query — Epics with label incubating AND Target end date < today
  • GitLab query — registry Issues with label state::incubating AND any custom date field past due

Findings are surfaced to the owners and their EMs/Leads. The goal is not to punish drift — it is to make drift visible.

Graduating a Labs Project​

Graduation moves the project from Labs to its production home. The mechanics differ sharply by origin — pick the tab that matches yours.

Graduation prerequisites (all origins)​

Before requesting graduation approval, complete all three:

  1. Security review. Open a security-review request with the Platform team. The review covers:

    • Secrets layout: dev|qa|stg/labs/json/<project> contents, rotation posture
    • IAM configuration: service account, role annotations, policy scope
    • Dependency scan results: from your CI's standard scanning step (npm audit, pip-audit, etc.) plus any extra scan from Imported provenance
    • For Imported-origin projects: the THIRD_PARTY_PROVENANCE.md and license compatibility with the receiving group's posture

    Attach the review outcome (approval comment or signed-off ticket) to the tracking record.

  2. CI tests passing + coverage threshold met. The CI pipeline must be green, and the coverage figure on the default branch must meet or exceed the bar set by the receiving group. Record both values (current coverage, required bar) in the tracking record.

  3. Evidence against the entry-time success criteria. For each numbered criterion in the tracking record, add evidence (metrics, test results, screenshots, replayed-traffic results). Be specific. "Met" without evidence does not count.

Receiving group lead approval comes after these three are complete.

A Fork-origin graduation does not move the Labs project anywhere. The work moves: a graduation MR is opened against the upstream repo, and once merged, the Labs fork is sunset.

Step 1: Sync from upstream​

The Labs fork has diverged from upstream while you've been working. Pull in any changes that have landed on upstream main since the fork was created:

# In your local clone of the Labs fork
git remote add upstream https://gitlab.com/wwnorton/app/<group>/<app>.git
git fetch upstream
git merge upstream/main
# Resolve conflicts; run tests; push to the Labs fork
git push origin main

Re-run CI on the Labs fork and confirm it's still green. Re-deploy and verify in the labs namespace.

Step 2: Prepare the graduation MR against upstream​

The graduation MR goes against the upstream repo (wwnorton/app/<group>/<app>), not the Labs fork.

  1. From your local clone of the Labs fork, push a branch directly to the upstream repo (you need Developer+ access on upstream):

    git remote add upstream https://gitlab.com/wwnorton/app/<group>/<app>.git
    git push upstream main:graduate/<initiative>-<short-description>

  2. Open the MR in upstream:

    • Source branch: graduate/<initiative>-<short-description>
    • Target branch: main
    • Title: Graduate <initiative>: <one-line summary>
    • Description: link to the Labs tracking record, list the success criteria + evidence, summarize the changes, attach the security review outcome

  3. The MR's CI runs against upstream's normal pipeline — including any production-tier gates (image scans, integration tests, etc.) that Labs didn't enforce.

Step 3: Receiving group lead approves and merges​

The receiving group lead — typically the tech lead of the upstream's owning group — reviews and approves. Standard upstream MR review process applies; the Labs context is provided via the MR description.

Once merged, upstream's deploy pipeline runs and deploys the change to its normal envs (dev → qa → stg → prod) per the upstream's deployment policy.

Step 4: Verify upstream deployment​

Wait for the upstream deployment to land and verify it in upstream's lower envs (dev/qa/stg). Run any smoke or integration tests the upstream team uses. Confirm with the receiving group lead that the change is healthy.

Do not sunset the Labs fork yet. The Labs deployment is your rollback if anything is wrong upstream.

Step 5: Sunset the Labs fork​

Once upstream is verified healthy:

  1. Update the tracking record: state → sunsetting, with a note that this is post-graduation cleanup (not a failed eval).
  2. Follow Sunsetting a Labs Project for the full teardown sequence.

Rollback During Graduation​

If the graduation goes wrong mid-flight, the rollback strategy is don't tear down Labs until prod is verified. The Labs deployment is your safety net.

Point of no return: when is rollback off the table?​

Step in graduationRollback possible?How
Before upstream MR / project transferYes — easyDiscard the MR or pause the transfer. No prod-side state yet.
After upstream MR merged (Fork path)Yes — revertOpen an upstream revert MR. Labs fork is still running as backup.
After project transfer (New path)Yes — transfer backGitLab Transfer the project back to wwnorton/lab/<initiative>.
After Labs deployment torn downNoLabs is gone. From here, fix-forward in the new home.

The instruction is simple: do not run the Labs teardown until the receiving group lead has signed off on the prod deployment being healthy. If you tear down Labs prematurely and prod has a problem, you are in fix-forward territory with no fallback.

What to do if a problem is found before the point of no return​

  1. Pause the graduation. Update the tracking record state back to incubating with a comment explaining the rollback.
  2. Notify the EM/Lead and the receiving group lead.
  3. Reproduce in Labs. If the problem also reproduces in the Labs deployment, fix it there first. If it only reproduces in the new namespace, the problem is in the migration, not the code.
  4. Address the root cause. Common ones:
    • Secrets didn't copy correctly (compare with aws secretsmanager get-secret-value)
    • IAM role annotation missing (kubectl get sa <name>-sa -n <ns> -o yaml | grep eks.amazonaws.com)
    • FluxCD reconciling against the wrong path (check the GitRepository URL)
    • Namespace-specific config (CORS, ingress hostnames, env-specific URLs)
  5. Re-attempt the graduation from the step that failed.

Sunsetting a Labs Project​

Sunset is run on two triggers: a failed evaluation, or post-graduation cleanup of a fork. The procedure is the same in both cases. The order of teardown matters — it follows the dependency graph so nothing is left orphaned.

Step 1: Tear down FluxCD resources​

Remove the HelmRelease and GitRepository for the project in the labs namespace via an MR to the cluster-manager repo:

  1. Delete the <project>.yaml (or equivalent) files referencing the project in clusters/*/labs/.
  2. Open an MR, get approval, merge.
  3. FluxCD reconciles and removes the HelmRelease, which removes the underlying Deployment, Service, and SecretProviderClass.

Verify no resources remain:

kubectl get all -n labs -l app.kubernetes.io/name=<project>
# Expect: No resources found

kubectl get sa -n labs | grep <project>
# Expect: empty (the IAM service account is removed by the terraform step below)

Step 2: Schedule AWS secrets for deletion​

Use scheduled deletion with a recovery window — not immediate delete. The default 30-day window lets an accidental sunset be reversed.

for ENV in dev qa stg; do
  aws secretsmanager delete-secret \
    --secret-id "${ENV}/labs/json/<project>" \
    --recovery-window-in-days 30
done

For graduation cleanup (where the receiving namespace's secrets are the live ones), the Labs secrets being scheduled for deletion is expected — the prod path is <env>/<new-namespace>/json/<new-name> now.

Step 3: Remove IAM config (terraform MR)​

Open an MR against wwnorton/ops/infrastructure removing the project's entry from accounts/{dev,qa,stg}/labs/config.json:

// Before
[
  { "name": "<project>", "type": "json" },
  { "name": "<other-project>", "type": "json" }
]

// After
[
  { "name": "<other-project>", "type": "json" }
]

Get Platform team approval, merge, and wait for terraform apply. The service account and IAM role for the project will be destroyed.

Verify:

kubectl get sa <project>-sa -n labs
# Expect: NotFound

Step 4: Archive the GitLab repositories​

For each project (code, then -deploy):

  1. Open the project in GitLab
  2. Settings → General → Advanced → Archive project
  3. Confirm by typing the project path

Archive — do not delete. The repo becomes read-only, but history, MRs, issues, and CI runs are preserved for future reference.

Why archive over delete? A failed Labs project that is archived rather than deleted is the cheapest reference for the next contributor who proposes the same evaluation. Archive preserves the answer to "what did we try and why didn't it work."

Step 5: Update the tracking record​

  1. In Jira: lifecycle state → archived. Add a final comment in the Lifecycle State Log with the archive date and the trigger (failed eval / post-graduation cleanup).
  2. In GitLab: update Topics to labs-archived on both projects, update the registry Issue label to state::archived, close the registry Issue with a comment explaining the trigger.

The project is now done. Quarterly reviews will skip it.

Don't forget data resources​

If the project provisioned S3 buckets, RDS instances, or other resources through the self-service Terraform modules, those are not removed by the steps above. See Cleaning Up Data Resources below.

Cleaning Up Data Resources​

Data resources (S3 buckets, RDS instances) created via the self-service Terraform modules are owned by the project's owner and are removed via the same self-service workflow used to create them. The Platform team does not centrally clean these up — the owner does, before the GitLab repos are archived.

For each data resource the project provisioned:

Resource typeCleanup procedure
S3 bucketOpen an MR removing the bucket entry from accounts/<env>/s3/terraform.tfvars — How-To. Consider lifecycle rules to transition contents to Glacier before destroy if data should be preserved.
RDS instanceOpen an MR removing the entry from the RDS terraform config — How-To. Take a final snapshot first if the data may be needed.
Route53 recordOpen an MR removing the record — How-To.

Order: data-resource cleanup happens after the application teardown (Steps 1–3 above) so the application is no longer trying to read or write the resource. Then archive the repos (Step 4).

FAQ​

How do I know if my project should graduate or sunset when the time-box expires?​

Compare the evidence you have against the success criteria you wrote at entry. If the evidence shows the criteria were met, graduate. If it shows they were not, sunset. If you genuinely don't know yet, extend the time-box with a written justification — but be honest: if you find yourself extending more than once without new information, that is itself a signal that the evaluation has failed.

What if no wwnorton/app/* group wants to receive my graduating project?​

This is a real failure mode for New / Imported projects. The receiving group is identified by the owner together with the candidate group's lead — there is no central "graduations team" you can hand off to. If no group will own it, the project cannot graduate, and its options are: (a) extend the time-box while you keep looking, (b) reshape the work so it fits an existing group's scope, or (c) sunset.

For Fork-origin projects this is not usually a problem because the upstream's owning group is the natural receiver.

Can I have multiple owners on a Labs project?​

The tracking record carries a single primary owner (the Jira assignee, the GitLab Issue assignee). Co-owners are listed in the description. The primary owner is the named human responsible for the lifecycle — they answer for the project at portfolio review and request the state transitions. Having one human's name on the record is what prevents the diffusion-of-responsibility failure mode.

My project hasn't done anything in a month. Is it abandoned?​

Activity isn't the metric — the time-box and success criteria are. A project that has hit its criteria and is just waiting for a graduation review can sit quietly. A project that has done nothing because the owner has moved on is a tracking-record problem: update the state to sunsetting, notify the EM/Lead, and run the teardown.

Can I move a project out of Labs without graduating it?​

No. The two terminal states are migrated (via graduation) and archived (via sunset). Moving a repo out of wwnorton/lab/ without going through graduation is the failure mode the lifecycle exists to prevent — it produces a production service with no security review, no test gates, and no receiving-group ownership.

What happens if I try to provision a production-tier resource from a Labs project?​

The OPA policies on the self-service Terraform modules will reject the MR. This is the correct outcome — if your project needs production-tier infrastructure, it has outgrown Labs and should graduate.

What if I need a public-facing URL for a demo or external review?​

Open a request with the Platform team. The review is short — the goal is to confirm that the URL is for a time-bound demo or review and not the beginning of shadow production. Approved demos typically get a one-off ingress that is scheduled for removal when the demo is over.

My Fork-origin project's upstream has changed significantly. Is graduation still possible?​

Probably, but it depends. If the upstream changes are compatible with your fork's work, sync them in (git merge upstream/main) and proceed with the graduation MR. If the upstream has shifted in a direction that makes your fork's work no longer applicable, sunset the fork and re-evaluate whether the work is still worth doing against the new upstream.

Where do I file bugs found in a Labs project during evaluation?​

In the Labs project's own GitLab Issues. The tracking record (Jira Epic / labs-registry Issue) is for the lifecycle state — not for day-to-day bug tracking.

Do I update the tracking record from a fork-pipeline run, or does the pipeline do it?​

You do. The fork-pipeline creates the GitLab projects, builds the image, opens the three MRs, and clones the secrets — it does not touch Jira or the labs-registry. Adding the tracking record is the first thing you do after the pipeline runs (Step 4 of the Fork-origin creation flow).