Azure DevOps Pipeline Reference¶

Version: 3.5
Last Updated: 21 March 2026
Pipeline File: azure-pipelines.yml

This document provides a detailed technical reference for the Forma3D.Connect CI/CD pipeline, including architecture, stages, caching strategies, and supply chain security.

Table of Contents¶

1. Pipeline Overview
2. Pipeline Architecture
3. Stages Reference
4. Variables and Parameters
5. Docker Build & Caching
6. Supply Chain Security
7. Registry Management
8. Conditional Execution

Pipeline Overview¶

The pipeline implements trunk-based development with the following philosophy:

High-Level Flow¶

Pipeline Architecture¶

Complete Stage Dependency Graph¶

Job Parallelism in Build Stage¶

Stages Reference¶

Stage 1: Validate & Test¶

Condition: Always runs (except loadTestOnly mode)

All 4 jobs run across MS-hosted and self-hosted agents. Lint, TypeCheck, and UnitTests run in parallel; CodeQuality runs after UnitTests:

Outputs:

• JUnit XML test results
• Cobertura code coverage (Azure DevOps)
• lcov coverage reports (published as pipeline artifact for SonarCloud)
• SonarCloud quality gate result

CodeQuality Job (SonarCloud)¶

The CodeQuality job runs SonarCloud static analysis after unit tests complete. It downloads the lcov coverage artifacts from the UnitTests job and feeds them to SonarCloud for combined code quality + coverage analysis.

Configuration:

Rule suppression: False positives and won't-fix items are managed in sonar-project.properties via sonar.issue.ignore.multicriteria. Inline // NOSONAR comments do not work for TypeScript/JavaScript in SonarCloud.

Stage 2: Build & Package¶

Condition: Main branch only, after Validate & Test succeeds

Stage 3: Deploy to Staging¶

Condition: Main branch, at least one app affected

Stage 4: Acceptance Tests¶

Condition: Main branch, loadTestOnly is false, Build succeeded, and either:

1. Deploy Staging succeeded and at least one of these was affected: Gateway, Web, Order / Print / Shipping / GridFlock services, or Slicer (docs and EventCatalog are not in this list), or
2. Deploy Staging was skipped and acceptance tests were affected (run Gherkin against the existing staging deployment without a new deploy).

Jobs:

• Verify Deployment Health & Version — runs only when an app service from (1) was deployed (same service set as above; excludes docs/EventCatalog).
• Run Gherkin Acceptance Tests — runs after verification completes or was skipped (covers both full deploy and acceptance-tests-only flows).

Outputs: Cucumber HTML report, Playwright traces, JUnit results.

Stage 5: Lighthouse Audit¶

Condition: Main branch, Deploy Staging succeeded, and Web was affected.

• Runs Lighthouse CI against WEB_URL (self-hosted agent, Chrome).
• Publishes HTML report tab and build artifact.
• Skipped when Web was not deployed in the run (for example docs-only or API-only). Attest Staging still allows LighthouseAudit result Skipped.

Stage 6: Registry Maintenance¶

Condition: Main branch (ordering after Build and Deploy Staging; runs even when deploy was skipped).

• Registry cleanup script and DigitalOcean garbage collection.
• Attest Staging waits for this stage to finish (success, failure, or skipped) so cosign operations are less likely to hit transient registry 401 errors during GC.

Stage 7: Attest Staging Promotion¶

Condition: Main branch, enableSigning true, loadTestOnly false, LighthouseAudit in Succeeded / SucceededWithIssues / Skipped, RegistryMaintenance finished (including Failed), and either:

• Acceptance Test succeeded (full E2E path), or
• Deploy Staging succeeded and docs and/or EventCatalog were in the affected set (static-site path; Acceptance stage is usually Skipped for those).

Job: AttestStagingPromotion (Microsoft-hosted ubuntu-latest, checkout: none).

• Runs only when at least one deployable target was affected (deploymentHappened).
• Per-image steps run only when that service’s *Affected flag is true and the digest from the corresponding Package* job in this run matches sha256: + 64 hex chars. Otherwise the step fails (no attesting stale or empty digests).
• Staging custom predicate includes verification.acceptanceTestsPassed: true only if the Acceptance stage completed as Succeeded or SucceededWithIssues; false on the docs/EventCatalog-only path (Acceptance Skipped).

Stage 8: Load Tests (Optional)¶

Condition: runLoadTests parameter is true (and upstream stages per azure-pipelines.yml)

• Runs K6 load tests
• Can run in baseline mode (no threshold failures)

Stage 9: Deploy to Production¶

Condition: When the production stage is enabled in azure-pipelines.yml: depends on Build, AcceptanceTest, AttestStaging, and LoadTest; manual approval gate on the production environment.

Variables and Parameters¶

Pipeline Parameters¶

Pipeline Variables¶

Variable Group: forma3d-staging¶

The forma3d-staging variable group contains all secrets and configuration:

Docker Build & Caching¶

BuildKit Registry Caching¶

The pipeline uses docker buildx with registry-based caching for optimal build performance on ephemeral Azure DevOps agents.

Build Command Structure¶

docker buildx build \
  --file apps/api/Dockerfile \
  --tag $(registry)/$(image):$(tag) \
  --tag $(registry)/$(image):latest \
  --cache-from type=registry,ref=$(registry)/$(image):cache \
  --cache-to type=registry,ref=$(registry)/$(image):cache,mode=max \
  --build-arg BUILD_NUMBER=$(tag) \
  --build-arg BUILD_DATE=$(date) \
  --build-arg GIT_COMMIT=$(commit) \
  --label "org.label-schema.version=$(tag)" \
  --push \
  .

Cache Behavior¶

Expected Build Times¶

Supply Chain Security¶

Image Signing Flow¶

Dependency License Check¶

Before any code is built or packaged, the Lint job runs a license compliance check against the full npm dependency tree using license-checker-rseidelsohn:

- script: pnpm run license-check
  displayName: 'Check dependency licenses (fail on non-permissive)'

The check is implemented in scripts/check-licenses.js and scans all installed packages for non-permissive licenses:

The check runs in ~1 second and exits with code 1 (failing the pipeline) if any dependency matches a disallowed license. Private packages (like the project root with "license": "UNLICENSED") are excluded.

See ADR-068: Dependency License Compliance Check for the full decision record.

CVE Scanning with Grype¶

After the SBOM is generated and attached, each image's SBOM is scanned by Grype for known vulnerabilities:

grype sbom:<service>-sbom.cdx.json --output table --fail-on high --only-fixed

Grype output columns:

EPSS (Exploit Prediction Scoring System) is a data-driven model by FIRST.org that predicts the likelihood a CVE will be exploited in the wild. It complements CVSS severity: a High-severity CVE with 0.1% EPSS (1^st percentile) is far less urgent than a Medium-severity CVE with 30% EPSS (96^th percentile). Grype includes EPSS data to help prioritize remediation.

Exclusions (.grype.yaml):

Some CVEs cannot be fixed at the project level. These are excluded in .grype.yaml at the repository root with documented rationale:

ignore:
  # docker/cli v28.5.1 → needs 29.2.0 (High, EPSS <0.1%)
  # Compiled into Alpine's docker-cli package; awaiting upstream update
  - vulnerability: GHSA-p436-gjf2-799p
    package:
      type: go-module

Current exclusions are all Go module CVEs from Alpine's docker-cli and containerd packages, affecting only the Gateway image. All have EPSS scores at the 0^th–1^st percentile.

Slicer exception: The Slicer's grype scan is disabled entirely (commented out in the pipeline) because its linuxserver/bambustudio:01.08.03 base image has 800+ unfixable CVEs from system packages. See TODO.md for the BambuStudio v2 upgrade plan.

Remediation strategies used:

Verification¶

# Verify image signature
cosign verify --key cosign.pub registry.digitalocean.com/forma-3d/forma3d-connect-api:latest

# Verify SBOM attestation
cosign verify-attestation --key cosign.pub --type cyclonedx \
  registry.digitalocean.com/forma-3d/forma3d-connect-api:latest

Image Metadata Labels¶

All images include Label Schema metadata:

Environment Promotion Attestations¶

The pipeline uses signed attestations to track image promotions through environments. This creates an auditable chain of custody and prevents deploying images that haven't passed earlier stages.

Attestation Types¶

Attestation Predicate Schema (staging / production custom type)¶

Shape produced by Attest Staging / production promotion steps (see azure-pipelines.yml). The verification block is the source of truth for whether Gherkin acceptance ran in the same pipeline run.

{
  "_type": "https://forma3d.com/attestations/promotion/v1",
  "environment": "staging",
  "promotedAt": "2026-03-21T12:00:00+00:00",
  "build": {
    "number": "20260321120000",
    "pipeline": "forma3d-connect",
    "pipelineId": "123",
    "runId": "456",
    "runUrl": "https://dev.azure.com/..."
  },
  "source": {
    "commit": "abc123",
    "branch": "main",
    "repository": "forma-3d-connect"
  },
  "verification": {
    "healthCheckPassed": true,
    "acceptanceTestsPassed": true
  }
}

acceptanceTestsPassed is true only when the Acceptance Test stage completed as Succeeded or SucceededWithIssues. For a docs-only or EventCatalog-only staging deploy, Acceptance is skipped and this field is false while the staging promotion attestation still records the build and registry digest that was pushed.

Verification Commands¶

# Verify staging attestation exists
cosign verify-attestation \
  --key cosign.pub \
  --type custom \
  registry.digitalocean.com/forma-3d/forma3d-connect-api:20260201143000

# Extract attestation content
cosign verify-attestation --key cosign.pub --type custom \
  registry.digitalocean.com/forma-3d/forma3d-connect-api:20260201143000 \
  | jq -r '.payload | @base64d | fromjson | .predicate'

Production Gate¶

Before production deployment, the pipeline verifies staging attestation:

1. Downloads the cosign public key
2. Calls cosign verify-attestation --type custom for each image
3. Extracts and validates predicate.environment == "staging"
4. Fails the pipeline if any image lacks valid staging attestation

This ensures images cannot bypass staging and go directly to production. Automation that interprets the predicate should not assume verification.acceptanceTestsPassed is always true: validate it explicitly if production should require a Gherkin run for that image.

Registry Management¶

Image Tags¶

Cleanup Script: scripts/cleanup-registry.sh¶

The cleanup script maintains registry hygiene by using the promotion attestations to determine image importance:

Cleanup Command¶

./scripts/cleanup-registry.sh \
  --key cosign.pub \
  --api-url https://staging-connect-api.forma3d.be \
  --web-url https://staging-connect.forma3d.be \
  --docs-url https://staging-docs.forma3d.be \
  --max-staging 5 \
  --dry-run  # Remove for actual cleanup

Conditional Execution¶

Nx Affected Detection¶

Stage Conditions Summary¶

Exact expressions live in azure-pipelines.yml; use this table as a guide, not a substitute for the YAML.

Troubleshooting¶

Cache Not Being Used¶

Symptoms: Build times not improving on subsequent runs

Check:

1. Verify :cache tag exists in registry
2. Check buildx is creating the builder: docker buildx inspect --bootstrap
3. Verify registry authentication succeeded

Solution:

# Manually populate cache
docker buildx build \
  --cache-to type=registry,ref=registry/image:cache,mode=max \
  --push .

Signing Failures¶

Symptoms: cosign sign fails with authentication error

Check:

1. COSIGN_PASSWORD secret is set in variable group
2. cosign.key is uploaded to Secure Files
3. File permissions are authorized for pipeline

Affected Detection Wrong¶

Symptoms: Apps not being detected as affected

Check:

1. Verify full git history is fetched: fetchDepth: 0
2. Check Nx dependency graph: pnpm nx graph
3. Test manually: pnpm nx show projects --affected --base=HEAD~1

Related Documentation¶

Revision History: