Self-Hosted Build Agent (DigitalOcean)¶

Version: 1.0 Last Updated: February 2026 Setup Script: deployment/build-agent/setup-build-agent.sh

This document describes the self-hosted Azure DevOps build agent running on a DigitalOcean droplet, used to accelerate Docker image builds through persistent local caching.

Table of Contents¶

1. Overview
2. Architecture
3. Why Self-Hosted
4. Droplet Specification
5. Agent Pool Configuration
6. Pipeline Integration
7. Job Distribution Strategy
8. Setup Instructions
9. Maintenance
10. Monitoring & Troubleshooting
11. Security Considerations
12. Cost Analysis

Overview¶

The CI/CD pipeline uses a hybrid agent strategy: a Microsoft-hosted agent handles lightweight jobs (linting, testing, Nx builds) while a self-hosted DigitalOcean droplet handles all Docker image packaging. This leverages persistent Docker layer caching on the self-hosted agent to dramatically reduce build times.

Key Benefits¶

• Docker layer caching: Persistent cache between builds (vs. cold cache on every MS-hosted run)
• Pre-installed tools: Cosign and Syft are pre-installed instead of downloaded each run
• Parallel execution: 2 agent instances enable true parallel job execution
• Cost-effective: ~$48/month for performance equivalent to several MS-hosted parallel jobs

Architecture¶

Why Self-Hosted¶

The Problem¶

Microsoft-hosted agents are ephemeral — each job gets a fresh VM. This means:

• Every Docker build starts with cold cache (pulls all base images, rebuilds all layers)
• Cosign and Syft are downloaded and installed on every single job (~30s each)
• With 1 free parallel job, 8+ Docker builds queue sequentially

The Solution¶

A self-hosted agent on a persistent droplet provides:

Expected Build Times (per service)¶

Full Pipeline Impact¶

Droplet Specification¶

Installed Software¶

Agent Pool Configuration¶

Azure DevOps Setup¶

Before running the setup script, create the agent pool in Azure DevOps:

1. Go to Organization Settings → Agent pools
2. Click Add pool
3. Select Self-hosted
4. Name: DO-Build-Agents
5. Grant access permission to all pipelines (or select specific pipelines)
6. Check Auto-provision this agent pool in all projects

Agent Instances¶

The droplet runs 2 agent instances, allowing 2 pipeline jobs to execute simultaneously:

Both instances share the same Docker daemon and layer cache, maximizing cache hit rates.

Pipeline Integration¶

Pool Configuration¶

The pipeline uses a default pool at the top level and per-job overrides for Docker-heavy jobs:

# Default: MS-hosted agent (lightweight jobs)
pool:
  vmImage: 'ubuntu-latest'

stages:
  - stage: Build
    jobs:
      # Runs on MS-hosted (inherits default pool)
      - job: BuildAll
        steps: ...

      # Runs on self-hosted (pool override)
      - job: PackageGateway
        pool:
          name: 'DO-Build-Agents'
        steps: ...

Jobs Using Self-Hosted Pool¶

Jobs Remaining on MS-Hosted¶

Job Distribution Strategy¶

Validate & Test Stage (3 agents in parallel)¶

Build & Package Stage (2 agents in parallel)¶

Note: Azure DevOps automatically distributes jobs across available agents in the pool. The 8 Docker package jobs will be picked up by whichever agent instance is free, naturally balancing across the 2 instances.

Setup Instructions¶

Prerequisites¶

1. DigitalOcean droplet created with Ubuntu 22.04 LTS (4 vCPU / 8 GB RAM)
2. SSH access to the droplet (root or sudo user)
3. Azure DevOps PAT with scope: Agent Pools (read, manage)
4. Agent pool DO-Build-Agents created in Azure DevOps (see Agent Pool Configuration)

Step 1: Create the Agent Pool in Azure DevOps¶

1. Navigate to https://dev.azure.com/{your-org}/_settings/agentpools
2. Click Add pool → Self-hosted
3. Name: DO-Build-Agents
4. Enable Auto-provision this agent pool in all projects
5. Click Create

Step 2: Generate a Personal Access Token (PAT)¶

1. Navigate to https://dev.azure.com/{your-org}/_usersSettings/tokens
2. Click New Token
3. Name: Build Agent Registration
4. Scopes: Agent Pools → Read & manage
5. Expiration: Set to a reasonable period (the PAT is only needed during setup)
6. Copy the token — it will not be shown again

Step 3: SSH into the Droplet and Run Setup¶

# SSH into the droplet
ssh root@<DROPLET_IP>

# Download the setup script (or SCP it from your machine)
# Option A: SCP from local machine
scp deployment/build-agent/setup-build-agent.sh root@<DROPLET_IP>:/tmp/

# Option B: Or copy-paste the script contents

# Make it executable and run
chmod +x /tmp/setup-build-agent.sh
/tmp/setup-build-agent.sh \
  --org-url https://dev.azure.com/YOUR_ORG \
  --pat YOUR_PERSONAL_ACCESS_TOKEN \
  --pool DO-Build-Agents \
  --agents 2

Step 4: Verify Agent Registration¶

1. Go to https://dev.azure.com/{your-org}/_settings/agentpools
2. Click on DO-Build-Agents
3. You should see 2 agents online: do-build-agent-1 and do-build-agent-2

Step 5: Warm the Docker Cache (Optional but Recommended)¶

Trigger a full pipeline run with ForceFullVersioningAndDeployment = true to populate the Docker layer cache on the self-hosted agents. Subsequent builds will be significantly faster.

Maintenance¶

Automated Maintenance¶

The setup script installs two cron jobs:

Cleanup logs are written to /var/log/docker-cleanup.log.

Manual Maintenance¶

Check Agent Status¶

# Check both agent services
systemctl status vsts.agent.*

# View agent logs (live)
journalctl -u vsts.agent.* -f

Restart Agents¶

# Restart a specific agent
/opt/azagent/agent-1/svc.sh stop
/opt/azagent/agent-1/svc.sh start

# Restart all agents
for i in 1 2; do
  /opt/azagent/agent-${i}/svc.sh stop
  /opt/azagent/agent-${i}/svc.sh start
done

Update Agent Version¶

# Stop the agent service first
/opt/azagent/agent-1/svc.sh stop

# The agent can self-update, or re-run the setup script with new version
cd /opt/azagent/agent-1
./config.sh remove --auth pat --token <PAT>

# Re-run setup or manually update

Manual Docker Cleanup¶

# Check disk usage
df -h /
docker system df

# Gentle cleanup (keep recent cache)
docker container prune --force --filter "until=168h"
docker image prune --force

# Aggressive cleanup (when disk is full)
docker system prune --all --force --filter "until=48h"

Update Pre-installed Tools¶

# Update Cosign
curl -sSL https://github.com/sigstore/cosign/releases/download/v2.x.x/cosign-linux-amd64 \
  -o /usr/local/bin/cosign
chmod +x /usr/local/bin/cosign

# Update Syft
curl -sSfL https://raw.githubusercontent.com/anchore/syft/main/install.sh | sh -s -- -b /usr/local/bin

# Update Node.js (if major version changes)
curl -fsSL https://deb.nodesource.com/setup_20.x | bash -
apt-get install -y nodejs

# Update .NET SDK
apt-get update && apt-get install -y dotnet-sdk-8.0

# Update reportgenerator (for azagent user)
su - azagent -c "dotnet tool update -g dotnet-reportgenerator-globaltool"

Monitoring & Troubleshooting¶

Agent Appears Offline in Azure DevOps¶

# Check if the service is running
systemctl status vsts.agent.*

# Check agent logs
journalctl -u vsts.agent.* --since "1 hour ago"

# Restart the agent
/opt/azagent/agent-1/svc.sh stop
/opt/azagent/agent-1/svc.sh start

Docker Builds Failing with OOM¶

# Check memory usage during builds
free -h
docker stats --no-stream

# If consistently OOM, consider:
# 1. Reducing to 1 agent instance (--agents 1)
# 2. Upgrading droplet to 8 vCPU / 16 GB RAM

Disk Space Running Low¶

# Check what's using space
df -h /
docker system df
du -sh /opt/azagent/*/

# Run manual cleanup
docker system prune --all --force --filter "until=48h"

# Check cleanup log
tail -50 /var/log/docker-cleanup.log

Docker Cache Not Being Used¶

# Verify buildx builder exists
docker buildx ls

# Inspect the builder
docker buildx inspect forma3d-builder

# If builder is missing, recreate it
docker buildx create --name forma3d-builder --driver docker-container --use
docker buildx inspect --bootstrap

Pipeline Jobs Not Picking Up Self-Hosted Agent¶

1. Verify the pool name in azure-pipelines.yml matches exactly: DO-Build-Agents
2. Check agent capabilities in Azure DevOps → Agent pools → DO-Build-Agents → Agents → Capabilities
3. Ensure the pipeline has permission to use the pool

Security Considerations¶

Agent User¶

• Agents run as a dedicated azagent user (non-root)
• The azagent user has Docker group access (required for builds)
• No sudo privileges on the agent user

PAT Scope¶

• The PAT used during setup only needs Agent Pools (read, manage) scope
• The PAT is only used during initial registration — it is not stored on the droplet
• After registration, the agent uses its own OAuth token to communicate with Azure DevOps

Docker Socket¶

• The Docker socket is accessible to the azagent user via the docker group
• This is standard for self-hosted agents running Docker builds
• The Docker daemon runs as root (standard Docker CE configuration)

Network Access¶

• The agent initiates outbound HTTPS connections to Azure DevOps (no inbound required)
• Docker builds push to DigitalOcean Container Registry over HTTPS
• No ports need to be opened in the droplet firewall for agent communication

Secrets Handling¶

• Pipeline secrets (DOCR_TOKEN, COSIGN_PASSWORD, etc.) are passed as environment variables at runtime
• Secrets are not persisted on the droplet between jobs
• The cosign.key file is downloaded via Azure DevOps Secure Files during each job and cleaned up after

Cost Analysis¶

Monthly Cost Comparison¶

ROI¶

• Cost: $48/month
• Time saved per full build: ~65 min (133 → 68 min)
• Builds per month (estimated): ~60 (2 per working day)
• Developer time saved: ~65 hours/month
• Effective cost per build: $0.80

Related Documentation¶

Revision History: