Forma3D.Connect Operations Runbook¶

Version: 1.0
Date: January 18, 2026
Status: Active

Overview¶

This runbook provides operational procedures for Forma3D.Connect in production. It covers health checks, common issues, incident response, and maintenance procedures.

1. Service Architecture¶

Components¶

Staging Environment¶

External Dependencies¶

2. Health Checks¶

Endpoints¶

# Full health check (includes database)
curl https://connect-api.forma3d.be/health

# Liveness probe (process running)
curl https://connect-api.forma3d.be/health/live

# Readiness probe (database connected)
curl https://connect-api.forma3d.be/health/ready

# External dependencies (Shopify, SimplyPrint, Sendcloud)
curl https://connect-api.forma3d.be/health/dependencies

Expected Responses¶

Healthy System:

{
  "status": "ok",
  "database": "connected",
  "timestamp": "2026-01-18T10:00:00.000Z",
  "version": "20260118100000",
  "build": {
    "number": "20260118100000",
    "date": "2026-01-18T10:00:00.000Z",
    "commit": "abc123def456"
  },
  "uptime": 86400
}

Note: The version field now displays the CI/CD build number (same as build.number), making it easy to identify which deployment is running.

Dependencies Healthy:

{
  "status": "ok",
  "info": {
    "shopify": { "status": "up", "message": "Shopify API is reachable" },
    "simplyprint": { "status": "up", "message": "SimplyPrint API is connected" },
    "sendcloud": { "status": "up", "message": "Sendcloud API is reachable" }
  }
}

Degraded (External Service Down):

{
  "status": "error",
  "error": {
    "shopify": {
      "status": "down",
      "message": "Connection timeout"
    }
  },
  "info": {
    "simplyprint": { "status": "up" },
    "sendcloud": { "status": "up" }
  }
}

Monitoring with cURL¶

# Quick health check script
while true; do
  HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" https://connect-api.forma3d.be/health)
  if [ "$HTTP_STATUS" != "200" ]; then
    echo "[$(date)] ALERT: Health check failed with status $HTTP_STATUS"
  else
    echo "[$(date)] OK: Health check passed"
  fi
  sleep 60
done

3. Common Issues and Resolutions¶

High Error Rate¶

Symptoms:

• Error rate > 5%
• Sentry alerts increasing
• Users reporting failures

Investigation:

1. Check Sentry for error patterns
2. Review API logs: docker logs forma3d-api --tail 100
3. Check database connectivity: curl .../health/ready
4. Verify external service status: curl .../health/dependencies

Resolution:

1. If database issue: Check connection pool, restart API if needed
2. If external service: Enable fallback/degraded mode, wait for recovery
3. If code bug: Deploy hotfix via normal pipeline

High Latency¶

Symptoms:

• 95^th percentile response time > 2 seconds
• Dashboard feels slow
• Webhook processing delays

Investigation:

1. Check database query performance
2. Review slow query logs
3. Check external API response times in logs
4. Monitor memory/CPU usage: docker stats

Resolution:

1. Scale up resources if needed
2. Optimize slow database queries
3. Add caching if appropriate
4. Contact external service provider if their API is slow

Database Connection Failed¶

Symptoms:

• Health check returns database: disconnected
• API returns 500 errors
• Readiness probe fails

Investigation:

1. Check PostgreSQL status in DigitalOcean dashboard
2. Verify connection string in environment
3. Check network connectivity
4. Review connection pool settings

Resolution:

# SSH to droplet
ssh root@<DROPLET_IP>

# Check API container
docker ps | grep forma3d-api

# Restart API container
docker-compose restart api

# Check logs for connection errors
docker logs forma3d-api --tail 50 | grep -i database

Shopify Webhooks Not Arriving¶

Symptoms:

• New Shopify orders not appearing
• Order count not increasing
• No webhook logs in event log

Investigation:

1. Check Shopify admin > Settings > Notifications > Webhooks
2. Verify webhook URL is correct
3. Check for failed webhook deliveries in Shopify
4. Review API logs for HMAC verification failures

Resolution:

1. Verify SHOPIFY_WEBHOOK_SECRET equals SHOPIFY_API_SECRET (the app's client secret). API-registered webhooks are signed with this, not the admin panel's webhook signing secret.
2. Re-register webhooks via the Shopify API if needed (API-registered webhooks do not appear in Shopify Admin > Notifications > Webhooks)
3. Check firewall/security group rules
4. List registered webhooks via API: GET /admin/api/{version}/webhooks.json

SimplyPrint API Unreachable¶

Symptoms:

• Print jobs not being created
• /health/dependencies shows SimplyPrint down
• Error logs show SimplyPrint connection failures

Investigation:

1. Check SimplyPrint dashboard manually
2. Verify API credentials: SIMPLYPRINT_API_KEY, SIMPLYPRINT_COMPANY_ID
3. Test API directly with curl

Resolution:

1. Wait for SimplyPrint service recovery
2. Failed jobs will be retried automatically via retry queue
3. Manual retry: POST /api/v1/print-jobs/{id}/retry with API key

Orders Stuck in Processing¶

Symptoms:

• Orders in PROCESSING state for > 60 minutes
• No print job status updates
• Customer complaints about delays

Investigation:

1. Check print job status in dashboard
2. Verify SimplyPrint job status in their dashboard
3. Check for failed webhooks from SimplyPrint
4. Review retry queue: GET /api/v1/admin/retry-queue

Resolution:

1. Force refresh print job status from SimplyPrint
2. Manually update order status if needed
3. Contact SimplyPrint support for print issues
4. Check printer status in SimplyPrint dashboard

Retry Queue Backlog¶

Symptoms:

• More than 50 items in retry queue for > 15 minutes
• Alerts for retry queue backlog
• Failed operations not recovering

Investigation:

1. Check retry queue status
2. Identify failing job types
3. Review error messages in retry entries
4. Check if external services are down

Resolution:

1. Fix underlying issue causing failures
2. Clear old/stale entries if safe:

   DELETE FROM "RetryQueue"
   WHERE status = 'FAILED'
   AND "createdAt" < NOW() - INTERVAL '7 days';
   

3. Increase retry processing capacity if needed

Stock Replenishment Not Running¶

Symptoms:

• Products with deficit > 0 but no STOCK-purpose print jobs being created
• replenishment/status shows enabled: false or no productsNeedingReplenishment
• No StockReplenishmentService log entries

Investigation:

1. Check replenishment status:

   curl https://connect-api.forma3d.be/api/v1/inventory/replenishment/status \
     -H "X-API-Key: $API_KEY"
   

2. Verify environment variables:
3. STOCK_REPLENISHMENT_ENABLED must be true
4. STOCK_REPLENISHMENT_MAX_CONCURRENT — check if limit is already reached
5. STOCK_REPLENISHMENT_ORDER_THRESHOLD — scheduler skips when order queue is too full
6. Check stock levels:

   curl https://connect-api.forma3d.be/api/v1/inventory/stock \
     -H "X-API-Key: $API_KEY"
   

7. Review container logs for scheduler activity:

   ssh root@167.172.45.47
   docker logs forma3d-api --since 1h 2>&1 | grep -i "StockReplenishment"
   

Resolution:

1. If enabled: false — set STOCK_REPLENISHMENT_ENABLED=true in the variable group and redeploy
2. If activeStockJobs >= maxConcurrentStockJobs — wait for existing stock jobs to complete, or increase the max
3. If order queue exceeds threshold — wait for order queue to drain, or increase STOCK_REPLENISHMENT_ORDER_THRESHOLD
4. If products show deficit: 0 — all products are already at or above their minimumStock

StockBatch Stuck in IN_PROGRESS¶

Symptoms:

• Stock batch has been IN_PROGRESS for an unusually long time
• currentStock not incrementing despite print jobs completing in SimplyPrint

Investigation:

1. Check stock batch status in the database:

   SELECT id, "productMappingId", "totalJobs", "completedJobs", status, "createdAt"
   FROM "StockBatch"
   WHERE status = 'IN_PROGRESS'
   ORDER BY "createdAt" DESC;
   

2. Check the associated print jobs:

   SELECT id, status, purpose, "stockBatchId"
   FROM "PrintJob"
   WHERE "stockBatchId" = '<batch-id>';
   

3. If a print job is stuck in PRINTING, see Print Job Stuck in PRINTING Status in the real-world testing guide

Resolution:

1. Force-complete stuck print jobs via the admin API
2. The batch will complete automatically when completedJobs >= totalJobs
3. If the batch is orphaned (all jobs cancelled), manually update:

   UPDATE "StockBatch"
   SET status = 'COMPLETED', "completedAt" = NOW()
   WHERE id = '<batch-id>';
   

Shopify OAuth Token Issues¶

Symptoms:

• API calls to Shopify failing with 401 Unauthorized
• "Token expired" errors in logs
• Orders not being fulfilled
• Dashboard showing "Shop disconnected" status

Investigation:

1. Check OAuth status: GET /api/v1/shopify/oauth/status
2. List connected shops: GET /api/v1/shopify/oauth/shops
3. Check ShopifyShop table for token expiry:

   SELECT "shopDomain", "isActive", "expiresAt", "updatedAt"
   FROM "ShopifyShop"
   WHERE "isActive" = true;
   

4. Review logs for token refresh failures

Resolution:

Token Expired and Refresh Failed:

1. Check if shop was uninstalled in Shopify admin
2. Re-authorize the app:
3. Navigate to: https://connect-api.forma3d.be/shopify/oauth/authorize?shop=<shop-domain>
4. Complete Shopify consent flow
5. Verify new token stored: GET /api/v1/shopify/oauth/shops

App Uninstalled:

1. Check for app/uninstalled webhook in event logs
2. Shop record will show isActive = false
3. Reinstall the app from Shopify admin or via OAuth authorize endpoint

HMAC Verification Failures:

1. Verify SHOPIFY_API_SECRET matches the app in Shopify Partners dashboard
2. Check for clock skew between servers (timestamp validation)
3. Ensure raw body is available for webhook signature verification

Token Encryption Key Issues:

1. If SHOPIFY_TOKEN_ENCRYPTION_KEY changed, tokens cannot be decrypted
2. All shops will need to re-authorize
3. Generate new key: openssl rand -hex 32

Fallback to Legacy Mode: If OAuth issues persist and legacy token is configured:

1. Verify SHOPIFY_SHOP_DOMAIN and SHOPIFY_ACCESS_TOKEN are set
2. Legacy mode activates automatically when OAuth token unavailable
3. Check status: GET /api/v1/shopify/oauth/status should show mode: "legacy"

4. Incident Response¶

Severity Levels¶

Incident Response Process¶

1. Detect - Alert received or user report
2. Assess - Determine severity and impact
3. Communicate - Notify stakeholders if P1/P2
4. Investigate - Follow runbook procedures
5. Resolve - Apply fix or workaround
6. Document - Create incident report
7. Review - Post-incident review (for P1/P2)

Incident Template¶

## Incident Report

**Date:** YYYY-MM-DD
**Severity:** P1/P2/P3/P4
**Duration:** HH:MM - HH:MM (X hours Y minutes)
**Impact:** [Description of user impact]

### Timeline

- HH:MM - Issue detected
- HH:MM - Investigation started
- HH:MM - Root cause identified
- HH:MM - Fix deployed
- HH:MM - Issue resolved

### Root Cause

[Description of what caused the issue]

### Resolution

[What was done to fix it]

### Prevention

[What will be done to prevent recurrence]

### Action Items

- [ ] Action 1
- [ ] Action 2

5. Maintenance Procedures¶

Deploying Updates¶

Updates are deployed automatically via Azure DevOps pipeline when merging to main. For manual deployment:

# SSH to staging/production server
ssh root@<DROPLET_IP>

# Pull latest images
docker-compose pull

# Deploy with zero downtime
docker-compose up -d --no-deps api
docker-compose up -d --no-deps web

# Verify deployment
curl https://<domain>/health

Database Migrations¶

Migrations run automatically during deployment. For manual migration:

# SSH to server
ssh root@<DROPLET_IP>

# Enter API container
docker exec -it forma3d-api sh

# Run migrations
npx prisma migrate deploy

# Exit container
exit

Rolling Back a Migration:

# Identify migration to rollback
npx prisma migrate resolve --rolled-back MIGRATION_NAME

Log Management¶

Logs are automatically rotated by Docker. Manual cleanup:

# SSH to server
ssh root@<DROPLET_IP>

# Check disk usage
df -h

# Clear old Docker logs
docker system prune --volumes -f

# View specific container logs
docker logs forma3d-api --since 1h --tail 100

Backup Procedures¶

Database backups are handled by DigitalOcean Managed Databases (automatic daily backups with 7-day retention).

Manual backup:

# Export database
pg_dump $DATABASE_URL > backup_$(date +%Y%m%d_%H%M%S).sql

# Compress backup
gzip backup_*.sql

Certificate Renewal¶

TLS certificates are automatically renewed by Traefik via Let's Encrypt. Verify renewal:

# Check certificate expiry
echo | openssl s_client -connect connect-api.forma3d.be:443 2>/dev/null | openssl x509 -noout -dates

# Check Traefik logs for renewal activity
docker logs forma3d-traefik 2>&1 | grep -i "acme\|certificate\|renew"

6. Contact Information¶

7. Quick Reference Commands¶

# SSH to staging
ssh root@<STAGING_IP>

# SSH to production
ssh root@<PRODUCTION_IP>

# View running containers
docker ps

# View container logs
docker logs forma3d-api --tail 100 -f
docker logs forma3d-web --tail 100 -f
docker logs forma3d-traefik --tail 100 -f

# Restart API
docker-compose restart api

# Restart all services
docker-compose restart

# Check disk space
df -h

# Check memory
free -h

# Check Docker resource usage
docker stats --no-stream

8. Load Testing¶

Running Load Tests¶

Load tests verify the system can handle expected production workloads (500+ orders/day).

# Run against staging
pnpm load-test:staging

# Run in baseline mode (no threshold failures)
k6 run --env ENV=staging --env BASELINE=true load-tests/k6/scenarios/order-throughput.js

Load Test via Azure DevOps Pipeline¶

1. Navigate to Pipelines in Azure DevOps
2. Click Run pipeline on the main branch
3. Enable "Run Load Tests (optional)"
4. Optionally enable "Load Test Baseline Mode" for data collection without threshold enforcement
5. Click Run

The load test stage runs between Acceptance Tests and Production deployment. Results are available in the HTML Viewer tab with visual metric cards and raw JSON data.

Performance Thresholds¶

Troubleshooting Load Test Failures¶

High failure rate (> 1%):

• Check /health/dependencies for external service issues
• Review Sentry for error spikes during the test window
• Check database connection pool exhaustion (DATABASE_POOL_SIZE)

Slow response times (p95 > 2s):

• Check database query performance in Sentry
• Review N+1 query patterns
• Check for resource contention (CPU/memory) on the droplet

Test seeding failures:

• Verify NODE_ENV is not production (seeding disabled in prod)
• Check if test-seeding module is loaded

9. Shopify Theme Operations¶

Theme Location¶

The Forma3D Shopify theme is located at deployment/shopify-theme/.

Deploying Theme Updates¶

Using Shopify CLI¶

# Navigate to theme directory
cd deployment/shopify-theme

# Push to staging store
shopify theme push --store forma3d-dev.myshopify.com

# Push to production store (with confirmation)
shopify theme push --store forma3d.myshopify.com

Manual Deployment¶

1. Create ZIP: cd deployment && zip -r forma3d-theme.zip shopify-theme/
2. Upload in Shopify Admin → Online Store → Themes → Add theme

Theme Health Checks¶

# Verify theme files are valid
cd deployment/shopify-theme
shopify theme check

# Test theme locally
shopify theme dev --store forma3d-dev.myshopify.com

Common Theme Issues¶

CSS Not Loading¶

Symptoms: - Unstyled page content - Missing colors, fonts, spacing

Resolution: 1. Check layout/theme.liquid includes stylesheets:

{{ 'base.css' | asset_url | stylesheet_tag }}
{{ 'components.css' | asset_url | stylesheet_tag }}
{{ 'sections.css' | asset_url | stylesheet_tag }}

Cart Drawer Not Working¶

Symptoms: - Cart button doesn't open drawer - Add to cart doesn't update count

Resolution: 1. Check global.js is loaded in theme.liquid 2. Verify cart drawer snippet exists: snippets/cart-drawer.liquid 3. Check browser console for JavaScript errors

IKEA Badges Missing¶

Symptoms: - Products don't show compatibility badges

Resolution: 1. Verify products have tags starting with IKEA- (e.g., IKEA-FÄRGKLAR) 2. Check Theme Settings → Product → "Show IKEA Compatibility Badge" is enabled 3. Verify product-card.liquid snippet is rendering badges

Theme Backup¶

Before major changes, backup the current theme:

# Pull current theme from Shopify
cd deployment
shopify theme pull --store forma3d-dev.myshopify.com --path shopify-theme-backup-$(date +%Y%m%d)

Theme Customization Reference¶

Logo Assets¶

The theme includes Forma3D logos as both SVG files and Liquid snippets:

Revision History: