STL Preview Cache Pre-Population — Research¶

Date: 2026-02-27 Status: Superseded — the full-preview-per-dimension approach documented here was replaced by the plate-level cache (ADR-061). The legacy cache (16,471 files, ~32 GB) and its scripts (populate-preview-cache.ts, upload-preview-cache.sh) have been removed. This document is retained as historical research.

1. Problem Statement¶

Currently, STL previews are generated on-demand when a customer first views a specific grid dimension in the storefront configurator. This means:

• First-time loads for a given dimension take 12–30+ seconds (depending on grid size).
• Only after the first request is the STL cached to disk for instant subsequent loads.

The goal is to pre-populate the entire cache so that every possible dimension combination is already generated before any customer requests it, eliminating all cold-start latency.

2. Combinatorics¶

Default parameters¶

Combination count formula¶

Since width and height are interchangeable, we need unordered pairs with repetition (a 30×30 grid is valid — same width and height):

n = (maxMm − minMm) / incrementMm + 1
combinations = n × (n + 1) / 2

With the defaults: n = (1000 − 100) / 5 + 1 = 181, combinations = 181 × 182 / 2 = 16,471.

Configurable range and increment¶

The admin should be able to configure three parameters when starting population:

The max dimension default is already stored in SystemConfig (key gridflock.max_dimension_mm, fallback DEFAULT_MAX_DIMENSION_MM = 1000 from constants.ts). The populate endpoint should read this value as the default but allow the admin to override it.

Impact of parameter changes on combination count and storage¶

These estimates use the same 13 KB/cell model but adjusted for the smaller average grid size when the range is restricted.

UI for parameter configuration¶

The populate dialog should expose these fields with sensible defaults:

┌─────────────────────────────────────────────────────────┐
│  Populate Preview Cache                                 │
│                                                         │
│  Dimension range (mm):                                  │
│  ┌─────────┐      ┌──────────┐      ┌────────┐         │
│  │   100   │  to  │   1000   │  by  │    5   │  mm     │
│  └─────────┘      └──────────┘      └────────┘         │
│                                                         │
│  This will generate 16,471 previews (~32 GB).           │
│  Estimated time: ~69 hours.                             │
│  Already cached: 89 (will be skipped).                  │
│                                                         │
│  ⚠ The server will be under sustained CPU load.         │
│                                                         │
│            [Cancel]    [Start Population]                │
└─────────────────────────────────────────────────────────┘

The combination count, estimated storage, and estimated time should recalculate live as the admin adjusts the parameters (pure client-side math, no API call needed).

Validation rules¶

• minMm must be ≥ 42 (at least 1 Gridfinity grid unit) and < maxMm.
• maxMm must be ≤ DEFAULT_MAX_DIMENSION_MM (or the tenant's configured max).
• incrementMm must divide (maxMm − minMm) evenly.
• incrementMm must be ≥ 1 mm (sub-millimeter increments produce near-identical STLs since floor(d / 42) groups many mm values to the same grid count anyway).

Cache key format¶

Current format: preview-{widthMm}x{heightMm}.stl

Examples: preview-450x320.stl, preview-1000x500.stl

Important finding: The current cache key is NOT normalized — a request for 450×320 produces preview-450x320.stl, while 320×450 would produce preview-320x450.stl. These are treated as separate cache entries even though the resulting grids are physically identical (just rotated). The computeSku() function in plate-set-calculator.ts already normalizes (larger dimension first), but generatePreview() does not.

Recommendation: Before implementing pre-population, normalize the cache key so the larger dimension always comes first (e.g., preview-450x320.stl for both 450×320 and 320×450). This halves lookup misses and prevents duplicate entries.

3. File Size Estimation¶

Empirical data (from staging, 2 vCPU droplet)¶

Grid cells = floor(width / 42) × floor(height / 42) where 42 mm is the Gridfinity grid unit.

Model¶

The data shows a remarkably consistent rate of ~13 KB per grid cell in the binary STL output. Per-plate overhead (base geometry, connectors, borders) is negligible relative to cell geometry.

estimated_file_size ≈ 13 KB × floor(w / 42) × floor(h / 42)

Size distribution across all 16,471 combinations¶

Total estimated storage¶

Using the 13 KB/cell model across all 16,471 unordered pairs:

• Weighted average: ~159 grid cells per combination → ~2.1 MB average file size
• Total estimate: 16,471 × 2.1 MB ≈ 32 GB (range: 25–40 GB)

Size extremes: - Smallest (100×100): 2×2 = 4 cells → ~52 KB - Largest (1000×1000): 23×23 = 529 cells → ~6.9 MB

Storage consideration: 32 GB is manageable on a Docker volume on the current staging droplet (which has ~75 GB disk). For production, ensure the volume has at least 50 GB headroom to accommodate the cache plus growth.

4. Generation Time Estimation¶

Empirical observations (2 vCPU staging server, 2 workers)¶

Generation time scales roughly linearly with the number of plates (and thus grid cells).

Estimate per combination¶

Total generation time (sequential, full population)¶

Weighted average per combination: ~15 seconds

16,471 × 15 s  =  247,065 s  ≈  69 hours  ≈  2.9 days

On a beefier machine (e.g., 4–8 vCPUs in production), generation would be proportionally faster due to more parallel workers, but the relationship is not perfectly linear because each combination already saturates available workers for its own plates.

Realistic estimate: 50–100 hours depending on hardware and system load.

5. Architecture Design¶

5.1 Available infrastructure¶

The codebase already has:

No new dependencies are needed.

5.2 Recommended approach: BullMQ queue + Socket.io progress¶

┌──────────────────┐     POST /populate      ┌────────────────────────┐
│   Admin UI       │ ──────────────────────>  │  PreviewCacheController │
│   (React)        │                          │  (NestJS)               │
│                  │ <── Socket.io events ──  │                         │
│  [Populate]      │     progress, done       └──────────┬─────────────┘
│  [Pause]         │                                     │
│  [Resume]        │                          ┌──────────▼─────────────┐
│  [Stop]          │                          │  BullMQ Queue          │
│                  │                          │  "preview-populate"    │
│  Progress: 42%   │                          │  16,471 jobs           │
│  1234 / 16471    │                          └──────────┬─────────────┘
│  Skipped: 89     │                                     │
│  ETA: ~38h       │                          ┌──────────▼─────────────┐
└──────────────────┘                          │  PopulateProcessor     │
                                              │  (Worker)              │
                                              │                        │
                                              │  For each job:         │
                                              │  1. Check if cached    │
                                              │  2. Skip if exists     │
                                              │  3. Generate + cache   │
                                              │  4. Emit progress      │
                                              └────────────────────────┘

5.3 Queue strategy¶

One job per dimension combination (16,471 jobs):

• BullMQ handles ordering, retries, and concurrency natively.
• Each job payload: { widthMm: number, heightMm: number }.
• Skip logic: check if preview-{w}x{h}.stl already exists → mark complete immediately.
• Concurrency: 1 (each combination already uses all CPU cores for plate parallelism).

5.4 Pause / Resume / Stop¶

BullMQ provides built-in support:

No custom state machine needed — these are reliable, battle-tested BullMQ primitives.

5.5 Progress tracking¶

Two complementary channels:

1. Polling endpoint (simple, works without WebSocket):

   GET /api/v1/gridflock/preview-cache/populate/status
   → { state, total, completed, skipped, failed, elapsedMs, estimatedRemainingMs }
   

   Uses queue.getJobCounts() to return waiting/active/completed/failed counts.

2. Socket.io events (real-time, optional enhancement):

   Event: "preview-cache:progress"
   Payload: { completed, total, skipped, currentDimensions, elapsedMs }
   

The polling approach is simpler and sufficient for the admin UI (poll every 2–5 seconds). Socket.io can be added later for a smoother UX without additional infrastructure.

5.6 Job ordering strategy¶

Generate smallest combinations first (fewest grid cells → fastest jobs):

1. Provides quick early feedback ("it's working").
2. Fills the most commonly requested dimensions first (smaller grids are more popular).
3. Gives better ETA estimates after the first batch completes.

Sort by floor(w/42) × floor(h/42) ascending before enqueueing.

5.7 Resilience¶

6. API Design¶

Endpoints¶

POST   /api/v1/gridflock/preview-cache/populate          Start population
POST   /api/v1/gridflock/preview-cache/populate/pause     Pause
POST   /api/v1/gridflock/preview-cache/populate/resume    Resume
DELETE /api/v1/gridflock/preview-cache/populate            Stop (drain queue)
GET    /api/v1/gridflock/preview-cache/populate/status     Get progress

All endpoints require PERMISSIONS.ADMIN.

Start request body¶

{
  "minMm": 100,
  "maxMm": 1000,
  "incrementMm": 5
}

All fields are optional; the server uses defaults when omitted (minMm: 100, maxMm: SystemConfig value or 1000, incrementMm: 5).

Start response¶

{
  "total": 16471,
  "skippedExisting": 89,
  "queued": 16382,
  "estimatedTimeSeconds": 245730
}

Status response¶

{
  "state": "running",
  "total": 16471,
  "completed": 1234,
  "skipped": 89,
  "failed": 2,
  "active": 1,
  "waiting": 15146,
  "elapsedSeconds": 18540,
  "estimatedRemainingSeconds": 227190,
  "currentDimensions": "550 x 420 mm"
}

7. Frontend Design¶

UI on the existing Preview Cache page¶

Add a "Populate Cache" section above the file list:

┌─────────────────────────────────────────────────────┐
│  Cache Pre-Population                               │
│                                                     │
│  Fill the cache with all 16,471 possible dimension  │
│  combinations. Already-cached dimensions are        │
│  skipped automatically.                             │
│                                                     │
│  ┌──────────────────────────────────────────────┐   │
│  │ ████████████░░░░░░░░░░░░░░░░░░░  42%        │   │
│  └──────────────────────────────────────────────┘   │
│  6,912 / 16,471 completed  ·  89 skipped  ·  2 err │
│  Elapsed: 5h 09m  ·  ETA: ~7h 12m                  │
│  Currently generating: 550 × 420 mm                 │
│                                                     │
│  [⏸ Pause]  [⏹ Stop]                               │
│                                                     │
│  (when idle)                                        │
│  [▶ Populate Cache]                                 │
└─────────────────────────────────────────────────────┘

States¶

Confirmation dialogs¶

• Populate: Shows the parameter configuration form (see Section 2) with live-updating counts. After clicking "Start Population", a confirmation: "This will generate ~16,382 previews (~32 GB). Estimated time: ~69 hours. The server will be under sustained CPU load during this time. Continue?"
• Stop: "This will cancel all remaining jobs. Already-generated previews are kept. Continue?"

8. Key Implementation Considerations¶

8.1 Cache key normalization (prerequisite)¶

Before implementing pre-population, the preview cache key must be normalized so the larger dimension always comes first:

// Before (current — NOT normalized):
const filename = `preview-${dto.widthMm}x${dto.heightMm}.stl`;

// After (normalized):
const w = Math.max(dto.widthMm, dto.heightMm);
const h = Math.min(dto.widthMm, dto.heightMm);
const filename = `preview-${w}x${h}.stl`;

This must be applied to both generatePreview() (cache read/write) and the population job producer. Without normalization, a customer requesting 320×450 would miss the cache entry stored as preview-450x320.stl.

8.2 Separate queue from order processing¶

The population queue (preview-populate) must be separate from the existing GRIDFLOCK_GENERATION_QUEUE used for order processing. Order generation must never be blocked or delayed by cache population.

8.3 Throttling and CPU impact¶

Each preview generation saturates all available CPU cores (2 workers on 2 vCPUs). During pre-population:

• Order processing (which also uses CPU for STL generation) would compete for resources.
• Consider adding a configurable delay between jobs (e.g., 1–2 seconds) to leave breathing room.
• Alternatively, run population during off-peak hours only (night/weekend).
• On the current 2-vCPU staging server, population will make the server noticeably slower for other operations.

8.4 Memory pressure¶

Each STL generation loads JSCAD geometry into memory. For large grids (500+ cells), this can consume significant RAM. With concurrency: 1 on the population queue, only one combination is generated at a time, keeping memory usage bounded.

8.5 Dimension values in millimeters¶

The dimension range is 10 cm to 100 cm in 0.5 cm increments. In millimeters: 100, 105, 110, 115, ..., 995, 1000 (181 values, step = 5 mm).

The population job producer must enumerate all unordered pairs:

for (let w = 100; w <= 1000; w += 5) {
  for (let h = w; h <= 1000; h += 5) {
    // (w, h) is a unique normalized pair
    queue.add('populate', { widthMm: w, heightMm: h });
  }
}

9. Offline Generation on a Powerful Machine¶

9.1 Concept¶

Instead of generating STLs on the staging/production server (which has limited CPU and serves live traffic), generate the entire cache offline on a powerful multi-core machine, then upload the resulting files to the server.

┌───────────────────────────┐         scp / rsync          ┌─────────────────────┐
│  Powerful local machine   │ ───────────────────────────>  │  Staging server     │
│  (16+ cores)              │                               │  (2 vCPU)           │
│                           │    preview-100x100.stl        │                     │
│  $ npx ts-node            │    preview-100x105.stl        │  /data/gridflock/   │
│      populate-cache.ts    │    preview-100x110.stl        │    preview-cache/   │
│      --min 100            │    ...                        │                     │
│      --max 1000           │    preview-1000x1000.stl      │                     │
│      --increment 5        │    (16,471 files, ~32 GB)     │                     │
│      --out ./cache-output │                               │                     │
└───────────────────────────┘                               └─────────────────────┘

9.2 What the gridflock-core library needs to run standalone¶

The @forma3d/gridflock-core library is a pure computation library with no server dependencies. It requires only:

It does not depend on NestJS, Prisma, Redis, BullMQ, or any server infrastructure. The key functions needed are all exported from the library's public API:

• calculatePlateSet() — compute plate layout for a dimension pair
• generatePlatesParallel() — generate all plates using worker threads
• combineStlBuffers() — merge plates into a single STL
• PRINTER_PROFILES['bambu-a1'] — printer profile constants
• DEFAULT_MAGNET_PARAMS, DEFAULT_EDGE_PUZZLE_PARAMS — generation defaults

9.3 Standalone CLI script design¶

A standalone script can reuse the exact same generation logic as GridflockPreviewService.generatePreview():

// scripts/populate-cache.ts
import {
  calculatePlateSet, generatePlatesParallel, combineStlBuffers,
  PRINTER_PROFILES, DEFAULT_MAGNET_PARAMS, DEFAULT_EDGE_PUZZLE_PARAMS,
  GRID_UNIT_MM, type StlPlacement, type PlateGenerationOptions,
} from '@forma3d/gridflock-core';
import { writeFileSync, existsSync, mkdirSync } from 'node:fs';
import { join } from 'node:path';

const PLATE_GAP_MM = 10;
const OPTIONS: PlateGenerationOptions = {
  connectorType: 'intersection-puzzle',
  magnets: { ...DEFAULT_MAGNET_PARAMS, enabled: false },
  edgePuzzle: DEFAULT_EDGE_PUZZLE_PARAMS,
};

async function generateOne(widthMm: number, heightMm: number, outDir: string) {
  // Normalize: larger dimension first
  const w = Math.max(widthMm, heightMm);
  const h = Math.min(widthMm, heightMm);
  const filename = `preview-${w}x${h}.stl`;
  const outPath = join(outDir, filename);

  if (existsSync(outPath)) return 'skipped';

  const plateSet = calculatePlateSet({
    targetWidthMm: w,
    targetDepthMm: h,
    printerProfile: PRINTER_PROFILES['bambu-a1'],
    ...OPTIONS,
  });

  const results = await generatePlatesParallel(plateSet.plates, OPTIONS);

  // Same offset logic as GridflockPreviewService
  const placements: StlPlacement[] = plateSet.plates.map((plate, i) => ({
    stlBuffer: results[i].stlBuffer,
    offsetX: /* computed offsets */,
    offsetY: /* computed offsets */,
  }));

  const combined = combineStlBuffers(placements);
  writeFileSync(outPath, combined);
  return 'generated';
}

// Main loop: enumerate all combinations
async function main() {
  const { min, max, increment, outDir } = parseArgs();
  mkdirSync(outDir, { recursive: true });

  const pairs = [];
  for (let w = min; w <= max; w += increment) {
    for (let h = w; h <= max; h += increment) {
      pairs.push([w, h]);
    }
  }

  // Sort by grid cell count (smallest first)
  pairs.sort((a, b) =>
    Math.floor(a[0]/42) * Math.floor(a[1]/42) -
    Math.floor(b[0]/42) * Math.floor(b[1]/42)
  );

  let completed = 0, skipped = 0;
  for (const [w, h] of pairs) {
    const result = await generateOne(w, h, outDir);
    if (result === 'skipped') skipped++;
    completed++;
    // Progress output every 100 combinations
    if (completed % 100 === 0) {
      console.log(`${completed}/${pairs.length} (${skipped} skipped)`);
    }
  }
}

This script would live in the monorepo and import from @forma3d/gridflock-core directly (using the workspace's TypeScript build). It can be run with npx ts-node or compiled and run with node.

9.4 Parallelism on multi-core machines¶

The current parallel-generator.ts uses MAX_WORKERS = Math.max(2, cpus().length). Each combination distributes its plates across all available workers. However, the parallelism per combination is limited by the number of plates:

Key insight: For small combinations (which are the majority), most cores sit idle. To fully utilize a powerful machine, the CLI script should run multiple combinations concurrently with a pool-based approach:

Strategy A: Sequential combinations, all cores per combination (current behavior)
  → Simple but wastes cores on small combinations

Strategy B: Concurrent combinations with adaptive worker count
  → Run N combinations in parallel, each gets cpus()/N workers
  → Better utilization but requires modifying MAX_WORKERS behavior

Strategy C: Fixed concurrency pool (recommended)
  → Run 4 combinations concurrently on a 16-core machine
  → Each combination gets ~4 workers (cpus()/4)
  → Simple to implement: use a promise pool (e.g., p-limit)

Recommendation: Strategy C is the best balance of simplicity and performance. The CLI script would use a concurrency pool (e.g., p-limit or a simple semaphore):

import pLimit from 'p-limit';

const CONCURRENCY = Math.max(1, Math.floor(cpus().length / 4));
const limit = pLimit(CONCURRENCY);

// Override MAX_WORKERS for this session
process.env['POPULATE_WORKERS_PER_JOB'] = String(
  Math.floor(cpus().length / CONCURRENCY)
);

await Promise.all(
  pairs.map(([w, h]) => limit(() => generateOne(w, h, outDir)))
);

This requires a small change to parallel-generator.ts to respect an environment variable for worker count (or pass it as a parameter to generatePlatesParallel).

9.5 Performance estimates by machine¶

Cloud cost (AWS on-demand, us-east-1): - c6i.4xlarge: \(0.68/hr × 14h = **~\)10** - c6i.8xlarge: \(1.36/hr × 10h = **~\)14** - c7g.16xlarge: \(2.18/hr × 5h = **~\)11**

The cloud option is remarkably cheap and would complete in a fraction of the time. A spot instance would reduce cost by ~60–70%.

9.6 Transfer to staging/production¶

After generation, the cache files need to be uploaded to the server's Docker volume.

Option A: tar + scp (simple, one-shot)

# On the generation machine
tar czf preview-cache.tar.gz -C ./cache-output .

# Upload to server (~32 GB compressed to ~20 GB with gzip)
scp -i ~/.ssh/key preview-cache.tar.gz root@server:/tmp/

# On the server
docker exec -u 0 forma3d-gridflock-service \
  tar xzf /tmp/preview-cache.tar.gz -C /data/gridflock/preview-cache/
rm /tmp/preview-cache.tar.gz

Estimated transfer time at 100 Mbps: ~20 GB / 12.5 MB/s ≈ ~27 minutes.

Option B: rsync (incremental, resumable)

# Sync only new/changed files (resumable if interrupted)
rsync -avz --progress \
  ./cache-output/ \
  root@server:/data/gridflock/preview-cache/

rsync is preferred for incremental updates (e.g., after adding new dimension ranges) and can resume interrupted transfers.

Option C: Direct Docker volume mount via SSH

# Stream tar directly into the container (no temp file on server)
tar czf - -C ./cache-output . | \
  ssh root@server "docker exec -i -u 0 forma3d-gridflock-service \
    tar xzf - -C /data/gridflock/preview-cache/"

File ownership: The cache directory is owned by nestjs:nodejs (UID 1001:1001). After uploading via root, fix permissions:

docker exec -u 0 forma3d-gridflock-service \
  chown -R nestjs:nodejs /data/gridflock/preview-cache/

9.7 Verifying cache integrity after upload¶

After uploading, the admin can use the existing Preview Cache screen to verify:

1. File count matches expected combinations.
2. Total size is within the expected range.
3. Spot-check a few dimensions in the storefront configurator (should load instantly).

A verification endpoint could also be added:

GET /api/v1/gridflock/preview-cache/populate/coverage
  ?minMm=100&maxMm=1000&incrementMm=5

→ { total: 16471, cached: 16471, missing: 0, coverage: "100%" }

9.8 Pros and cons of offline vs. server-side generation¶

9.9 Recommendation¶

Both approaches should be implemented — they serve different use cases:

1. Server-side BullMQ population for:
2. Small/incremental runs (e.g., after changing max dimension setting)
3. "Quick populate" with coarse increment (100 mm step = 55 combinations, ~14 minutes)

4. Topping off the cache after individual deletions

5. Offline CLI script for:

6. Initial full population of 16,471 combinations
7. Re-generating the full cache after STL geometry changes (e.g., connector redesign)
8. Environments where the server cannot afford sustained CPU load

10. Other Alternatives Considered¶

10.1 Pre-compute during CI/CD¶

Generate the full cache as a build step and bake it into the Docker image or push to an object store.

• Pro: Fully deterministic, no runtime cost.
• Con: 32 GB Docker image is impractical; would need object store.
• Con: Build times would increase by ~70 hours (unacceptable).
• Verdict: Not viable with current generation speed.

10.2 Lazy population with priority queue¶

Instead of generating all 16,471 up front, pre-populate only the most common dimensions (e.g., multiples of 5 cm → ~361 combinations) and let the rest fill on-demand.

• Pro: Covers the common cases with ~2% of the work.
• Con: Long tail of uncommon dimensions still has cold starts.
• Verdict: Good pragmatic middle ground. Could be offered as a "quick populate" option.

11. Summary¶

Recommended implementation order¶

1. Normalize cache keys (larger dimension first) — small change, big prerequisite.
2. Add configurable parameters to populate API (minMm, maxMm, incrementMm with defaults).
3. Add preview-populate BullMQ queue with processor.
4. Add API endpoints (start, pause, resume, stop, status, coverage).
5. Add frontend UI (populate button, parameter fields, progress bar, controls).
6. Create offline CLI script (scripts/populate-cache.ts) with concurrent pool.
7. Optional: Add Socket.io real-time progress events.
8. Optional: Add coverage verification endpoint.