AI Prompt: STL Preview Cache Pre-Population — Offline Scripts¶

Purpose: Create two scripts: (1) a CLI script to generate all STL preview cache files offline on a powerful machine, and (2) a shell script to upload the generated files to the staging server via rsync Estimated Effort: 4–6 hours Prerequisites: Working @forma3d/gridflock-core library, SSH access to staging server Research: docs/03-architecture/research/stl-cache-prepopulation-research.md Output: Two production-ready scripts in scripts/ that together eliminate all cold-start latency for STL previews Status: ✅ DONE (2026-02-27)

🎯 Mission¶

Create two standalone scripts that, together, pre-populate the STL preview cache on the staging server:

scripts/populate-preview-cache.ts — A Node.js CLI script that generates all STL preview files locally on a powerful machine, writing them to a local output directory.
scripts/upload-preview-cache.sh — A shell script that uploads the generated files to the staging server's cache directory using rsync.

This is the simplified, offline-only approach described in Section 9 of the research document. No server-side changes, no BullMQ queue, no admin UI, no API endpoints.

📐 Architecture¶

┌───────────────────────────┐         rsync over SSH          ┌─────────────────────────┐
│  Powerful local machine   │ ──────────────────────────────> │  Staging server         │
│  (8–16+ cores)            │                                 │  (Docker container)     │
│                           │    preview-100x100.stl          │                         │
│  $ pnpm tsx               │    preview-105x100.stl          │  /data/gridflock/       │
│      scripts/populate-    │    preview-110x100.stl          │    preview-cache/       │
│        preview-cache.ts   │    ...                          │                         │
│      --min 100            │    preview-1000x1000.stl        │                         │
│      --max 1000           │                                 │                         │
│      --increment 5        │                                 │                         │
│      --out ./cache-output │                                 │                         │
└───────────────────────────┘                                 └─────────────────────────┘

📋 Implementation¶

Phase 1: Extract Preview Generation into `@forma3d/gridflock-core` (1–2 hours)¶

Priority: P0 | Impact: Critical | Dependencies: None

Currently, the entire preview generation pipeline — plate offset computation, orchestration of calculatePlateSet → compute offsets → generatePlatesParallel → build placements → combineStlBuffers, and the associated constants (PLATE_GAP_MM, DEFAULT_PREVIEW_OPTIONS) — lives inside the NestJS service class GridflockPreviewService in apps/gridflock-service/src/gridflock/gridflock-preview.service.ts. This logic has no dependency on NestJS and is purely computational.

To ensure a single source of truth (so changes to STL generation only need to happen in one place), extract this logic into @forma3d/gridflock-core.

1. Add `maxWorkers` parameter to `generatePlatesParallel()`¶

In libs/gridflock-core/src/lib/parallel-generator.ts, the worker count is currently hardcoded:

const MAX_WORKERS = Math.max(2, cpus().length);

This prevents callers from controlling CPU usage. When running multiple combinations concurrently (essential for the population script), each combination would spawn cpus().length workers, creating massive thread oversubscription (e.g., 4 concurrent combinations × 16 workers = 64 threads on 16 cores).

Add an optional maxWorkers parameter to the function signature:

export async function generatePlatesParallel(
  plates: PlateSpec[],
  options: PlateGenerationOptions,
  log?: (msg: string) => void,
  maxWorkers?: number,
): Promise<ParallelPlateResult[]>

Inside the function, use the parameter when provided, falling back to the current default:

const workerLimit = maxWorkers ?? Math.max(2, cpus().length);
const concurrency = Math.min(plates.length, workerLimit);

This is a backward-compatible change — all existing callers pass no maxWorkers argument and get identical behavior.

2. Create `generatePreviewStl()` in gridflock-core¶

Create a new file libs/gridflock-core/src/lib/preview-generator.ts that exports a single function:

export interface PreviewStlOptions {
  maxWorkers?: number;
  log?: (msg: string) => void;
}

export async function generatePreviewStl(
  widthMm: number,
  heightMm: number,
  options?: PreviewStlOptions,
): Promise<Buffer>

This function encapsulates the full preview generation pipeline, exactly replicating what GridflockPreviewService.generatePreview() does today (minus the caching). Specifically:

Uses PRINTER_PROFILES['bambu-a1'] as the printer profile
Uses DEFAULT_PREVIEW_OPTIONS (intersection-puzzle connectors, magnets disabled)
Calls calculatePlateSet() to compute the plate layout
Computes plate offsets using the offset helpers (moved here from the service)
Calls generatePlatesParallel(plates, options, log, maxWorkers) — passing through the maxWorkers parameter so callers can control CPU usage
Builds StlPlacement[] with computed offsets
Calls combineStlBuffers() to produce the final STL buffer

Move these into preview-generator.ts (they are currently private methods on the service class — extract them as standalone functions):

computeUniformOffsets() — lines 176–203 of gridflock-preview.service.ts
computeOffsetsPerColumn() — lines 209–242 of gridflock-preview.service.ts
PLATE_GAP_MM = 10 — line 17
DEFAULT_PREVIEW_OPTIONS — lines 21–25

3. Export from gridflock-core index¶

Add to libs/gridflock-core/src/index.ts:

export { generatePreviewStl, type PreviewStlOptions } from './lib/preview-generator';

4. Refactor `GridflockPreviewService` to use the new function¶

The service's generatePreview() method should become a thin wrapper that handles only caching and logging:

async generatePreview(dto: PreviewGridDto): Promise<Buffer> {
  const w = Math.max(dto.widthMm, dto.heightMm);
  const h = Math.min(dto.widthMm, dto.heightMm);
  const filename = `preview-${w}x${h}.stl`;

  const cached = await this.readFromDisk(filename);
  if (cached) {
    this.logger.log(`Preview cache hit: ${filename}`);
    return cached;
  }

  this.logger.log(`Generating preview for ${w}x${h}mm...`);
  const combined = await generatePreviewStl(w, h, {
    log: (msg) => this.logger.log(msg),
  });
  this.logger.log(`Preview generated: ${combined.length} bytes`);

  await this.writeToDisk(filename, combined);
  return combined;
}

The service passes no maxWorkers, so generatePlatesParallel uses the default cpus().length — identical to current behavior. Only the population script overrides this for controlled parallelism.

This also applies the cache key normalization (larger dimension first) — which is the prerequisite from the research document (Section 8.1). The computeUniformOffsets, computeOffsetsPerColumn, PLATE_GAP_MM, and DEFAULT_PREVIEW_OPTIONS private members are removed from the service class entirely since they now live in gridflock-core.

5. Verify with a manual test¶

After the refactoring: - Requesting a preview for 320×450 should produce preview-450x320.stl (normalized key) - The STL output must be byte-for-byte identical to the current implementation for the same dimensions - The existing Preview Cache admin screen (file listing, delete) should work unchanged since it reads filenames from disk

Phase 2: Generation Script — `scripts/populate-preview-cache.ts` (2–3 hours)¶

Priority: P0 | Impact: Critical | Dependencies: Phase 1

Create a standalone CLI script that generates all STL preview files locally. This script imports generatePreviewStl from @forma3d/gridflock-core (the function extracted in Phase 1) and has no dependency on NestJS, Prisma, Redis, or any server infrastructure.

Because all generation logic now lives in @forma3d/gridflock-core, the script itself is thin — it only handles CLI arguments, combination enumeration, concurrency, file I/O, and progress reporting.

1. Script location and runner¶

File: scripts/populate-preview-cache.ts
Run with: pnpm tsx scripts/populate-preview-cache.ts [options]
The monorepo's TypeScript path resolution already allows importing @forma3d/gridflock-core

2. CLI arguments¶

Use a simple argument parser (e.g., parseArgs from node:util — Node 18.3+ built-in, no extra dependency needed):

Argument	Default	Description
`--min`	`100`	Minimum dimension in mm
`--max`	`1000`	Maximum dimension in mm
`--increment`	`5`	Step size in mm
`--out`	`./preview-cache-output`	Output directory
`--concurrency`	`auto`	Number of concurrent combinations (auto = `floor(cpus / 4)`, min 1)
`--dry-run`	`false`	Only count combinations and estimate storage, don't generate

3. Validation¶

Before starting generation, validate inputs:

min must be ≥ 42 (at least 1 grid unit of 42 mm) and < max
max must be ≤ 2000
increment must be ≥ 1 and must divide (max - min) evenly
out directory is created if it doesn't exist

4. Combination enumeration¶

Enumerate all unordered pairs (since dimensions are interchangeable — a 30×40 grid is the same as 40×30):

const pairs: [number, number][] = [];
for (let w = min; w <= max; w += increment) {
  for (let h = w; h <= max; h += increment) {
    pairs.push([w, h]);
  }
}

With defaults (100–1000, step 5): n = 181, combinations = 181 × 182 / 2 = 16,471.

Sort by grid cell count ascending (smallest first) for quick early feedback:

import { GRID_UNIT_MM } from '@forma3d/gridflock-core';

pairs.sort((a, b) =>
  Math.floor(a[0] / GRID_UNIT_MM) * Math.floor(a[1] / GRID_UNIT_MM) -
  Math.floor(b[0] / GRID_UNIT_MM) * Math.floor(b[1] / GRID_UNIT_MM)
);

5. Generation logic¶

The generation for each combination is a single call to the generatePreviewStl function extracted in Phase 1. No generation logic is duplicated in the script.

import { generatePreviewStl } from '@forma3d/gridflock-core';
import { existsSync, writeFileSync } from 'node:fs';
import { join } from 'node:path';

async function generateOne(w: number, h: number, outDir: string): Promise<'generated' | 'skipped'> {
  const filename = `preview-${w}x${h}.stl`;
  const outPath = join(outDir, filename);

  if (existsSync(outPath)) return 'skipped';

  const combined = await generatePreviewStl(w, h);
  writeFileSync(outPath, combined);
  return 'generated';
}

That's it. The script does not import calculatePlateSet, generatePlatesParallel, combineStlBuffers, offset helpers, or any constants like PLATE_GAP_MM or DEFAULT_PREVIEW_OPTIONS. All of that is encapsulated inside generatePreviewStl() in gridflock-core. If the STL generation logic ever changes, it changes in one place.

6. Automatic CPU-aware parallelism¶

The script must fully utilize all available cores without oversubscription. This is critical for performance across 16,000+ combinations.

The problem: Each combination calls generatePreviewStl() which internally calls generatePlatesParallel() to distribute plate generation across worker threads. Small combinations (1–2 plates, which are the majority) only need 1–2 workers, leaving most cores idle. Running combinations sequentially on a 16-core machine wastes 14 cores most of the time.

The solution: Run multiple combinations concurrently, each with a limited worker count, so the total worker threads across all concurrent combinations equals the available CPU cores.

import pLimit from 'p-limit';
import { cpus } from 'node:os';
import { generatePreviewStl } from '@forma3d/gridflock-core';

const cores = cpus().length;
const concurrency = Math.max(1, Math.floor(cores / 4));
const workersPerJob = Math.max(2, Math.floor(cores / concurrency));

const limit = pLimit(concurrency);

await Promise.all(
  pairs.map(([w, h]) =>
    limit(() => generateOne(w, h, outDir, workersPerJob))
  )
);

Where generateOne passes maxWorkers through to generatePreviewStl:

async function generateOne(
  w: number, h: number, outDir: string, maxWorkers: number,
): Promise<'generated' | 'skipped'> {
  const filename = `preview-${w}x${h}.stl`;
  const outPath = join(outDir, filename);

  if (existsSync(outPath)) return 'skipped';

  const combined = await generatePreviewStl(w, h, { maxWorkers });
  writeFileSync(outPath, combined);
  return 'generated';
}

How cores are partitioned by machine:

Machine	Cores	Concurrent jobs	Workers/job	Total threads
2 vCPU staging server	2	1	2	2 (no oversubscription)
8-core laptop (M-series)	8	2	4	8
10-core laptop (M-series)	10	2	5	10
16-core workstation	16	4	4	16
32 vCPU cloud instance	32	8	4	32
64 vCPU cloud instance	64	16	4	64

The script auto-detects the CPU count at startup and prints the computed parallelism settings in the header. The --concurrency CLI argument allows overriding the auto-detected value (the workersPerJob is then recalculated as floor(cores / concurrency)).

Add p-limit as a dev dependency: pnpm add -D p-limit

7. Progress reporting¶

Print progress to stdout at regular intervals:

STL Preview Cache Population
=============================
Range: 100–1000 mm, increment: 5 mm
Combinations: 16,471
Output: ./preview-cache-output
CPUs: 16, concurrent jobs: 4, workers/job: 4

[00:00:05]  100 / 16,471  (0.6%)  |  97 generated, 3 skipped  |  ETA: ~24h
[00:00:12]  200 / 16,471  (1.2%)  |  195 generated, 5 skipped  |  ETA: ~16h
...
[14:23:45]  16,471 / 16,471  (100%)  |  16,382 generated, 89 skipped

Done! Generated 16,382 files (89 skipped) in 14h 23m 45s.
Total size: 31.7 GB

Key details: - Print a summary header before starting - Print a progress line every 100 completed combinations (or every 30 seconds, whichever comes first) - Show elapsed time, count, percentage, generated vs skipped, and ETA - ETA is computed from the average time per combination so far - Print a final summary with total count, total size, and elapsed time

8. Resumability¶

The script is inherently resumable: if interrupted (Ctrl+C) and restarted, it skips any files that already exist in the output directory. This is handled by the existsSync(outPath) check in generateOne().

9. `--dry-run` mode¶

When --dry-run is passed, the script should:

Enumerate all combinations
Count how many already exist in the output directory (if it exists)
Print a summary without generating anything:

Dry run — no files will be generated.

Range: 100–1000 mm, increment: 5 mm
Total combinations: 16,471
Already cached: 89
Remaining: 16,382
Estimated storage: ~32 GB
CPUs: 16, concurrent jobs: 4, workers/job: 4
Estimated time: ~10–14 hours

Phase 3: Upload Script — `scripts/upload-preview-cache.sh` (30 min)¶

Priority: P0 | Impact: Critical | Dependencies: Phase 2

Create a shell script that uploads generated STL files to the staging server.

1. Script location¶

File: scripts/upload-preview-cache.sh
Must be executable: chmod +x scripts/upload-preview-cache.sh

2. Script behavior¶

The script should:

Accept the local cache directory and the remote server as arguments
Use rsync to transfer only new/changed files (incremental, resumable)
After transfer, fix file ownership inside the Docker container

3. Script implementation¶

#!/usr/bin/env bash
set -euo pipefail

# --- Configuration ---
LOCAL_DIR="${1:-./preview-cache-output}"
REMOTE_HOST="${2:-root@staging-connect-api.forma3d.be}"
REMOTE_CACHE_DIR="/data/gridflock/preview-cache"
CONTAINER_NAME="forma3d-gridflock-service"

# --- Validation ---
if [ ! -d "$LOCAL_DIR" ]; then
  echo "Error: Local directory '$LOCAL_DIR' does not exist."
  echo "Usage: $0 [local-dir] [remote-host]"
  exit 1
fi

FILE_COUNT=$(find "$LOCAL_DIR" -name 'preview-*.stl' | wc -l | tr -d ' ')
TOTAL_SIZE=$(du -sh "$LOCAL_DIR" | cut -f1)

echo "Upload Preview Cache to Staging"
echo "================================"
echo "Source:      $LOCAL_DIR ($FILE_COUNT files, $TOTAL_SIZE)"
echo "Destination: $REMOTE_HOST:$REMOTE_CACHE_DIR"
echo ""
read -p "Continue? [y/N] " -n 1 -r
echo ""

if [[ ! $REPLY =~ ^[Yy]$ ]]; then
  echo "Aborted."
  exit 0
fi

# --- Transfer ---
echo ""
echo "Syncing files with rsync..."
rsync -avz --progress \
  "$LOCAL_DIR/" \
  "$REMOTE_HOST:$REMOTE_CACHE_DIR/"

# --- Fix permissions ---
echo ""
echo "Fixing file ownership inside Docker container..."
ssh "$REMOTE_HOST" "docker exec -u 0 $CONTAINER_NAME chown -R nestjs:nodejs $REMOTE_CACHE_DIR/"

echo ""
echo "Done! $FILE_COUNT files uploaded to $REMOTE_HOST:$REMOTE_CACHE_DIR"

4. Usage¶

# Generate locally first
pnpm tsx scripts/populate-preview-cache.ts --out ./preview-cache-output

# Upload to staging
./scripts/upload-preview-cache.sh ./preview-cache-output root@<staging-ip>

5. Important notes¶

The staging server's gridflock service runs in Docker. The cache directory at /data/gridflock/preview-cache is a Docker volume that is bind-mounted from the host.
After rsync copies files to the host, the chown command inside the container ensures the NestJS process (running as nestjs:nodejs, UID 1001:1001) can read the files.
rsync is incremental — running it again only transfers new or modified files. This makes it safe to run repeatedly (e.g., after generating additional dimension ranges).
If the SSH connection drops mid-transfer, just re-run the script. rsync resumes where it left off.

✅ Validation Checklist¶

Extraction into gridflock-core¶

Generation Script¶

Upload Script¶

scripts/upload-preview-cache.sh exists and is executable
Running the upload script transfers files to the staging server
File ownership is fixed after upload (nestjs:nodejs)
Running the script again only transfers new files (rsync incremental behavior)
After upload, previews load instantly in the storefront configurator for uploaded dimensions

Integration Test¶

Generate a small set locally: --min 100 --max 300 --increment 50
Upload to staging
Verify in the storefront configurator that those dimensions load instantly (no generation delay)
Verify in the admin Preview Cache screen that the files appear in the file listing

Build & Lint¶

pnpm nx run-many -t build --all passes
pnpm nx run-many -t lint --all passes
No any, ts-ignore, or eslint-disable introduced

🚫 Constraints and Rules¶

MUST DO¶

Add optional maxWorkers parameter to generatePlatesParallel() in gridflock-core (backward-compatible)
Extract generatePreviewStl() into @forma3d/gridflock-core so both the service and the script use the exact same function — single source of truth
generatePreviewStl must accept and pass through maxWorkers to generatePlatesParallel
Move computeUniformOffsets, computeOffsetsPerColumn, PLATE_GAP_MM, and DEFAULT_PREVIEW_OPTIONS out of the service class into gridflock-core's preview-generator.ts
Refactor GridflockPreviewService.generatePreview() to be a thin caching wrapper that calls generatePreviewStl()
Normalize the cache key (larger dimension first) in the service
The population script must import generatePreviewStl from gridflock-core — no generation logic in the script itself
The script must auto-detect CPU count and partition cores: concurrency = floor(cpus / 4) concurrent jobs, each with workersPerJob = floor(cpus / concurrency) workers — no thread oversubscription
Sort combinations by grid cell count ascending (smallest first)
Support --dry-run for safe previewing before long generation runs
Make the generation script resumable (skip existing files)
Use rsync for uploads (incremental, resumable)
Fix file ownership after upload (nestjs:nodejs)
Write to a temp file and rename on completion to prevent corrupted partial files on Ctrl+C

MUST NOT¶

Duplicate any STL generation logic in the script (no offset computation, no plate set calculation, no STL combining — all of that stays in gridflock-core)
Add any server-side API endpoints, controllers, or BullMQ queues
Add any frontend/UI changes
Use any, ts-ignore, or eslint-disable
Hardcode SSH credentials, server IPs, or secrets in the scripts
Generate files with non-normalized names (e.g., preview-320x450.stl where w < h)

SHOULD DO (Nice to Have)¶

Write to a temp file then rename() to final path to prevent partial files on interruption
Show estimated total storage in --dry-run mode
Show a final summary with total disk usage after generation completes
Handle SIGINT gracefully: finish current in-progress combinations, then exit cleanly with a summary

📚 Key References¶

Codebase: - Research document: docs/03-architecture/research/stl-cache-prepopulation-research.md (Sections 2, 9) - Preview service (to refactor): apps/gridflock-service/src/gridflock/gridflock-preview.service.ts — the generatePreview() method and offset computation helpers that must be extracted into gridflock-core - Gridflock-core exports: libs/gridflock-core/src/index.ts - Gridflock-core library directory: libs/gridflock-core/src/lib/ — new preview-generator.ts goes here - Defaults: libs/gridflock-core/src/lib/defaults.ts - Printer profiles: libs/gridflock-core/src/lib/printer-profiles.ts - Parallel generator (to modify — add maxWorkers param): libs/gridflock-core/src/lib/parallel-generator.ts - STL combiner: libs/gridflock-core/src/lib/stl-combiner.ts - Existing scripts: scripts/ directory

Combinatorics (from research): - 181 values per axis (100–1000 mm, step 5 mm) - 16,471 unordered pairs (with repetition) - ~32 GB total storage - ~13 KB per grid cell in binary STL - ~15 seconds average generation time per combination (2 vCPU)

END OF PROMPT

This prompt first extracts the preview generation pipeline from the NestJS service into @forma3d/gridflock-core as a reusable generatePreviewStl() function — creating a single source of truth for STL preview generation. Both the server and the offline script call this same function, so changes to generation logic only happen in one place. The generation script (populate-preview-cache.ts) is a thin CLI wrapper that handles enumeration, concurrency, file I/O, and progress. The upload script (upload-preview-cache.sh) transfers the results to the staging server via rsync. No server-side API, BullMQ, or UI changes are involved.