AI Prompt: Forma3D.Connect — SimplyPrint Print History Reconciliation¶

Purpose: Instruct an AI to implement print history-based reconciliation for self-healing terminal state detection
Estimated Effort: 8–12 hours (implementation + tests)
Prerequisites: SimplyPrint webhook integration (Phase 2), reconciliation service, ID mismatch fix (fix/seeded-data-cross-process-sharing)
Output: Self-healing reconciliation that detects completed/failed/cancelled jobs via SimplyPrint's print history API, eliminating the need for manual forceStatus intervention
Status: ✅ DONE

🎯 Mission¶

Enhance the SimplyPrint reconciliation service to use SimplyPrint's Print Jobs API (GET /{id}/jobs/Get and GET /{id}/jobs/GetDetails) for detecting terminal job states (completed, failed, cancelled) that were missed due to webhook delivery failures.

Problem being solved:

Currently, if a SimplyPrint webhook (job.done, job.failed, job.cancelled) is missed: - The print job stays stuck in PRINTING (or QUEUED/ASSIGNED) forever - An operator must manually use the forceStatus admin endpoint to resolve the stuck job - Downstream processing (shipping label generation, Shopify fulfillment) never triggers automatically

The existing reconciliation service can only detect: - Jobs still in the queue → verifies QUEUED status - Jobs actively on a printer → detects PRINTING status

It cannot detect terminal states because completed/failed/cancelled jobs disappear from both the queue and the active printer list. The reconciliation logs "not found in SimplyPrint — may have completed/failed" and does nothing:

// Current behavior in simplyprint-reconciliation.service.ts
if (!mappedStatus && this.isActiveStatus(job.status)) {
  this.logger.debug(
    `Job ${job.id} (SP: ${job.simplyPrintJobId}) not found in SimplyPrint - may have completed/failed`
  );
  // ← Does nothing. Relies on webhooks for terminal states.
  continue;
}

Solution: Use SimplyPrint's print history API to look up the actual status of "missing" jobs and emit the appropriate status change events so the automation pipeline continues.

Deliverables:

• getJobHistory() method on SimplyPrintApiClient that calls GET /{id}/jobs/Get
• Updated reconciliation service that queries print history for jobs missing from queue/printers
• Terminal state detection (COMPLETED, FAILED, CANCELLED) via history lookup
• Automatic status change event emission so downstream processing resumes
• Comprehensive tests covering the new reconciliation path
• The forceStatus endpoint remains available as an ultimate fallback but should rarely be needed

Critical constraints:

• Respect SimplyPrint API rate limits (add delays between calls as done elsewhere in the client)
• Handle the queue-item ID → job UID mismatch (the stored simplyPrintJobId may be the old queue-item created_id or the updated job uid depending on whether job.started webhook was received)
• Don't auto-transition to terminal states too aggressively — add a grace period (e.g., job must be "missing" for >5 minutes before querying history)
• Log all history-based reconciliation actions clearly for auditability

📌 Context (Current State)¶

What Exists¶

SimplyPrint API Client (apps/api/src/simplyprint/simplyprint-api.client.ts): - getQueue() — lists items in the print queue - getPrinters() — lists printers with current job info - getJob(jobId) — gets a single job via GET /{id}/jobs/GetDetails?id={job_uid} with fallback to queue/printer lookup - addToQueue(), cancelJob(), createJob() — job management - Rate-limit-aware with 500ms delays between recursive calls

Reconciliation Service (apps/api/src/simplyprint/simplyprint-reconciliation.service.ts): - Runs every minute via @Cron(CronExpression.EVERY_MINUTE) - Queries all active print jobs (QUEUED, ASSIGNED, PRINTING) with a simplyPrintJobId (includes printerName for backfill logic) - Builds a status map from queue items and active printers (including queue-item ID via getJob() for printers) - Emits SIMPLYPRINT_EVENTS.JOB_STATUS_CHANGED for status mismatches and to backfill printer name when a job is already PRINTING but has no printerName (so "Unassigned" is filled on the next run) - Indexes by job uid, numeric currentJobNumericId, and queue-item ID from job details

ID Mismatch Fix (recently applied): - When a job is added to the queue, created_id (integer, e.g., 384847) is stored as simplyPrintJobId - When SimplyPrint starts the job, webhooks use a different uid (UUID) - The webhook handler now does a fallback lookup by numeric ID and updates the stored ID to the uid - The reconciliation service indexes printers by both uid and numeric ID - Important: If the job.started webhook was also missed, the stored ID is still the queue-item created_id, not the job uid. The history lookup must handle both cases.

Print Jobs Service (apps/api/src/print-jobs/print-jobs.service.ts): - handleSimplyPrintStatusChange() — event handler for status changes (lookup by uid, numeric ID, or queue-item ID via API) - updateJobStatus() — updates DB, logs event, emits downstream events; when status is unchanged, still applies printerId / printerName / errorMessage if provided (backfill so reconciliation can fill "Unassigned" printer name) - forceStatus() — admin endpoint for manual terminal state forcing

Shared Types (libs/api-client/src/simplyprint/simplyprint.types.ts): - SimplyPrintJob interface with id, uid, status, printerId, printerName, etc. - SimplyPrintJobStatus enum: QUEUED, ASSIGNED, PREPARING, PRINTING, COMPLETED, FAILED, CANCELLED

SimplyPrint API Endpoints Available¶

From the SimplyPrint API documentation (https://apidocs.simplyprint.io/):

1. GET /{id}/jobs/Get — List print jobs (print history)
2. Returns paginated list of historical print jobs
3. Likely supports filtering by status, date range, printer

4. This is the endpoint that powers the "Print history" page in SimplyPrint's UI

5. GET /{id}/jobs/GetDetails?id={job_uid} — Get details for a specific job

6. Already partially implemented in getJob() method
7. Returns job state, printer info, timestamps
8. API returns states: ongoing, printing, done, finished, cancelled, failed, queued

Recent changes (webhook refactoring and printer backfill)¶

• Webhook validation: Global ValidationPipe uses forbidNonWhitelisted: false; webhook DTO allows optional panel_url, optional data, and numeric coercion so SimplyPrint payloads are accepted. On 400, request body and validation details are logged for debugging.
• Idempotency: SimplyPrint webhook idempotency key is now composite: {webhook_id}:{event}:{job.uid} (or timestamp if no job), so job.started and job.done for the same job no longer collide and the second is no longer skipped.
• Printer name backfill: If a job is already PRINTING but has no printer name (e.g. webhook didn’t include printer), reconciliation emits an event with the same status and printer info; updateJobStatus() applies printer/error backfill when status is unchanged, so the next run fills "Unassigned" with the actual printer name.

What's Missing¶

1. getJobHistory() method — No method to list/filter print jobs from history
2. History-based reconciliation — The reconciliation service doesn't query history for missing jobs (terminal states still rely on webhooks or manual forceStatus)
3. Grace period logic — No mechanism to avoid premature terminal-state transitions
4. Robust history matching — Need to handle both queue-item IDs and job UIDs when looking up history

🛠️ Tech Stack Reference¶

Same as existing stack — no new dependencies required:

• Backend: NestJS (TypeScript), Prisma, PostgreSQL
• Testing: Jest
• SimplyPrint API: Axios-based HTTP client

🏗️ Architecture Requirements¶

1) Print History API Integration¶

Add a getJobHistory() method to SimplyPrintApiClient that calls the SimplyPrint print jobs list endpoint. Based on the API documentation pattern and existing endpoints:

GET /{companyId}/jobs/Get

Expected parameters (to be confirmed against actual API response): - Pagination: page, page_size - Filters: status, date range, printer ID

Expected response (based on patterns from other SimplyPrint endpoints): - Top-level data array or jobs array containing job objects - Each job: id (integer), uid (string/UUID), state (string), printer info, timestamps

2) Reconciliation Enhancement Strategy¶

The reconciliation flow should be enhanced as follows:

For each active job not found in queue/printers:
  1. Check if job has been "missing" for long enough (grace period)
  2. Try GetDetails by stored simplyPrintJobId (could be uid or numeric ID)
  3. If GetDetails fails, try history list endpoint with recent time filter
  4. If terminal state found → emit status change event
  5. If still not found after extended period → log warning for operator attention

3) Grace Period¶

To avoid false positives (e.g., job briefly between queue and printer assignment):

• Track when a job was first seen as "missing" (add lastSeenInSimplyPrintAt or use a transient in-memory map)
• Only query history API if job has been missing for >5 minutes
• Only escalate to operator attention if job has been missing for >30 minutes and history lookup also fails

4) Rate Limit Awareness¶

• SimplyPrint API has rate limits
• Add 500ms delay between history API calls (same pattern as getFilesInFolder)
• Limit history lookups to a reasonable batch size per reconciliation cycle (e.g., max 10 jobs)
• Use GetDetails (single job) before falling back to Get (list) to minimize API calls

📁 Files to Create/Modify¶

Backend (NestJS)¶

apps/api/src/simplyprint/
├── simplyprint-api.client.ts              # UPDATE: add getJobHistory(), enhance getJob()
├── simplyprint-reconciliation.service.ts  # UPDATE: add history-based terminal state detection
└── __tests__/
    ├── simplyprint-api.client.spec.ts           # UPDATE: tests for getJobHistory()
    └── simplyprint-reconciliation.service.spec.ts  # UPDATE: tests for history reconciliation

libs/api-client/src/simplyprint/
└── simplyprint.types.ts                   # UPDATE: add history-related types if needed

🔧 Implementation Details¶

1) SimplyPrint API Client — getJobHistory()¶

Add to simplyprint-api.client.ts:

/**
 * Parameters for querying print job history.
 */
interface GetJobHistoryParams {
  page?: number;
  pageSize?: number;
}

/**
 * Get print job history (completed/failed/cancelled jobs).
 *
 * Endpoint: GET /{id}/jobs/Get
 * Docs: https://apidocs.simplyprint.io/#get-print-jobs
 *
 * Returns historical jobs — use this to detect terminal states
 * for jobs that are no longer in the queue or on a printer.
 */
async getJobHistory(params?: GetJobHistoryParams): Promise<SimplyPrintJob[]> {
  this.ensureEnabled();

  const requestParams: Record<string, unknown> = {
    page: params?.page ?? 1,
    page_size: params?.pageSize ?? 50,
  };

  const response = await this.request<Record<string, unknown>>(
    'GET',
    `/${this.config.companyId}/jobs/Get`,
    requestParams
  );

  // Parse the response — structure TBD based on actual API response.
  // SimplyPrint typically puts data at the top level.
  const raw = response as unknown as Record<string, unknown>;
  const rawJobs = (raw['data'] ?? raw['jobs'] ?? []) as Array<Record<string, unknown>>;

  return rawJobs.map((j) => this.mapRawJob(j));
}

/**
 * Map a raw job object from the API to our SimplyPrintJob type.
 * Centralised so both getJob() and getJobHistory() use the same mapping.
 */
private mapRawJob(j: Record<string, unknown>): SimplyPrintJob {
  const printer = j['printer'] as Record<string, unknown> | undefined;
  const state = String(j['state'] ?? j['status'] ?? '');

  return {
    id: String(j['id'] ?? ''),
    uid: String(j['uid'] ?? j['id'] ?? ''),
    fileId: String(j['filesystem_id'] ?? j['file_id'] ?? ''),
    fileName: String(j['file'] ?? j['filename'] ?? ''),
    printerId: printer ? String(printer['id'] ?? '') : undefined,
    printerName: printer ? String(printer['name'] ?? '') : undefined,
    status: this.mapJobState(state),
    progress: j['percentage'] as number | undefined,
    startedAt: j['started'] ? String(j['started']) : undefined,
    completedAt: j['ended'] ? String(j['ended']) : undefined,
  };
}

Important notes: - The exact response shape of GET /{id}/jobs/Get needs to be confirmed by making a test API call. The implementation above handles both data and jobs top-level keys. - Reuse the existing mapJobState() method for status mapping. - Add a mapRawJob() helper to centralise job mapping (extract from existing getJob() code).

2) Enhance getJob() for Numeric ID Fallback¶

The existing getJob() method uses GetDetails?id={jobId}. If the stored ID is a queue-item created_id (integer), GetDetails might not find it by that ID. Enhance to also try the numeric ID:

async getJob(jobId: string): Promise<SimplyPrintJob | null> {
  this.ensureEnabled();

  // Try GetDetails endpoint first (works with uid)
  try {
    const response = await this.request<Record<string, unknown>>(
      'GET',
      `/${this.config.companyId}/jobs/GetDetails`,
      { id: jobId }
    );

    const raw = response as unknown as Record<string, unknown>;
    const job = raw['job'] as Record<string, unknown> | undefined;

    if (job) {
      return this.mapRawJob(job);
    }
  } catch {
    this.logger.debug(`GetDetails failed for job ${jobId}, trying fallback`);
  }

  // Fallback: check queue (queue item IDs match our stored created_id)
  const queue = await this.getQueue();
  const queueItem = queue.find((item) => item.id === jobId);
  if (queueItem) {
    return {
      id: queueItem.id,
      uid: queueItem.id,
      fileId: queueItem.fileId,
      fileName: queueItem.fileName,
      status: SimplyPrintJobStatus.QUEUED,
    };
  }

  // Fallback: check printers (both uid and numeric ID)
  const printers = await this.getPrinters();
  for (const printer of printers) {
    if (
      printer.currentJobId === jobId ||
      (printer.currentJobNumericId != null && String(printer.currentJobNumericId) === jobId)
    ) {
      return {
        id: jobId,
        uid: printer.currentJobId ?? jobId,
        fileId: '',
        fileName: '',
        printerId: printer.id,
        printerName: printer.name,
        status: SimplyPrintJobStatus.PRINTING,
      };
    }
  }

  return null;
}

3) Reconciliation Service — History-Based Terminal State Detection¶

Update simplyprint-reconciliation.service.ts:

// Add an in-memory map to track when jobs were first seen as "missing"
private readonly missingJobTimestamps = new Map<string, Date>();
private static readonly GRACE_PERIOD_MS = 5 * 60 * 1000;      // 5 minutes
private static readonly ESCALATION_PERIOD_MS = 30 * 60 * 1000; // 30 minutes
private static readonly MAX_HISTORY_LOOKUPS_PER_CYCLE = 10;

// In the reconciliation loop, replace the "not found" block:

// OLD: just log and continue
// NEW: query history API for terminal state

if (!mappedStatus && this.isActiveStatus(job.status)) {
  await this.handleMissingJob(job, result);
  continue;
}

// New method:
private async handleMissingJob(
  job: ActivePrintJob,
  result: ReconciliationResult
): Promise<void> {
  const now = new Date();

  // Track when we first noticed this job was missing
  if (!this.missingJobTimestamps.has(job.id)) {
    this.missingJobTimestamps.set(job.id, now);
    this.logger.debug(
      `Job ${job.id} (SP: ${job.simplyPrintJobId}) first seen missing from SimplyPrint`
    );
    return; // Give it time before checking history
  }

  const firstMissingAt = this.missingJobTimestamps.get(job.id)!;
  const missingDurationMs = now.getTime() - firstMissingAt.getTime();

  // Grace period: don't check history too early (job may be transitioning)
  if (missingDurationMs < SimplyPrintReconciliationService.GRACE_PERIOD_MS) {
    this.logger.debug(
      `Job ${job.id} missing for ${Math.round(missingDurationMs / 1000)}s, ` +
      `waiting for grace period (${SimplyPrintReconciliationService.GRACE_PERIOD_MS / 1000}s)`
    );
    return;
  }

  // Rate limit: only check a limited number of jobs per cycle
  if (this.historyLookupsThisCycle >= SimplyPrintReconciliationService.MAX_HISTORY_LOOKUPS_PER_CYCLE) {
    this.logger.debug(`History lookup limit reached, deferring job ${job.id} to next cycle`);
    return;
  }
  this.historyLookupsThisCycle++;

  // Query SimplyPrint for the job's actual status
  try {
    const simplyPrintJob = await this.simplyPrintClient.getJob(job.simplyPrintJobId);

    if (simplyPrintJob) {
      const resolvedStatus = this.mapSimplyPrintStatus(simplyPrintJob.status);

      if (this.isTerminalStatus(resolvedStatus)) {
        this.logger.log(
          `History reconciliation: Job ${job.id} (SP: ${job.simplyPrintJobId}) ` +
          `detected as ${resolvedStatus} via print history API`
        );

        const statusEvent: SimplyPrintJobStatusChangedEvent = {
          simplyPrintJobId: job.simplyPrintJobId,
          previousStatus: job.status,
          newStatus: resolvedStatus,
          printerId: simplyPrintJob.printerId,
          printerName: simplyPrintJob.printerName,
          timestamp: now,
        };

        this.eventEmitter.emit(SIMPLYPRINT_EVENTS.JOB_STATUS_CHANGED, statusEvent);
        result.jobsUpdated++;

        // Clear from missing map — it's resolved
        this.missingJobTimestamps.delete(job.id);

        await this.eventLogService.log({
          eventType: 'system.simplyprint_history_reconciliation',
          severity: 'INFO',
          message: `Print job ${job.id} status resolved via history: ${job.status} → ${resolvedStatus}`,
          metadata: {
            printJobId: job.id,
            simplyPrintJobId: job.simplyPrintJobId,
            previousStatus: job.status,
            newStatus: resolvedStatus,
            missingDurationMs,
            source: 'history_api',
          },
        });

        return;
      }
    }

    // Job not found in history either — escalate if missing too long
    if (missingDurationMs > SimplyPrintReconciliationService.ESCALATION_PERIOD_MS) {
      this.logger.warn(
        `Job ${job.id} (SP: ${job.simplyPrintJobId}) has been missing from SimplyPrint ` +
        `for ${Math.round(missingDurationMs / 60000)} minutes. May require manual intervention.`
      );

      await this.eventLogService.log({
        eventType: 'system.simplyprint_job_missing',
        severity: 'WARNING',
        message: `Print job ${job.id} not found in SimplyPrint for ${Math.round(missingDurationMs / 60000)} minutes`,
        metadata: {
          printJobId: job.id,
          simplyPrintJobId: job.simplyPrintJobId,
          currentStatus: job.status,
          missingDurationMs,
        },
      });
    }
  } catch (error) {
    this.logger.error(
      `Failed to look up job ${job.id} in SimplyPrint history: ` +
      `${error instanceof Error ? error.message : 'Unknown error'}`
    );
  }
}

// Helper
private isTerminalStatus(status: PrintJobStatus): boolean {
  return [
    PrintJobStatus.COMPLETED,
    PrintJobStatus.FAILED,
    PrintJobStatus.CANCELLED,
  ].includes(status);
}

Important: Add private historyLookupsThisCycle = 0; and reset it to 0 at the start of each runReconciliation() call.

4) Cleanup of Missing Job Timestamps¶

Add cleanup logic to prevent memory leaks:

// In runReconciliation(), after processing all jobs:

// Clean up missingJobTimestamps for jobs that are no longer active
const activeJobIds = new Set(activeJobs.map((j) => j.id));
for (const jobId of this.missingJobTimestamps.keys()) {
  if (!activeJobIds.has(jobId)) {
    this.missingJobTimestamps.delete(jobId);
  }
}

🧪 Testing Requirements¶

Unit Tests¶

SimplyPrint API Client:

Reconciliation Service:

Integration Scenario (Manual Verification)¶

1. Create an order → print job created with simplyPrintJobId
2. SimplyPrint starts printing (webhook may or may not arrive)
3. SimplyPrint completes the print (webhook intentionally blocked/missed)
4. Wait for reconciliation cycle (1 minute)
5. After grace period (5 minutes), reconciliation queries history
6. Job status updates to COMPLETED
7. Downstream events fire (order completion, shipping label, fulfillment)

✅ Validation Checklist¶

Infrastructure¶

• pnpm nx build api succeeds
• pnpm lint passes on modified files
• No TypeScript errors
• All existing tests still pass

Reconciliation Enhancement¶

• getJobHistory() method implemented on SimplyPrintApiClient
• getJob() enhanced with numeric ID fallback for printers
• mapRawJob() helper extracted for consistent job mapping
• Reconciliation service queries history for "missing" jobs
• Grace period prevents premature terminal state transitions
• Rate limiting prevents excessive API calls (max 10 history lookups per cycle)
• Escalation logging after 30 minutes of job being missing
• Missing job timestamp map cleaned up to prevent memory leaks
• Event log entries created for history-based reconciliation actions
• SIMPLYPRINT_EVENTS.JOB_STATUS_CHANGED events emitted correctly

Testing¶

• API client tests for getJobHistory()
• API client tests for enhanced getJob() fallbacks
• Reconciliation tests for grace period behavior
• Reconciliation tests for history-based terminal state detection
• Reconciliation tests for rate limiting
• Reconciliation tests for escalation logging
• Reconciliation tests for missing map cleanup

🎬 Execution Order¶

Phase 1: API Client Enhancement (3 hours)¶

1. Add getJobHistory() method to SimplyPrintApiClient
2. Extract mapRawJob() helper from existing code
3. Enhance getJob() with numeric ID printer fallback
4. Add types if needed to simplyprint.types.ts
5. Write unit tests for new/modified API client methods
6. Verify actual API response shape by making a test call to GET /{companyId}/jobs/Get and adjust mapping accordingly

Phase 2: Reconciliation Enhancement (4 hours)¶

1. Add missingJobTimestamps map and grace period constants
2. Add historyLookupsThisCycle counter
3. Extract handleMissingJob() method
4. Add isTerminalStatus() helper
5. Implement history lookup with grace period and rate limiting
6. Add event logging for history-based reconciliation
7. Add escalation logging for long-missing jobs
8. Add cleanup logic for missing job timestamps map
9. Write comprehensive tests for all new reconciliation behavior

Phase 3: Validation & Documentation (1–2 hours)¶

1. Run full test suite
2. Verify no linter errors
3. Test against staging environment with a real SimplyPrint account
4. Confirm the actual GET /jobs/Get response shape matches the implementation
5. Update architecture documentation if needed (sequence diagram C4_Seq_04_PrintJobSync.puml)

📊 Expected Output¶

Verification Commands¶

# Build
pnpm nx build api

# Lint
pnpm nx lint api

# Run affected tests
pnpm nx test api --testPathPattern="simplyprint-api.client.spec|simplyprint-reconciliation.service.spec"

# Run all tests
pnpm nx test api

Success Criteria¶

• A print job whose job.done webhook was missed is automatically detected as COMPLETED within ~6 minutes (1 min reconciliation cycle + 5 min grace period)
• The downstream automation pipeline (shipping label, Shopify fulfillment) triggers automatically after history-based reconciliation
• The forceStatus endpoint is no longer needed for routine operations (only for edge cases)
• No false positives: jobs that are simply transitioning between states are not prematurely marked as terminal
• API rate limits are respected: max 10 history lookups per reconciliation cycle with 500ms delays

Observability¶

After implementation, the following should be visible in logs/event log:

# History-based reconciliation resolving a missed webhook:
[INFO] History reconciliation: Job pj-123 (SP: abc-uid) detected as COMPLETED via print history API

# Grace period in action:
[DEBUG] Job pj-456 missing for 120s, waiting for grace period (300s)

# Escalation for truly stuck jobs:
[WARN] Job pj-789 (SP: 384847) has been missing from SimplyPrint for 35 minutes. May require manual intervention.

🚫 Constraints and Rules¶

MUST DO¶

• Respect SimplyPrint API rate limits (500ms delay between calls, max 10 history lookups per cycle)
• Handle both queue-item created_id (integer) and job uid (UUID) as stored simplyPrintJobId
• Add grace period before querying history (avoid false positives during state transitions)
• Log all history-based reconciliation actions to the event log for auditability
• Clean up the in-memory missing-job timestamps map to prevent memory leaks
• Verify actual API response shape with a real API call before finalising the mapping code

MUST NOT¶

• Query the history API for every "missing" job on every cycle (use grace period + rate limit)
• Auto-transition to terminal states without the grace period
• Assume the GET /jobs/Get response shape — confirm it empirically
• Remove the forceStatus endpoint (it remains as an ultimate fallback)
• Use any or ts-ignore to bypass type checking
• Break existing reconciliation behavior for queue and printer status detection

END OF PROMPT

This prompt implements self-healing terminal state detection for SimplyPrint print jobs. The enhancement ensures that missed webhooks for job completion, failure, or cancellation are automatically detected via SimplyPrint's print history API, eliminating the need for manual operator intervention in routine operations.