CaseDesk & caseAssist: Closed-Loop AI Queue Governance and 26-Gate Pre-Send Verification

Architecting a privacy-preserving, human-in-the-loop AI support workflow across 11 major iterations, featuring default PII redaction, provenance tagging, and a 26-gate pre-send verification checklist.

Executive Summary & Context

Tier 3 enterprise technical support across modern AI SaaS platforms requires navigating high-stakes constraints simultaneously: diagnosing multi-system billing ledgers (Stripe Sigma SQL, Metronome) and identity federation (SAML 2.0 / SSO, SCIM, JIT), enforcing strict contractual policy matrices, and operating inside restricted client tool boundaries where automated scraping of source systems is prohibited.

Deploying raw, unconstrained Large Language Models into this workflow introduces unacceptable operational risks: hallucinated policy exceptions, unverified root-cause claims, and customer PII leakage.

Built and iterated across 11 major versions (v1 to v11.7.4) while supporting an enterprise AI software client queue at Boldr, CaseDesk and its companion governance framework caseAssist transform AI assistance from an unconstrained chatbot into a deterministic, audit-traced support pipeline with 100% human-in-the-loop sign-off.

Architecture & Technical Execution

                   [Inbound Tier 3 Escalation Telemetry]
                                    |
                                    v
+-----------------------------------------------------------------------+
| Stage 1: Ingestion & Default PII Redaction (pii-scrub)                |
| - Strips account UUIDs, email handles, and payment card tokens        |
| - Zero reversible mapping keys generated in session context           |
+-----------------------------------------------------------------------+
                                    |
                                    v
+-----------------------------------------------------------------------+
| Stage 2: Modular Skill Routing & Epistemic Provenance Tagging         |
| - Skills: case-intake, case-record, kb-applicability, case-artifacts  |
| - Claims tagged: [Direct] | [Deduced] | [Unverified] | [Theory]       |
| - Cross-checks Stripe Sigma SQL, Metronome ledgers & SAML/IdP state   |
+-----------------------------------------------------------------------+
                                    |
                                    v
+-----------------------------------------------------------------------+
| Stage 3: The 26-Gate Pre-Send Verification Engine                     |
| - Validates factual assertions against authoritative systems of record|
| - Enforces refund/credit approval matrix & escalation thresholds      |
| - Drafting-Only Boundary: Populates structured templates, never sends |
+-----------------------------------------------------------------------+
                                    |
                                    v
             [Human-in-the-Loop Second-Eye Review & Dispatch]
                                    |
                                    v
+-----------------------------------------------------------------------+
| Stage 4: Closed-Loop Debrief & QA Rule Folding (case-debrief)         |
| - Every instruction revision traced to a logged failure mode          |
| - Failed QA gates folded into versioned, sha256-verified releases     |
+-----------------------------------------------------------------------+

Key Engineering & Governance Decisions

Privacy-First by Construction (pii-scrub): Before any case context enters the reasoning workflow, account UUIDs, customer email handles, and billing tokens are scrubbed without generating a reversible mapping key in the session context—ensuring full compliance with client access boundaries.
Epistemic Provenance Tagging: Customer assertions and diagnostic hypotheses are never blended into stated fact. Every statement in the internal case record is tagged as Direct (verified in Stripe Sigma, Metronome, Forge, or Intercom), Deduced, Unverified, or Theory. Missing telemetry is surfaced explicitly as a diagnostic gap rather than smoothed over by generative prose.
Isolated Sandbox vs. Production Architecture: Skill modules (case-intake, case-record, kb-applicability, knowledge-candidate, case-artifacts, pii-scrub, case-debrief) are tested inside an isolated sandbox project before promotion to the production assistant.
Closed-Loop QA Rule Folding: Rather than endlessly bloating system prompts, QA review findings and workflow misses are logged in a formal Debrief Log (39 recorded architectural and privacy rulings). Failed gates are converted into numbered live rules (capped at 25 active rules) and folded into specific skill definitions across 9 sha256-verified releases.

Measurable Impact & Results

90%+ QA Compliance Pass Rate: Sustained high accuracy and policy compliance across complex Tier 3 billing, usage ledger, and SSO/SCIM escalations.
Zero PII Exposure Incidents: Enforced default redaction prior to case intake across all 11 major iterations (v1 to v11.7.4).
Traceable Change Management: Shipped 9 cryptographically verified (sha256) releases backed by 39 documented architectural decisions, with every rule change tied directly to a logged failure mode.