Head of Quality & Reliability

Responsibilities:

• Own the evals platform for non-deterministic systems: golden datasets with expert-supplied ground truth, regression evals that gate every prompt, model, rule, and agent change, and drift detection.
• Design fail-explicit systems: every module is independently verifiable, and every failure stops the transaction and surfaces immediately.
• Build pre-production environments with end-to-end simulation, plus shadow, canary, and per-customer rollouts with fast rollback of every versioned component.
• Set SLOs and error budgets for correctness as well as uptime; own actionable alerting, on-call, and incident management.
• Define auto-lockdown thresholds that stop output, contain downstream impact, and escalate when quality breaches.
• Reconcile what we send against what customer systems accept, and close the loop on downstream outcomes such as rejections and reversals.
• Put validation and timing checks on data pipelines at ingestion, transformation, and output.
• Design statistically sound sampling for human validation, including with scarce or noisy labels.
• Build automated checks and hard blocks for sensitive-data exposure and retention violations.
• Make sure every new capability defines its failure modes before launch, and every escape, incident, and customer ticket becomes a permanent check.
• Drive developer productivity: fast CI, flakiness management, ephemeral environments, and guardrails that make coding agents safe at scale.
• Hire and grow a small senior platform and eval team, set quality standards, and communicate quality risk to product, ops, and leadership in business terms.

Requirements:

• 10+ years in software engineering, SRE/platform, data/ML quality, or quality engineering, including 3+ years leading teams or programs.
• Strong software engineering foundation, with the credibility to review architecture and code alongside senior engineers.
• Has built quality, evaluation, or reliability infrastructure for complex, high-stakes systems; ML/LLM evals, fraud, or risk models, data pipelines, payments, or comparable; and taken it from low to high maturity.
• Hands-on with release gating, CI/CD, progressive delivery, and rollback.
• Has owned SLOs, observability, on-call, incident management, and RCA in production.
• Statistical rigor: comfortable with sampling, confidence intervals, and telling a real regression from noise.
• Experience handling sensitive or regulated data (health, financial, or similar).