Sheli Kohan

AI systems in production evolve continuously. New jailbreaks appear, misuse patterns shift, and guardrails drift.
Static datasets and one-time fine-tuning cannot keep pace.
In this talk, I present Vector, a production-grade agentic teacher–student distillation framework that converts real-world failures into structured learning signals for adaptive AI safety.
Vector operates as a closed feedback loop. A Teacher model generates adversarial and policy-aligned examples based on observed gaps. A deployment-optimized Student model retrains on evolving behavior distributions. An orchestration layer detects structured failure clusters, identifies weak decision boundaries, and triggers targeted data generation, rebalancing, and retraining.
Distillation becomes a control system rather than a one-time compression step.
All data generation and retraining cycles are traceable and governed. Trade-offs such as recall versus precision, coverage versus overfitting, and synthetic expansion versus drift are explicitly managed.
The key insight is that AI safety at scale is not only a modeling problem. It is a feedback systems problem.
This session provides a practical blueprint for building traceable, adaptive guardrails for production AI, with lessons applicable to personalization, domain adaptation, and continuously evolving systems.