Osnat Haj Yahia

Reinforcement fine-tuning can teach language models to reason — but the hardest part isn’t the model or the optimizer. It’s the reward function. A reward function looks like engineering: it compiles, it runs, it returns a number. But it behaves like a contract with an adversarial optimizer that will find every loophole you left open.
Through a real-world case study — teaching LLMs to match financial transactions across record-keeping systems — we iterated through three rounds of reward redesign across both OpenAI’s proprietary RFT API and open-weight GRPO training. Each iteration uncovered a new failure mode hiding behind the one we just fixed: the Abstention Trap (asymmetric penalties that make silence optimal), the Partial Credit Cliff (elegant math with catastrophic gradient signal), the Stratum Blind Spot (aggregate reward rising while subcategories collapse), the Entropy Death Spiral (mode collapse invisible behind an API), and the Reasoning Ceiling (performance walls that no reward engineering can break without chain-of-thought).
We distill these failures into a Reward Stress Testing framework: five concrete diagnostic checks practitioners can run before and during any RL fine-tuning job. Your reward function has bugs. This talk shows you how to find them before the optimizer does.