Chatbots are often evaluated based on how well they answer questions, resolve issues, or satisfy users. Much less attention is given to how they fail under intentional misuse.
In this roundtable, we will focus exclusively on breaking chatbots: how real users manipulate, exploit, and misuse conversational AI systems in production environments. Rather than discussing theoretical security threats, the session centers on practical, everyday interactions that cause bots to behave in unintended, misleading, or harmful ways.
We will examine common failure patterns such as prompt exploitation, instruction hijacking, policy evasion, context flooding, repeated probing, and edge case abuse. Many of these behaviors do not look malicious at first glance, yet they can significantly degrade system reliability and trust.
The session will explore how users intentionally test system boundaries, how misuse evolves once a bot is deployed at scale, and why many current evaluation practices fail to capture these failure modes. By shifting the focus from “how to make the bot better” to “how the bot can be broken,” this roundtable aims to help practitioners design more resilient conversational systems.
The discussion is interactive by design and encourages participants to share examples from real systems, unexpected failures, and open challenges around misuse detection and prevention.
Ortal Ashkenazi is a Data Scientist at Wix, specializing in natural language processing and applied machine learning. She enjoys turning messy, human language into structured, useful insights — bridging the gap between raw data and real-world decisions. With an MSc from the Technion, she combines academic depth with hands-on product thinking to build AI tools that people actually use.
Mor Hananovitz is a Senior ML / AI manager at LSports with over six years of experience managing high-performing teams and delivering applied machine learning and Generative AI solutions. She holds both B.Sc. and M.Sc. degrees in Engineering from Ben-Gurion University. Mor teaches in leading AI engineering programs, including HUJI, Reichman University, and Nebius Academy, where she mentors the next generation of AI practitioners. She also leads the Women in Data Science (WiDS) Israel community, actively working to empower and advance women in the data and AI ecosystem.
Keynote session: Hadas Grossmon Ella
Break
Lightning talks session
Roundtable closing
Talk by Hila Paz
Talk by Dr. Moran Mizrahi
Closing remarks
End
