09-15, 13:00–13:30 (Europe/Amsterdam), Qux
Recommendation systems shape personalized experiences across various sectors, but evaluating their effectiveness remains a significant challenge. Drawing on experiences from industry leaders such as Booking.com, this talk introduces a robust, practical approach to A/B testing for assessing the quality of recommendation systems. The talk is designed for data scientists, statisticians, and business professionals, offering real-world insights and industry tricks on setting up A/B tests, interpreting results, and circumventing common pitfalls. While basic familiarity with recommendation systems and A/B testing is beneficial, it's not a prerequisite.
This talk aims to provide attendees with a practical understanding of A/B testing in the evaluation of recommendation systems, including unique insights from industry practices and specific tricks that enhance effectiveness.
My report includes next steps:
- Introduction to recommendation systems, their ubiquity, and the imperative for evaluation, including industry examples.
- An overview of A/B testing and its vital role in assessing recommendation systems, supported by insights from Booking.com and other industry leaders.
- Techniques for designing effective hypotheses for A/B tests, focusing on recommendation systems.
- Choosing pertinent metrics for robust evaluation of recommendation systems with industry examples.
- Conducting A/B tests: industry best practices, common pitfalls, and strategies for mitigation, reinforced by real-world cases.
- Accurate interpretation of A/B testing results and management of statistical biases, with insights from the field.
By the end of the talk, attendees will have a comprehensive understanding of how to apply A/B testing effectively to recommendation systems, select relevant metrics, interpret results accurately, and navigate common challenges, backed by industry best practices and practical examples.
No previous knowledge expected
- Machine Learning Scientist in the Booking.com
- Experienced manager in MLE/DS/SE/DA, I possess extensive expertise in machine learning, analytics, and software engineering. I excel at leading teams to create groundbreaking businesses and delivering innovative solutions for real-world business cases across various industries, including IT, banking, telecommunications, marketplaces, game development, shops, Travel-tech and streaming platforms.
- Expert in building recommendation and ranking systems, as well as personalization automation with machine learning, and advanced A/B testing.
- Co-author and lecturer of a popular online course on recommender system development with over 1000 students.
- Co-author an open-source Python library called RecTools, specifically designed for building recommender systems. The library is hosted on GitHub at RecTools and has received widespread recognition and adoption in the industry.
- Graduate with a Master’s degree in Mathematics and Computer Science and over 6 years of experience in data science.