09-14, 14:10–14:40 (Europe/Amsterdam), Qux
How can you evaluate your production models when the data is not structured and you have no labels? To start, by tracking patterns and changes in the input data and model outputs. In this talk, I will give an overview of the possible approaches to monitor NLP and LLM models: from embedding drift detection to using regular expressions.
Once LLMs or NLP models are in production, you want to ensure they work as intended. But how can you observe their behavior in the wild and detect when something goes wrong?
First, you often lack true labels. To add to this, the data is unstructured - how exactly can you track a pile of texts?
Monitoring the patterns in the input data and model outputs is often the first line of defense. In the talk, I will review possible approaches to monitoring drift and data quality issues in text data and explain their pros and cons.
I will cover:
- Statistical embedding drift detection
- Tracking interpretable text descriptors like text length and sentiment
- Using regular expressions to validate outputs
- Explaining drift through model-based drift detection
- Detecting changes in multi-modal data
I will also introduce open-source tools, models, and visualization techniques one can use to monitor LLM and NLP models.
This talk will benefit data scientists and machine learning engineers who work with NLP and LLM in production.
No previous knowledge expected
Emeli Dral is a Co-founder and CTO at Evidently AI, a startup developing open-source tools to evaluate, test, and monitor the performance of machine learning models.
Earlier, she co-founded an industrial AI startup and served as the Chief Data Scientist at Yandex Data Factory. She led over 50 applied ML projects for various industries - from banking to manufacturing. Emeli is a data science lecturer at GSOM SpBU and Harbour.Space University. She is a co-author of the Machine Learning and Data Analysis curriculum at Coursera with over 100,000 students.