This talk will discuss the advances in Performance Estimation for classification models without access to the ground truth data. Once you've deployed your model to production, you want to ensure it performs well. However, this is often hard to do if you predict an event far in the future or automate certain tasks.

We will introduce three algorithms: Confidence-Based Performance Estimation (CBPE), Importance Weighting (IW), and Multicalibrated Confidence-Based Performance Estimation (M-CBPE). We show that all three give better performance estimates than the test set performance.

We will explain how they work, the intuition behind the algorithms, and their strengths and weaknesses. We will also showcase typical scenarios where Performance Estimation is vital and how to use it in ML Monitoring and Root Cause Analysis.

Timestamps:
0:00 - Introduction
0:33 - Paper results overview
4:02 - Agenda
4:50 - Why is performance estimation vital?
6:21 - How is performance estimation possible?
13:15 - Uncertainty to Performance metric
32:20 - Importance weighting
39:25 - M-CBPE