Lecture
EPP meta-measure and rethinking machine learning benchmarks: A recipe for meta-learning success?
- Date
- Wednesday 8 February 2023
- Time
- Address
-
Snellius
Niels Bohrweg 1
2333 CA Leiden - Room
- 402
Abstract: The most commonly used model performance measure, such as AUC, accuracy, or RMSE, returns a numerical assessment of how well the predictions of the selected model satisfy specific properties e.g., correctly assigning the probability of belonging to the chosen class. From an application point of view, however, we lack information on the probability that a given model gets a better performance model than another and whether the differences we observe between models are statistically significant.
In my talk, I will present a new meta-measure of model performance called EPP (Elo-based Predictive Power). It is inspired by the Elo ranking used in chess and other sports games. By comparing the EPP rankings of two players and transforming them accordingly, we obtain information on the probability that one player is better. EPP adapts this property to the specific conditions of benchmarks in machine learning but allows for universal application in many benchmarking schemes.
A natural question is how to leverage the advantages of EPP in benchmarks for automated machine learning and meta-learning. In this presentation, I will outline potential applications but also the challenges of EPP and its use in meta-learning.
About Saber Salehkaleybar
Saber Salehkaleybar is a Scientific Collaborator in the School of Computer and Communication Sciences (IC) and College of Management Technology (CDM) at Ecole Polytechnique Federale de Lausanne (EPFL). He received B.Sc., M.Sc. and Ph.D. degrees in Electrical Engineering from Sharif University of Technology (SUT), in 2009, 2011, and 2015, respectively. He spent a year as a postdoctoral researcher in Coordinated Science Laboratory at University of Illinois at Urbana-Champaign in 2016-2017. Since 2017, he has been an Assistant Professor of Electrical Engineering at SUT. He is the recipient of INSF starting grant, junior faculty career award, and EPFL IC school fellowship award. His research interests include causal inference, reinforcement learning, and distributed learning.