Course Description
Replicability is vital to ensuring scientific conclusions are reliable, but failures of replicability have been a major issue in nearly all scientific areas of study, and machine learning is no exception. In this course, we will study replicability as a property of learning and other statistical algorithms, developing a theory of replicable learning. We will cover recent formalizations of replicability and their relationships to other common stability notions such as differential privacy and adaptive generalization. We will survey replicable algorithms for fundamental learning tasks, and discuss the limitations of replicable algorithms. If time permits, we will discuss replicability in other settings, such as reinforcement learning and clustering, or other useful and related stability notions such as list replicability and global stability.
Lecture Notes
- Lecture 1: Introduction, Markov, Chebyshev [notes]
- Lecture 2: Hoeffding [notes]
- Lecture 3: Statistical queries, PAC learning for finite classes [notes]
- Lecture 4: More SQ algorithms [notes]
- Lecture 5: Replicable SQs [notes]
- Lecture 6/7: Replicable SQ lower-bound [notes]
- Lecture 8: Replicable SQ lower-bound, adaptive statistical queries [notes]
Project Ideas
Resources
- Adaptive Data Analysis course notes. https://adaptivedataanalysis.com/about/
- The Algorithmic Foundations of Differential Privacy. https://www.cis.upenn.edu/~aaroth/Papers/privacybook.pdf
- Reproducibility in Learning. https://arxiv.org/abs/2201.08430
- Stability is Stable: Connections between Replicability, Privacy, and Adaptive Generalization. https://arxiv.org/abs/2303.12921
- Replicability in High Dimensional Statistics. https://arxiv.org/abs/2406.02628
- Generalization in Adaptive Data Analysis and Holdout Reuse. https://arxiv.org/abs/1506.02629
- Max-Information, Differential Privacy, and Post-Selection Hypothesis Testing. https://arxiv.org/abs/1604.03924
- User-Level Privacy via Correlated Sampling. https://arxiv.org/abs/2110.11208
- Replicability in reinforcement learning. https://arxiv.org/abs/2305.19562
- Replicable Reinforcement Learning. https://arxiv.org/abs/2305.15284
- Replicable Clustering. https://arxiv.org/abs/2302.10359
- Replicability and Stability in Learning. https://arxiv.org/abs/2304.03757