Calendar

Links to Slides, Relevant Chapters in [WM22] and Extra Reading, and Presenters

Lecture 1

Introduction to Low-Dimensional Models

John Wright

• Chapters 2-6, WM22

Lecture 2

Learning Unrolled Networks for Sparse Recovery

Atlas Wang

• J. Liu, X. Chen, Z. Wang, W. Yin, and H. Cai. “Towards Constituting Mathematical Structures for Learning to Optimize.” ICML 2023
• (α-β) T. Chen, X. Chen, W. Chen, H. Heaton, J. Liu, and Z. Wang, W. Yin, “Learning to Optimize: A Primer and A Benchmark”, Journal of Machine Learning Research (JMLR), 2022
• X. Chen, J. Liu, Z. Wang, and W. Yin. “Hyperparameter Tuning is All You Need for LISTA.” NeurIPS 2021
• J. Liu, X. Chen, Z. Wang, and W. Yin. “ALISTA: Analytic weights are as good as learned weights in LISTA.” ICLR 2019
• X. Chen, J. Liu, Z. Wang, and W. Yin. “Theoretical Linear Convergence of Unfolded ISTA and its Practical Weights and Thresholds.” NeurIPS 2018.

Lecture 3

Design Deep Networks for Pursuing Low-Dimensional Structures

Yi Ma

• Chapter 16, WM22
• ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reduction
• CTRL: Closed-Loop Transcription to an LDR via Minimaxing Rate Reduction

Lecture 4

Learning Low-Dimensional Models via Nonconvex Optimization

Yuqian Zhang

• Chapters 7, 9, WM22
• Survey: “From Symmetry to Geometry: Tractable Nonconvex Problems”

Lectures 5–6

Low-Dimensional Representations in Deep Networks (Part I , Part II)

Zhihui Zhu, Qing Qu

• A Geometric Analysis of Neural Collapse with Unconstrained Features
• On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Features
• Robust Training under Label Noise by Over-parameterization
• The Law of Parsimony in Gradient Descent for Learning Deep Linear Networks
• Principled and Efficient Transfer Learning of Deep Models via Neural Collapse
• Are All Losses Created Equal: A Neural Collapse Perspective

Lecture 7

Deep Representation Learning from the Ground Up

Sam Buchanan

• Chapter 16, WM22
• Deep Networks and the Multiple Manifold Problem
• Deep Networks Provably Classify Data on Curves
• White-Box Transformers via Sparse Rate Reduction