MathsDL-spring18

Lecture 1: The Curse of Dimensionality

Main References

• “Breaking the curse of dimensionality with convex neural networks”, F. Bach

• “Understanding Machine Learning: From Theory to Algorithms”, S. Shalev-Swartz, Ben-David

• “Nesterov Punctuated Equilibrium”, argmin post by Frostig & Recht

• “Failures of Gradient-Based Deep Learning”, S. Shalev-Shwartz et al.

Further References

• “Introduction to Non-parametric Estimation”, A. Tsybakov

• “EQUIVALENCE OF DISTANCE-BASED AND RKHS-BASED STATISTICS IN HYPOTHESIS TESTING”, Sejdinovic et al

• “A primer on OT”, M.Cuturi and J.Solomon, NIPS’17 tutorial

• “Sharp asymptotic and finite-sample rates of convergence of empirical measures in Wasserstein distance”, J.Weed and F.Bach

• “High-Dimensional Probability”, R. Vershynin

• “Random Gradient-Free Minimization of Convex Functions”, Y.Nesterov

Lecture 2: Geometric Stability in Euclidean Domains.

Main References:

• Group Invariant Scattering, S. Mallat

• Invariant Scattering Convolutional Networks, J.Bruna, S. Mallat

Lecture 3: The Scattering Transform and Beyond

Main References:

• Group Invariant Scattering, S. Mallat

• Scattering Representations for Recognition, J.Bruna PhD Thesis.

• Rotation, Scaling and Deformation Invariant Scattering for Texture Discrimination, Sifre and Mallat, CVPR15.

Further References:

• Exponential Decay of Scattering Coefficients, I. Waldspurger.

• Analysis of Time-Frequency Scattering Transforms, Czaja and Li.

• Energy Propagation in Deep Convolutional Neural Networks, Wiatowski et al.

Lecture 4: Non-Euclidean Geometric Stability and Graph Neural Networks

Main References:

• Geometric Deep Learning: Going beyond Euclidean Data, M. Bronstein et al, 17

Further References:

• i-RevNet: Deep Invertible Networks, Jacobsen, Smeulders, Oyallon, ICLR’18

• Spherical CNNs, Cohen, Welling et al, ICLR’18

• Deep Image Prior, Ulyanov, Vedaldi et al,’17

• Community Detection with Graph Neural Networks, B. and Li’18

Lecture 5: Graph Neural Network Applications

Main References:

• Geometric Deep Learning: Going beyond Euclidean Data, M. Bronstein et al, 17

• Community Detection with Graph Neural Networks, B. and Li’18

• Neural Message Passing for Quantum Chemistry, Gilmer et al.17

Further References:

• Semi-Supervised Classification with Graph Convolutional Networks, Kipf & Welling

• Representation Learning on Graphs: Methods and Applications, Hamilton, Ying and Leskovec

• Quadratic Assignment with Graph Neural Networks, Nowak et al

Lecture 6: Unsupervised Learning under Geometric Priors

Main References:

• Geometrical Insights for Implicit Generative Modelling, Bottou et al. ‘18

• Sparse Multiscale Microcanonical Models, B. and Mallat’18

Further References:

• Autoencoding Variational Bayes, Kingma & Welling’14.

• Generalization and Equilibrium in GANs, Arora et al

Lecture 7: Discrete vs Continuous Time Optimization: The Convex Case

Main References:

• Large-scale Machine Learning and convex optimization, F. Bach, 17

• A differential Equation for Modelling Nesterov’s Accelerated Gradient Method, Su, Boyd, Candes, ‘14

Further References

• Convex Optimization, Bubeck’15

• A Lyapunov Analysis of Momentum Methods in Optimization, Wilson, Recht, Jordan’18

• On Symplectic Optimization, Betancourt, Jordan, Wilson

Lecture 8: Discrete vs Continuous Time Optimization: Stochastic and Nonconvex case

Main References:

• A Variational Perspective on Accelerated Methods in Optimization, Wibisono, Wilson and Jordan

• Bridging the Gap between Constant Step size Stochastic Gradient Descent and Markov Chains, Dieuleveut, Durmus, Bach

Lecture 9: Discrete vs Continuous Time Optimization: Stochastic and Nonconvex case

Main References:

• Stochastic Modified Equations and Adaptive Stochastic Gradient Descent Algorithms, Li, Tai & E

• Non-Convex Learning via Stochastic Gradient Langevin Dynamics: A nonasymptotic Analysis, Raginsky, Rakhlin, Telgarsky

• Fokker-Plank Equations, Pavlotis Lecture Notes

Lecture 10: Nonconvex Optimization

Main References:

• Gradient Descent Converges to Minimisers, Lee et al.’16

• Gradient Descent can take exponential time to escape saddle points, Lee et al.’17

• Escaping from Saddle points– online stochastic gradient for tensor decomposition, Ge et al.’15

• Deep Learning without poor local minima, Kawaguchi’16

Further References

• How to escape Saddle points efficiently, Jin et al.’17

Lecture 11: Landscape of Optimization

Main References:

• Random Matrices and the complexity of Spin Glasses, Auffinger et al’10

• Complex energy landscapes in spiked-tensor and simple glassy models: ruggedness, arrangements of local minima and phase transitions, Ross, Ben Arous, Biroli, Camarotta

• Neural Networks with Finite Intrinsic Dimension have no Spurious Valleys, Venturi et al.

• Topology and Geometry of Half-Rectified Network Optimization, Freeman et al

Further References

• On the Optimization Lanscape of Tensor Decomposition, Ge and Ma

• The landscape of the spiked tensor model, Ben Arous et al

Lecture 12: Guest Lecture Behnam Neyshabur (IAS/NYU): Generalization in Deep Learning

Main References:

• Understanding Machine Learning: From Theory to Algorithms. Shai Shalev-Shwartz and Shai Ben-David. Cambridge University Press, 2014: Part I (Foundations) and Part IV (Advanced Theory).

Further References:

• Implicit Regularization in Deep Learning. Behnam Neyshabur. PhD Thesis, 2017. Part I (Implicit Regularization and Generalization)

• Spectrally-normalized margin bounds for neural networks. Bartlett, Peter L., Dylan J. Foster, and Matus J. Telgarsky. NIPS 2017

• Computing Nonvacuous Generalization Bounds for Deep (Stochastic) Neural Networks with Many More Parameters than Training Data. Dziugaite, Gintare Karolina, and Daniel M. Roy. UAI 2017

Lecture 13: Landscape of Optimization of Deep Neural Networks. Positive and Negative Results

Main References:

• Neural Networks with Finite Intrinsic Dimension have no Spurious Valleys, Venturi et al.

• A Critical View of Global Optimiality in Deep Learning, Yun et al.’18

• Are Resnets provably better than Linear Predictors?, Shamir’18

Further References

• Solving the quantum many-body problem with artificial neural networks, Carleo et al, Science’17

• Essentially no barriers in Neural Network energy landscape, Draxler et al.’18

• The mechanics of n-player differentiable games, Balduzzi et al.’18