CS671: Probabilistic Agent Learning (Spring 2019)

https://storage.googleapis.com/deepmind-live-cms/documents/iclrgif.gif
[Image from https://deepmind.com/blog/reinforcement-learning-unsupervised-auxiliary-tasks/]

Please note that this syllabus is not final. There may be further adjustments. For exact information, visit sakai course page.


Overview
This seminar course covers recent advances in deep & probabilistic generative approaches for agent learning. This includes (1) fundamental advances in probabilistic generative modelling, mostly based on variational inference, and (2) its applications to designing various cognitive abilities such as imagination, exploration, and planning for a probabilistic agent. This includes topics such as structured latent representation learning, spatiotemporal models for future imagination, reinforcement learning as probabilistic inference, exploration/intrinsic motivation, hierarchical option discovery, meta/continual learning, model-based learning, and so on. 


Time, Location, Instructor

  • When: Thursdays 3:20pm - 6:20pm, from Jan 24 to May 9
  • Where: LSH B269
  • Instructor: Sungjin Ahn
  • Email: sungjin.ahn@cs.rutgers.edu (put "CS671" in the subject)

Office Hours

Tuesdays 9:30am - 10:30am in CBIM 9


Pre-Requisites

  • As pre-requisite, the course requires knowledge of fundamental concepts in machine learning, probability theory, and reinforcement learning. 
  • Programming skill in either TensorFlow or PyTorch is required.

Special Permission Number 
For those who needs SPN, please directly request it. The course registration is closed. 


Expected Work
Students are expected to read a few papers every week. Each week, we will also have two presentation sessions and discussion on the week's papers. There will be no mid/final written exam and homeworks. Students are requred to present the final project and submit a written report on the project.


Grading

  • Paper review (20%)
  • Presentation (20%)
    • How well the presenter understood the papers (7%)
    • How well the presenter explained the topic (7%)
    • How well the presenter lead the discussion and answered questions (6%)
  • Final project (60%) 
    • Significance of the result (30%)
    • Completeness of the final report (15%)
    • Presentation (10%)
    • Proposal (5%)

Detailed Schedule and Readings (Not finalized. Papers can be changed)

 
1. Jan 24 - Course Introduction
  • Course Overview - (Sungjin Ahn) 
  • Overview on Probabilistic Agent Leanring - (Sungjin Ahn)
  • Variational Autoencoders - (Gautam Singh)
    • *Auto-Encoding Variational Bayes
    • Importance Weighted Autoencoders
 
2. Jan 31 - DRL I & Advances in Variational Inference I
  • Deep Reinforcement Learning: Part I - (Sungjin Ahn)
  • Advanced VAE - 1 (Mun Kim)
    • DRAW: A Recurrent Neural Network For Image Generation
    • Variational Inference with Normalizing Flows
    • (optional) Towards Conceptual Compression
 
3. Feb 7 - DRL II & Advances in Variational Inference II 
  • Deep Reinforcement Learning: Part II - (Sungjin Ahn)
  • Advanced VAE - 2 (Bingchen Liu)
    • Categorical Reparameterization with Gumbel-Softmax 
    • REINFORCE: )>􏰟=􏰁􏰞􏰟􏰓􏰚􏰗􏰍􏰖􏰈A@(􏰋!􏰟 􏰈 􏰃 􏰈7􏰔?􏰚&%31>􏰟=􏰁 􏰈<:;􏰋9􏰑􏰕􏰁􏰞􏰟&%􏰕􏰁􏰎􏰋8􏰔7􏰈6􏰟0􏰟&%54 􏰚&%312􏰑􏰃0/􏰐􏰋.􏰁􏰓􏰚&%-)􏰒􏰆,+)*􏰟􏰄􏰁('􏰖%􏰉􏰆􏰄􏰆&%􏰞#$"􏰓􏰋!􏰟􏰈􏰉􏰁􏰞􏰝􏰗􏰍􏰜􏰚􏰛􏰘􏰙􏰆􏰗􏰍􏰖􏰔􏰕􏰁􏰐􏰋􏰓􏰑􏰒􏰁􏰐􏰋􏰏􏰍􏰎􏰋􏰌􏰀􏰊􏰈􏰉􏰆􏰇􏰅􏰃􏰄􏰁􏰂􏰀)>􏰟=􏰁􏰞􏰟􏰓􏰚􏰗􏰍􏰖􏰈A@(􏰋!􏰟 􏰈 􏰃 􏰈7􏰔?􏰚&%31>􏰟=􏰁 􏰈<:;􏰋9􏰑􏰕􏰁􏰞􏰟&%􏰕􏰁􏰎􏰋8􏰔7􏰈6􏰟0􏰟&%54 􏰚&%312􏰑􏰃0/􏰐􏰋.􏰁􏰓􏰚&%-)􏰒􏰆,+)*􏰟􏰄􏰁('􏰖%􏰉􏰆􏰄􏰆&%􏰞#$"􏰓􏰋!􏰟􏰈􏰉􏰁􏰞􏰝􏰗􏰍􏰜􏰚􏰛􏰘􏰙􏰆􏰗􏰍􏰖􏰔􏰕􏰁􏰐􏰋􏰓􏰑􏰒􏰁􏰐􏰋􏰏􏰍􏰎􏰋􏰌􏰀􏰊􏰈􏰉􏰆􏰇􏰅􏰃􏰄􏰁􏰂􏰀Simple Statistical Gradient-Following Algorithms for Connectionist ...Simple Statistical Gradient-Following Algorithms for Connectionist ...Simple Statistical Gradient-Following Algorithms for Connectionist ...Simple Statistical Gradient-Following Algorithms for Connectionist ...◊Simple Statistical Gradient-Following Algorithms for Connectionist ...
4. Feb 14 - Bayesian Neural Networks & Generative Temporal Models 
  • Bayesian Neural Networks (Weihao Sun)
    • Weight Uncertainty in Neural Networks
    • Multiplicative Normalizing Flows for Variational Bayesian Neural Networks 
  • Generative Temporal Models - 1 (Gautam Sing)
    • A Recurrent Latent Variable Model for Sequential Data
    • Structured Inference Networks for Nonlinear State Space Models
    • (optional) 
      • Sequential Neural Models with Stochastic Layers
      • Deep Variational Bayes Filters: Unsupervised Learning of State Space ...
 
5. Feb 21 - Generative Temporal Models (cont.) & Memory
  • Generative Temporal Models - 2 (Jindong Jiang)
    • Auto-Encoding Sequential Monte Carlo
    • Deep Variational Reinforcement Learning for POMDPs
    • (optional)
      • Variational Sequential Monte Carlo
 
6. Feb 28 - Memory (cont.) & Representation
  • Memory 2: Generative Distributed Memory (Vladimir Ivanov)
    • The Kanerva Machine: A Generative Distributed Memory
    • Learning Attractor Dynamics for Generative Memory
    • (optional) Approximating Bayesian inference with a sparse distributed memory system
  • Representation 1: Beta-VAE (Ligong Han)
    • beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework 
    • Understanding disentangling in beta-VAE
    • (optional) Isolating Sources of Disentanglement in VAEs
 
7. Mar 7 - Representation & Structured Perception
  • Representation 2: Contrastive Predictive Coding (Eunyoung Kim)
    • Representation Learning with Contrastive Predictive Coding
    • Neural Predictive Belief Representations
  • Structured Perception - 1 
    • AIR: Attend, Infer, Repeat: Fast Scene Understanding with Generative Models
    • SQAIR: Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects
 
8. Mar 14 
  • Project Proposal 
 
9. Mar 21 - SPRING RECESSION

 
10. Mar 28 - Structured Perception & Graph-Relation 
  • Structured Perception - 2
    • RNEM: Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactions
    • Unsupervised Learning of 3D Structure from Images
    • MONet: Unsupervised Scene Decomposition and Representation
  • Graph & Relation
    • Neural Relational Inference for Interacting Systems
    • Graph networks as learnable physics engines for inference and control
    • Relational recurrent neural networks
 
11. Apr 4 - Meta & Continual Learning
  • Meta Learing 
    • Towards a Neural Statistician
    • Amortized Bayesian Meta-Learning
    • (optional) Recasting Gradient-Based Meta-Learning as Hierarchical Bayes
    • (optional) The Variational Homoencoder: Learning to learn high capacity generative models from few examples
    • (optional) VERSA: Versatile and Efficient Few-shot Learning
 
12. Apr 11 - Generative Query Networks (GQN) and Program Learning
  • GQN & Neural Processes
    • Neural Scene Representation and Rendering
    • Neural Processes
    • (optional) CGQN: Consistent Generative Query Networks
    • (optional) Attentive Neural Processes
 
13. Apr 18 - Model-Based RL
  • Model-Based RL - 1: Imagination-Augmented Agents
    • I2A: Imagination-Augmented Agents for Deep Reinforcement Learning
    • I2A-SS: Learning and Querying Fast Generative Models for Reinforcement ...
 
14. Apr 25 - Unsupervised RL (Exploration, Intrinsic Motivation, Options)
  • Options
    • Variational Intrinsic Control
    • Diversity is All You Need: Learning Skills without a Reward Function
    • (optional) Variational Option Discovery Algorithms
 
15. May 2
TBD (Probabilistic Reinforcement Learning)
TBD (Spatial Navigation)
  • Spatial Learning - 1
    • Vector-Based Navigation Using Grid-Like Representations in Artificial Agents (Nature 18)
    • Neural Map: Structured Memory for Deep Reinforcement Learning (ICLR 18)
  • Spatial Learning - 2
    • Emergence of Grid-Like Representations by Training Recurrent Neural Networks to Perform Spatial Localization (ICLR 18)
    • Semi-parametric Topological Memory for Navigation (ICLR 18)
  • Spatial Learning - 3
    • Unsupervised Emergence of Spatial Structure from Sensorimotor Prediction (ICLR 19)
 
16. May 9 - Final Project Presentation