CS671: Probabilistic Agent Learning (Spring 2019)

https://storage.googleapis.com/deepmind-live-cms/documents/iclrgif.gif
[Image from https://deepmind.com/blog/reinforcement-learning-unsupervised-auxiliary-tasks/]
Instructor
Sungjin Ahn

Description
This seminar course covers recent advances in deep & probabilistic generative approaches for agent learning. This includes (1) fundamental advances in probabilistic generative modelling, mostly based on variational inference, and (2) its applications to designing various cognitive abilities such as imagination, exploration, and planning for a probabilistic agent. This includes topics such as structured latent representation learning, spatiotemporal models for future imagination, reinforcement learning as probabilistic inference, exploration/intrinsic motivation, hierarchical option discovery, meta/continual learning, model-based learning, and so on. 

Time and Location 
Thursday 3:20pm - 6:20pm
Busch SEC 206

Pre-requisites
- Machine Learning
- Understaning on Bayesian inference
- Reinforcement Learning

Expected Work
Reading papers, participating on/offline paper discussion, oral presentation, and final project

Grading
Final Project: (50%)
Written reactions to the reading (15%)
Participating in discussions (15%)
Oral Presentation (20%)





Final Assignment



Course Policies



Schedule and Overview of Topics
  • Fundamentals of Variational Inference
    • VAE, Gumbel-Softmax, REINFORCE, BNN
  • Temporal Models
    • VRNN
  • Exploration/Intrinsic Motivation/Options
    • VIME
    • Variational Intrinsic Motivation
    • Variational Option Discovery
    • Diversity is all you need
    • Stochastic Neural Networks for Hierarchical Reinforcement Learning
    • Probabilistic inference for determining options in reinforcement learning
    • Principled Option Learning in Markov Decision Processes
    • Multi-level Discovery of Deep Options
    • L EARNING AND P OLICY S EARCH IN S TOCHASTIC DYNAMICAL S YSTEMS WITH BAYESIAN N EURAL N ETWORKS
  • Imagination - 1
    • df
  • Representation Learning
    • Beta-VAE
    • SCAN
    • DARLA
    • Representation Learning with Contrastive Predictive Coding
  • Structured Perception
    • AIR
    • SQAIR
    • RNEM
  • GQN for 3D Representation Learning 
    • GQN
    • CGQN
    • (Conditional) Neural Processes
    • Attentive Neural Processes
  • Meta/Continual Learning
    • Neural Statistician
    • Generative Matching Network
    • Variational Continual Learning
  • Episodic Memory
    • df
  • Probabilistic Reinforcement Learning
    • Reinforcement Learning and Control as Probabilistic Inference ...
  • Navigation and Space Learning
    • EMERGENCE OF GRID-LIKE REPRESENTATIONS BY TRAINING RECURRENT NEURAL NETWORKS TO PERFORM SPATIAL LOCALIZATION 
    • UNSUPERVISED EMERGENCE OF SPATIAL STRUCTURE FROM SENSORIMOTOR PREDICTION
  • Language Grounding


Detailed Schedule and Readings
Total 14 classes (each 3 hours)
  • 1/24 Introduction and Overview
    • Basics on VAE and RL
  • 1/31
  • 2/7
  • 2/14
  • 2/21
  • 2/28
  • 3/7
  • 3/14
  • 3/21 No Class - Spring Recession
  • 3/28
  • 4/4
  • 4/11
  • 4/18
  • 4/25
  • 5/2