CS671: Probabilistic Agent Learning (Spring 2019)

https://storage.googleapis.com/deepmind-live-cms/documents/iclrgif.gif
[Image from https://deepmind.com/blog/reinforcement-learning-unsupervised-auxiliary-tasks/]

This syllabus is not final. There may be further adjustments.

Overview
This seminar course covers recent advances in deep & probabilistic generative approaches for agent learning. This includes (1) fundamental advances in probabilistic generative modelling, mostly based on variational inference, and (2) its applications to designing various cognitive abilities such as imagination, exploration, and planning for a probabilistic agent. This includes topics such as structured latent representation learning, spatiotemporal models for future imagination, reinforcement learning as probabilistic inference, exploration/intrinsic motivation, hierarchical option discovery, meta/continual learning, model-based learning, and so on. 

Time, Location, Instructor
  • When: Thursdays 3:20pm - 6:20pm, from Jan 24 to May 9
  • Where: LSH B269
  • Instructor: Sungjin Ahn
  • Email: sungjin.ahn@cs.rutgers.edu (put "CS671" in the subject)
Office Hours
Tuesdays 9:30am - 10:30am in CBIM 9

Pre-Requisites
As pre-requisite, the course requires knowledge of fundamental concepts in machine learning, probability theory, and reinforcement learning. Programming skill in either TensorFlow or PyTorch is required.

Special Permission Number 
For those who needs SPN, please directly request it. The course registration is closed. The course is full. There are no more available SPNs.

Expected Work
Students are expected to read a few papers every week and participates in discussion. Each student is , a differnt student will have oral presentation and discussion on the week's papers prepared by a student. There will be no mid/final written exam and homeworks. Students are requred to present the final project and submit a written report on the project.

Presentation
  • Each week consists of three 50-minute topic sessions. We have total 36 (3 x 12 weeks) topic sessions during the semester. 
  • Each student chooses a topic-session to present. During a session, a presenter is expected to (1) present a tutorial on the session papers and (2) lead discussions. The time allocation between tutorial presentation and discussion can vary for each session depending on the papers. Typically, 20~30 minutes for tutorial and 20~30 minutes for discussion is expected. 
  • Some papers require some background. The presenter is expected to study the background.
  • The three topic sessions of a week are usually closely relevant each other. The three presenters need to read not only the papers of an assigned session but also papers of the other relevant sesssions. The presenters then need to collaborate each other to make all three sessions flow smoothely without redundancy.
  • Evaluation will be done 70% by the instructor and 30% by the peers.

Understanding of the topic
Making others understand
Discussion 
Connection to other session

Final Project
  • The final project is to write a research paper on one of the topics covered in the class. Specifically, a project is expected to 
    • (1) contain scientific progress from the previous work (i.e., one should be able to explain why this is better than the previous works)
    • (2) implement and evaluate the proposed model using either Tensorflow or PyTorch. 
    • (3) share the code in a git repository.
    • (4) write a final report in a two column 4 pages template. Reference starts from 5th page without limit.
  • One can do the final project individually or make a team of max 2 persons. For team projects, the final report needs to specify who did what.
  • Each team (including single person teams) will propose the project proposal by XXXX
  • On May 9th, we have the final presentation.
Grading
Final project (60%) Scientific contribution (30%)Written final report (10%)Presentation (10%)Proposal (10%)Paper review (20%)12~13 weeks 
  • Session presentation (20%)
  • Online discussion (10%)

    Overview [Tentative]
    1. Fundamentals of Variational Inference and Reinforcement Learning
    2. Generative Temporal Models
    3. Exploration/Intrinsic Motivation/Options
    4. Imagination and model-based RL
    5. Representation Learning
    6. Structured Perception
    7. 3D Representation Learning and Neural Processes
    8. Meta/Continual Learning
    9. Episodic Memory
    10. Probabilistic Reinforcement Learning
    11. Navigation and Space
    12. Language Grounding