CS492(H): Machine Learning for 3D Data

Minhyuk Sung, KAIST, Spring 2021

Description

3D Data (both 3D scans captured by depth sensors and 3D models created by designers) are widely used in many applications in computer vision, computer graphics, and robotic, such as autonomous driving, AI-assisted 3D object/scene design, augmented reality, and physical robot interaction. Along with the recent increasing demands on processing and analyzing such 3D data, there has been tremendous progress in developing novel technologies, especially based on deep learning. In this course, we will cover the recent advances in machine learning techniques for 3D data and also discuss the remaining challenges. Most of the course material will be less-than-5-year-old research papers in CVPR/ICCV/ECCV (Vision), SIGGRAPH/SIGGRAPH Asia (Graphics), and NeurIPS/ICML (Machine Learning). The course will be project-oriented (no exam, no paper-and-pencil homework, but easy programming assignment), and it will combine pedagogical lectures and seminar-style reading group presentations (followed by interactive discussions). Lectures will be held online via zoom.

Prerequisites

This course is intended for undergraduate/master students who have a basic background in deep learning and experience with PyTorch. But there will be jump-start lectures/sessions for students who do not have any background in deep learning.

Time & Location

Time: Tue/Thu 9:00am - 10:15am (KST)
Location: Zoom

Come join the course introduction session!
Mar 02 (Tue) 9:00am - 10:15am (KST)
https://kaist.zoom.us/j/84555052588?pwd=Q0xvSUczcU1aQmtNeVZyNFU3ZFJvQT09
Passcode: cs492-3dml

You’ll need a KAIST email address to join.

Course Staff

Instructor: Minhyuk Sung

• Email:
• Office hours: By appointment (See #announcements channel in slack.)
  

TA: Kisoo Kim

• Email: k014520@kaist.ac.kr
  

Links

• Course Website: http://mhsung.github.io/courses/kaist-cs492-3dml-2021-spring
  
• Zoom Links & Submission & Grading: KLMS
  
• Discussion and Q&A: Slack (An invitation will be sent to the registered students by email.)
  

Grading

• Project: 50%
  • Project Pitch Video
  • Project Submission Guidelines
  • Poster Sessions
• Paper Review Presentation: 30%
• Programming Assignments: 10%
  • KCloud Setup Instruction: Link
  • PointNet Code: Link
• In-Class Participation: 10%
  

Course Logistics

Link (Last Updated: Mar 01)

Important Dates

(ALL ASSIGNMENTS ARE DUE 23:59 KST. NO LATE DAYS!)

• Project Sign-Up: Due Mar 11 (Thu) Due Mar 05 (Fri)
• 1st Programming Assignment: Due Mar 23 (Tue)
• 2nd Programming Assignment: Due Apr 01 (Thu)
• Project Proposal: Due Apr 01 (Thu)
• Project Pitch Video: Due Apr 13 (Tue)
• Presentation Preparation Check-In: Due two weeks before your presentation date
• Presentation Slides: Due 4 days before your presentation date
• Project Midterm Check-In: Due May 13 (Thu)
• Project Report/Poster/Code: Due Jun 06 (Sun)
• Project Review: Due Jun 17 (Thu)
• Project Rebuttal & Final Decision: Due Jun 20 (Sun)

Schedule

Click for details.

Green: Lectures
Red: Student presentations

Week	Tue	Topic	Thu	Topic
01	Mar 02	Course Introduction	Mar 04	Deep Learning Jump-Start Session 1 Material: Stanford CS231n
02	Mar 09	Deep Learning Jump-Start Session 2 Material: Stanford CS231n	Mar 11	Neural Networks for Point Cloud Data PointNet (CVPR 2017) PointNet++ (NeurIPS 2017) Point Set Generation (CVPR 2017) Project Sign-Up Due Date
03	Mar 16	PyTorch / PointNet Session	Mar 18	Examples of Supervised / Weakly-Supervised Learning SPFN (CVPR 2019) Deep Functional Dictionaries (NeurIPS 2018)
04	Mar 23	Spectral Geometry Processing Material: SGP Summer School 2016 (Laplace-Beltrami) Spectral Geometry Proceesing (Eurographics 2008) 1st Programming Assignment Due Date	Mar 25	Functional Map / Deep Spectral Processing Tutorial (SIGGRAPH 2017) Functional Maps (SIGGRAPH 2012) Deep Functional Maps (ICCV 2017) SyncSpecCNN (CVPR 2017)
05	Mar 30	Guest Lecture 1 Hao Su (UCSD) Title: Compositional Generalizability in Geometry, Physics, and Policy Learning	Apr 01	Neural Networks for Volumetric Data Paper 1: O-CNN (SIGGRAPH 2017) Presenter: Byeoli Choi OctNet (CVPR 2017) Adaptive O-CNN (SIGGRAPH Asia 2018) Paper 2: SparseConvNet (arXiv) Presenter: Kyounga Woo SparseConvNet (CVPR 2018) 2nd Programming Assignment Due Date Project Proposal Due Date
06	Apr 06	Guest Lecture 2 Vladimir G. Kim (Adobe Research) Title: Neural Mesh Processing	Apr 08	Neural Networks for Implicit Functions Paper 1: DeepSDF (CVPR 2019) Presenter: Wonkwang Lee IM-NET (CVPR 2019) Occupancy Networks (CVPR 2019) Deep Meta Functionals (ICCV 2019) Paper 2: Sirens (NeurIPS 2020) Presenter: Andréas Meuleman
07	Apr 13	Neural Networks for Meshes Paper 1: MeshCNN (SIGGRAPH 2019) Presenter: Dahyun Kang MeshNet (AAAI 2019) Paper 2: DualConvMesh-Net (CVPR 2020) Presenter: Juil Koo Project Pitch Video Due Date	Apr 15	Project Pitches
08	Apr 20	Midterm Week (No Class)	Apr 22	Midterm Week (No Class)
09	Apr 27	Supervised 2D-to-3D Paper 1: Pixel2Mesh (ECCV 2018) Presenter: Chaeyeon Chung Pixel2Mesh++ (ICCV 2019) Paper 2: PIFu (CVPR 2019) Presenter: Whie Jung PIFuHD (CVPR 2020)	Apr 29	Unsupervised 2D-to-3D Paper 1: Soft Rasterizer (ICCV 2019) Presenter: Hakyung Kim Paper 2: Unsup3D (CVPR 2020) Presenter: Minsoo Lee
10	May 04	Shape Parsing / Abstraction Paper 1: Volumetric Primitives (CVPR 2017) Presenter: Taegyu Jin Superquadrics Revisited (CVPR 2019) Hierarchical Cuboid Abstractions (SIGGRAPH Asia 2019) Paper 2: BSP-Net (CVPR 2020) Presenter: Jihyun Lee (Auditor) CvxNet (CVPR 2020)	May 06	Shape Alignment Paper 1: Deep Closest Point (ICCV 2019) Presenter: Shinjeong Kim DeepICP (ICCV 2019) Paper 2: Deep Global Registration (CVPR 2020) Presenter: Jaesung Choe
11	May 11	Learning 3D Structure 1 Paper 1: GRASS (SIGGRAPH 2017) Presenter: Hankyu Jang Im2Struct (CVPR 2018) SCORES (SIGGRAPH Asia 2018) Paper 2: StructureNet (SIGGRAPH Asia 2019) Presenter: Inhee Lee (Auditor) PartNet (CVPR 2019) StructEdit (CVPR 2020)	May 13	Learning 3D Structure 2 Paper 1: CSGNet (CVPR 2018) Presenter: Chanhyeok Park UCSG-Net (NeurIPS 2020) Paper 2: GAN2Shape (ICLR 2021) Presenter: Hangil Park Project Midterm Check-In Due Date
12	May 18	Detection/Semantic Segmentation in Scenes Paper 1: Deep Hough Voting (ICCV 2019) Presenter: Jeonghyun Kim Paper 2: MinkowskiNet (ICCV 2019) Presenter: Seungwoo Yoo	May 20	Instance Segmentation in Scenes Paper 1: 3D-SIS (CVPR 2019) Presenter: Chungsu Jang Paper 2: PointGroup (CVPR 2020) Presenter: Junho Lee OccuSeg (CVPR 2020)
13	May 25	3D Generative Models Paper 1: Point Cloud GAN (ICML 2018) Presenter: Yunpyo An Paper 2: MeshVAE (CVPR 2018) Presenter: Hojun Cho Automatic Unpaired Shape Deformation Transfer (SIGGRAPH Asia 2018)	May 27	Neural Rendering Paper 1: DeepVoxels (CVPR 2019) Presenter: Soomin Park State of the Art on Neural Rendering (EG 2020) Tutorial (CVPR 2020) Paper 2: NeRF (ECCV 2020) Presenter: In-young Cho NeRD (arXiv)
14	Jun 01	3D Shape Flow Paper 1: NeuralODE (NeurIPS 2019) Presenter: Hyunsoo Kim Paper 2: PointFlow (ICCV 2019) Presenter: Mustafa Berk Yaldiz Neural Mesh Flow (NeurIPS 2020) ShapeFlow (NeurIPS 2020)	Jun 03	3D Transformers Paper 1: PCT: Point Cloud Transformer (arXiv) Presenter: Seongjoo Moon Paper 2: Point Transformer (arXiv) Presenter: Shyngys Aitkazinov
15	Jun 08	Project Presentations 1 Project Report/Poster/Code Due Jun 06 (Sun)	Jun 10	Project Presentations 2 Project Review Due Jun 13 (Sun)
16	Jun 15	Final Week (No Class)	Jun 17	Final Week (No Class)

Useful Links

This webpage kindly provides a comprehensive summary of resources regarding 3D machine learning:
https://github.com/timzhang642/3D-Machine-Learning

Acknowledgements

Most of the lectures will be based on the materials in the following courses:
Stanford CS468: Machine Learning for 3D Data (Spring 2017)
UCSD CSE291-I00: Machine Learning for 3D Data (Winter 2018)
A Tutorial on 3D Deep Learning (CVPR 2017)