Objectives of the course:

Recent trends in Neural Networks imposed hardware accelerators as the most viable solution for several classes of applications such as image recognition and classification for self-driving cars, computer vision and speech recognition. Main goal of the course is to present the major concepts and different hardware optimization techniques used in computer architectures to accelerate Deep Neural Networks. More in detail, the course presents and discusses how the mapping of DNNs interact with the runtime usage of hardware resources to optimize performance and memory bandwidth. Overall goal is to understand the most recent computer architecture commercial solutions and the trends that drive the research evolution in the field.

Program of the course:

1. Recap on Deep Neural Networks: Basic concepts and terminology; Fully-connected Neural Networks; Convolutional Neural Networks; Recurrent Neural Networks; Long Short-Term Memory;
2. Overview on Deep Neural Networks: LeNet, AlexNet, OverFeat, VGGNet, GoogleNet, ResNet; Datasets and metrics to evaluate and compare hardware solutions;
3. Deep Convolutional Neural Networks: Compiler optimizations of convolutional kernels; convolutional kernel computation in hardware; memory access and data reuse; fused layers; energy efficiency transformations; approximate precision;
4. Methodologies for design space exploration and runtime mapping of DCNNs to HW accelerators;
5. Survey of DNN architectures: Intel CPUs for Deep Learning; Nvidia GPUs for Deep Learning;
6. Survey of DNN accelerators: Google TPU family; STMicroelectronics Orlando SoC; Intel Movidius VPUs; MIT Eyeriss DNN Accelerator; Imperial College FPGAConvNet for DL; Nvidia NVDLA on Xavier SoC; GreenWave GAP8;

Pre-requirements:
Knowledge on computer architectures at the level of the MS. Course on "Advanced Computer Architectures" is required. Basic concepts on deep learning is required.

Expected learning outcomes:

DdD 1 (Knowledge and understanding): Students will learn how to:

• Understand and analyze the key design solutions to implement DCNNs;
• Understand the most advanced techniques used to accelerate in hardware DNNs;
• Understand recent architectures and technology trends and opportunities;

DdD 2 (Applying knowledge and understanding): Given specific problems and project cases, students will be capable to:

• Evaluate the performance of DNN accelerators by applying the appropriate metrics;
• Understand power/performance tradeoffs among various design architecture solutions;

DdD3 (Making judgements): Given specific problems and project cases, students will be capable to analyze the performance goals and compare autonomously different architectures for Deep Neural Networks in terms of power/performance tradeoffs.

DdD4 (Communication skills): Students will learn how to present orally a scientific text and how to discuss in front of their colleagues the benefits and drawbacks of different technical solutions.

DdD5 (Learning skills): Studens will be capable lo learn autonomously the most advanced research techniques related to computer architectures for Deep Neural Networks.

Final exam:

At the end of the course, an oral exam will verify the level of knowledge reached per each student. The oral exam will cover concepts and techniques presented during the course -- Expected learning outcomes 1, 2, 3, 5.

Optionally, students can ask to be assigned to: 1) a scientific paper to be presented orally and discussed in terms of benefits and drawbacks; or 2) to develop a design project to implement some solutions and run some empirical experiments by using open-source design tools -- Expected learning outcomes 1, 2, 3, 5.

Slides will be available through the course website in Beep.