Ing Ind - Inf (Mag.)(ord. 270) - MI (474) TELECOMMUNICATION ENGINEERING - INGEGNERIA DELLE TELECOMUNICAZIONI
093294 - INFORMATION THEORY
Ing Ind - Inf (Mag.)(ord. 270) - MI (476) ELECTRONICS ENGINEERING - INGEGNERIA ELETTRONICA
093294 - INFORMATION THEORY
This course deals with the theoretical foundations of source coding and of channel coding. Information measures are first introduced, and then applied to the analysis of the theoretical performance achievable in data compression and communication over noisy channels. The goal of the course is to teach students how the concepts related to the source and channel coding are used to model, analyze and design modern communication systems in order to enable efficient information processing. Since the application of these concepts is a fundamental aspect in every modern communication system, another goal is to establish concrete links of these concepts with advanced technologies used to process the information in different telecommunication systems (wireless and optical). Some of the topics will be presented using a more practical approach by means of examples built using commercial software tools.
Risultati di apprendimento attesi
1 - Knowledge and understanding After the end of the course, students should be able to: - define quantitative measure of information - know theoretical and methodological skills related to the concept of information - understand properties of entropy, mutual information as it applied to information - comprehend what is the rate-distortion theory and its related consequences 2 Applying knowledge and understaning After the end of the course, students should be able to: - understand and solve complex problems in different interdisciplinary areas - integrate and individually find and re-adapt known solutions to growing problems - analyse uncertainty in terms of information entropy and other information measures 3 - Making judgements After the end of the course, students should be able to: - define mathematical models by considering the informational aspects of a system and the related measures - evaluate concrete schemes for compression and communication
1 - Entropy and source coding
Introduction to information theory. Entropy of a memoryless source. Coding of memoryless sources. Prefix codes. Kraft inequality. Huffman codes and Shannon codes. Source coding theorems (for memoryless sources). Joint entropy and conditional entropy. Chain rules. Entropy of sources with memory. Source coding theorem. Practical methods for source coding. Universal codes. Arithmetic coding. Lempel-Ziv coding.
2 - Channel capacity
Channel models. Discrete channels. Mutual information. Data processing inequality. Channel capacity. Coding of information for transmission on unreliable channels. Entropy, mutual information, and capacity for continuous channels. Gaussian AWGN channel. Channel coding theorem. Error exponent. Fano’s inequality. Converse of the channel coding theorem. Hints for practical channel codes.
3 - Rate distortion theory
Rate-distortion function. Coding of discrete and continuous sources with a fidelity criterion. Vector quantization. Channel coding with a fidelity criterion.
4 - Network information theory
Another look at source coding. Slepian-Wolf source coding. Multiple-access channels. Capacity regions. Gaussian multiple-access channel. Gaussian broadcast channel. Capacity regions.
This course requires a basic knowledge in computer science and solid understanding of probability theory and random variables.
Modalità di valutazione
The assessment will consist in a written exam followed by oral examination. The written part consists of numerical exercises (2/3) and open questions (2/1) on all the topics covered in the course. The written exam can assign up to 30 points, which constitute the starting point for the final mark after the oral. The mark 30 cum laude will be assigned when the total score after the oral is greater or equal 31. Optional projects, proposed during the course, can be selected to get assigned additional 3 points.
Lecture noteshttps://beep.metid.polimi.it/T. M. Cover, J. A. Thomas, Elements of information theory (1st or 2nd edition), Editore: John Wiley & Sons, 1991 (1st ed.), 2006 (2nd ed.). First edition available online for PoliMi students (http://onlinelibrary.wiley.com/book/10.1002/0471200611)
D. J. C. MacKay, Information Theory, Inference, and Learning Algorithms, Editore: D. J. Cambridge University Press, 2003
R. G. Gallager, Information Theory and Reliable Communication, Editore: John Wiley & Sons, 1968
Tipo Forma Didattica
Ore di attività svolte in aula
Ore di studio autonome
Laboratorio Di Progetto
Informazioni in lingua inglese a supporto dell'internazionalizzazione
Insegnamento erogato in lingua
Disponibilità di materiale didattico/slides in lingua inglese
Disponibilità di libri di testo/bibliografia in lingua inglese
Possibilità di sostenere l'esame in lingua inglese
Disponibilità di supporto didattico in lingua inglese