Ing Ind - Inf (Mag.)(ord. 270) - MI (474) TELECOMMUNICATION ENGINEERING - INGEGNERIA DELLE TELECOMUNICAZIONI

*

A

ZZZZ

093294 - INFORMATION THEORY

Ing Ind - Inf (Mag.)(ord. 270) - MI (476) ELECTRONICS ENGINEERING - INGEGNERIA ELETTRONICA

*

A

ZZZZ

093294 - INFORMATION THEORY

Obiettivi dell'insegnamento

This course deals with the theoretical foundations of source coding and of channel coding. Information measures are first introduced, and then applied to the analysis of the theoretical performance achievable in data compression and communication over noisy channels. The goal of the course is to teach students how the concepts related to the source and channel coding are used to model, analyze and design modern communication systems in order to enable efficient information processing. Since the application of these concepts is a fundamental aspect in every modern communication system, another goal is to establish concrete links of these concepts with advanced technologies used to process the information in different telecommunication systems (wireless and optical). Some of the topics will be presented using a more practical approach by means of examples built using commercial software tools.

Risultati di apprendimento attesi

1 - Knowledge and understanding After the end of the course, students should be able to: - define quantitative measure of information - know theoretical and methodological skills related to the concept of information - understand properties of entropy, mutual information as it applied to information - comprehend what is the rate-distortion theory and its related consequences 2 Applying knowledge and understaning After the end of the course, students should be able to: - understand and solve complex problems in different interdisciplinary areas - integrate and individually find and re-adapt known solutions to growing problems - analyse uncertainty in terms of information entropy and other information measures 3 - Making judgements After the end of the course, students should be able to: - define mathematical models by considering the informational aspects of a system and the related measures - evaluate concrete schemes for compression and communication

Argomenti trattati

1 - Entropy and source coding

Introduction to information theory. Entropy of a memoryless source. Coding of memoryless sources. Prefix codes. Kraft inequality. Huffman codes and Shannon codes. Source coding theorems (for memoryless sources). Joint entropy and conditional entropy. Chain rules. Entropy of sources with memory. Source coding theorem. Practical methods for source coding. Universal codes. Arithmetic coding. Lempel-Ziv coding.

2 - Channel capacity

Channel models. Discrete channels. Mutual information. Data processing inequality. Channel capacity. Coding of information for transmission on unreliable channels. Entropy, mutual information, and capacity for continuous channels. Gaussian AWGN channel. Channel coding theorem. Error exponent. Fano’s inequality. Converse of the channel coding theorem. Hints for practical channel codes.

3 - Rate distortion theory

Rate-distortion function. Coding of discrete and continuous sources with a fidelity criterion. Vector quantization. Channel coding with a fidelity criterion.

4 - Network information theory

Another look at source coding. Slepian-Wolf source coding. Multiple-access channels. Capacity regions. Gaussian multiple-access channel. Gaussian broadcast channel. Capacity regions.

Prerequisiti

This course requires a basic knowledge in computer science and solid understanding of probability theory and random variables.

Modalità di valutazione

The assessment will consist in a written exam followed by oral examination. The written part consists of numerical exercises (2/3) and open questions (2/1) on all the topics covered in the course. The written exam can assign up to 30 points, which constitute the starting point for the final mark after the oral. The mark 30 cum laude will be assigned when the total score after the oral is greater or equal 31. Optional projects, proposed during the course, can be selected to get assigned additional 3 points.

Bibliografia

Lecture noteshttps://beep.metid.polimi.it/T. M. Cover, J. A. Thomas, Elements of information theory (1st or 2nd edition), Editore: John Wiley & Sons, 1991 (1st ed.), 2006 (2nd ed.). First edition available online for PoliMi students (http://onlinelibrary.wiley.com/book/10.1002/0471200611)
D. J. C. MacKay, Information Theory, Inference, and Learning Algorithms, Editore: D. J. Cambridge University Press, 2003
R. G. Gallager, Information Theory and Reliable Communication, Editore: John Wiley & Sons, 1968

Forme didattiche

Tipo Forma Didattica

Ore di attività svolte in aula

(hh:mm)

Ore di studio autonome

(hh:mm)

Lezione

30:00

45:00

Esercitazione

20:00

30:00

Laboratorio Informatico

0:00

0:00

Laboratorio Sperimentale

0:00

0:00

Laboratorio Di Progetto

0:00

0:00

Totale

50:00

75:00

Informazioni in lingua inglese a supporto dell'internazionalizzazione

Insegnamento erogato in lingua
Inglese

Disponibilità di materiale didattico/slides in lingua inglese

Disponibilità di libri di testo/bibliografia in lingua inglese

Possibilità di sostenere l'esame in lingua inglese

Disponibilità di supporto didattico in lingua inglese