Risorse bibliografiche
Risorsa bibliografica obbligatoria
Risorsa bibliografica facoltativa
Scheda Riassuntiva
Anno Accademico 2019/2020
Scuola Scuola di Ingegneria Industriale e dell'Informazione
Insegnamento 054322 - INFORMATION THEORY
Docente Magarini Maurizio
Cfu 5.00 Tipo insegnamento Monodisciplinare
Didattica innovativa L'insegnamento prevede  1.0  CFU erogati con Didattica Innovativa come segue:
  • Cotutela con mondo esterno

Corso di Studi Codice Piano di Studio preventivamente approvato Da (compreso) A (escluso) Insegnamento

Obiettivi dell'insegnamento

This course is about how to measure, represent, and communicate information effectively. Why bits have become the universal currency for information exchange. How information theory bears on the design and operation of modern-day systems such as smartphones and the Internet. Information measures are first introduced, and then applied to the analysis of the theoretical performance achievable in data compression and communication over noisy channels.
The goal of the course is to teach students how the concepts related to the source coding and channel capacity are used to model, analyze and design modern communication systems in order to enable efficient information processing. Since the application of these concepts is a fundamental aspect in every modern communication system, another goal is to establish concrete links of these concepts with advanced technologies used to process the information in different systems. To this aim, a project will be assigned to apply the concepts and principles learned during the lessons.

Risultati di apprendimento attesi

1 - Knowledge and understanding
After the end of the course, students should be able to:
- define quantitative measure of information
- know theoretical and methodological skills related to the concept of information
- understand properties of entropy, mutual information as it applied to information processing
- define the multi-terminal source and channel coding problems and describe the coding theorems concerning distributed compression, multiple access, and broadcast.

2 - Applying knowledge and understaning
After the end of the course, students should be able to:
- understand and solve complex problems in different interdisciplinary areas
- integrate and individually find and re-adapt known solutions to growing problems
- analyse uncertainty in terms of information entropy and other information measures
- summarize and present the results achieved during the analysis and specification activities

3 - Making judgements
After the end of the course, students should be able to:
- define mathematical models by considering the informational aspects of a system and the related measures
- evaluate concrete schemes for compression and communication

Argomenti trattati

1 – Introduction to information theory

Historical background. Introduction to information theory and its applications.

2 - Entropy and source coding

Entropy of a memoryless source. Coding of memoryless sources. Prefix codes. Kraft inequality. Huffman codes and Shannon codes. Source coding theorems (for memoryless sources). Joint entropy and conditional entropy. Chain rules. Entropy of sources with memory. Source coding theorem. Practical methods for source coding. Universal codes. Arithmetic coding.

3 - Channel capacity

Channel models. Discrete channels. Mutual information. Data processing inequality. Channel capacity. Coding of information for transmission on unreliable channels. Entropy, mutual information, and capacity for continuous channels. Gaussian AWGN channel. Channel coding theorem. Error exponent. Fano’s inequality. Converse of the channel coding theorem.

4 - Network information theory

Another look at source coding. Slepian-Wolf source coding. Multiple-access channels. Capacity regions. Gaussian multiple-access channel. Gaussian broadcast channel. Capacity regions.

5 - Project on selected topics in information theory

A project laboratory, which is integral part of the course, will be assigned during the semester. This activity, which constitutes the innovative teaching part of the course, will be done also in cooperation with invited companies that will suggest projects related to the application of information theory in their specific field. The objective of projects is to help students in applying the concepts and principles learned during the lessons. The project, to be done in groups of two/three students, will be assigned in the first weeks of the semester. Project artifacts are expected to be released at the end of the course. The evaluation of projects will be based on the produced artifacts (documentation, code, …)  and on a final presentation. The project laboratory will include class discussions supervised by the course instructors and researchers from invited companies.





This course requires a basic knowledge in computer science and solid understanding of probability theory and random variables.

Modalità di valutazione

The assessment will consist in a written exam and a presentation of the project laboratory.
The written part consists of numerical exercises (2/3) and open questions (2/1) on all the topics covered in the course.
The written exam can assign up to 27 points, which constitute the starting point for the final mark after the presentation of the project laboratory.
The mark 30 cum laude will be assigned when the total score after the presentation of the project laboratory is greater or equal 31.

Risorsa bibliografica obbligatoriaLecture notes https://beep.metid.polimi.it/
Risorsa bibliografica facoltativaT. M. Cover, J. A. Thomas, Elements of information theory (1st or 2nd edition), Editore: John Wiley & Sons, 1991 (1st ed.), 2006 (2nd ed.). First edition available online for PoliMi students (http://onlinelibrary.wiley.com/book/10.1002/0471200611)
Risorsa bibliografica facoltativaD. J. C. MacKay, Information Theory, Inference, and Learning Algorithms, Editore: D. J. Cambridge University Press, 2003
Risorsa bibliografica facoltativaR. G. Gallager, Information Theory and Reliable Communication, Editore: John Wiley & Sons, 1968

Software utilizzato
Nessun software richiesto

Forme didattiche
Tipo Forma Didattica Ore di attività svolte in aula
Ore di studio autonome
Laboratorio Informatico
Laboratorio Sperimentale
Laboratorio Di Progetto
Totale 50:00 75:00

Informazioni in lingua inglese a supporto dell'internazionalizzazione
Insegnamento erogato in lingua Inglese
Disponibilità di materiale didattico/slides in lingua inglese
Disponibilità di libri di testo/bibliografia in lingua inglese
Possibilità di sostenere l'esame in lingua inglese
Disponibilità di supporto didattico in lingua inglese
schedaincarico v. 1.8.4 / 1.8.4
Area Servizi ICT