Information Theory

Data is displayed for the academic year: 2024./2025.

Lecturers

Prof.
Alen Bažant
PhD
Prof.
Željko Ilić
PhD
Prof.
Igor Sunday Pandžić
PhD
Assoc. Prof.
Marin Vuković
PhD

Laboratory exercises

Antonia Žaja
univ. mag. ing. comp.

Course Description

Introduction to the quantitative Shannon’s theory of information and its applications. Mathematical definition and properties of information. Message. Amount information carried by a message. Entropy. Mutual information. Discrete channel capacity. Discrete memoryless sources. Discrete sources with memory (Markov chains). Source coding theorem, lossless data compression and optimal lossless coding. Shannon-Fano coding. Huffman coding. Arithmetic coding. Dictionary-based coding (LZ77, LZW). Error control. Parity coding. Cyclic code. Hamming code. Convolutional codes. Viterbi algorithm.

Prerequisites

Mathematical Analysis

Study Programmes

University undergraduate
[FER3-EN] Computing - study
(3. semester)

Learning Outcomes

  1. identify information, coding and communication problems
  2. explain coding and compression methods and information limits
  3. apply accepted knowledge to real systems analysis
  4. analyze complex information and communication systems
  5. explain phenomens in different areas of science
  6. estimate performances of different information and communication systems
  7. apply techniques of entropy and error correcting codes

Forms of Teaching

Lectures

Lectures are held in three hour units.

Independent assignments

Students solve problems, not mandatory for everyone.

Laboratory

Mandatory for all students, each subgroup solves one problem.

Grading Method

Continuous Assessment Exam
Type Threshold Percent of Grade Threshold Percent of Grade
Mid Term Exam: Written 10 % 50 % 0 %
Final Exam: Written 10 % 50 %
Exam: Written 40 % 100 %
Comment:

Although laboratory exercises does not contribute to a total number of points won on this course, their accomplishment is necessary requirement.

Week by Week Schedule

  1. Information theory history and importance, Symbol, message, information, communication
  2. Discrete communication system, probabilistic view and information measures
  3. Entropy, noiseless coding theorem, Mutual information
  4. Information sources
  5. Types of codes, Optimal code, Entropy coding
  6. Entropy coding
  7. Entropy coding, Lossy coding
  8. Midterm exam
  9. Error detecting and correcting codes, block codes
  10. Hamming distance, code equivalence, perfect codes
  11. Binary linear block codes, generating matrix, parity check matrix, syndrom
  12. Types of binary linear block codes
  13. Convolutional and turbo coding
  14. Channel capacity, noisy-channel coding theorem
  15. Final exam

Literature

Igor S. Pandžić et al. (2009.), Uvod u teoriju informacije i kodiranje, Element
Željko Ilić, Alen Bažant, Tomaž Beriša (2014.), Teorija informacije i kodiranje, Element
Roberto Togneri, Christopher J.S deSilva (2003.), Fundamentals of Information Theory and Coding Design, CRC Press

General

ID 209649
  Winter semester
4 ECTS
L0 English Level
L1 e-Learning
45 Lectures
0 Seminar
0 Exercises
15 Laboratory exercises
0 Project laboratory
0 Physical education excercises

Grading System

85 Excellent
70 Very Good
55 Good
40 Sufficient

Similar Courses