Information Theory

Course Description

Introduction to the quantitative Shannon?s theory of information and its applications, especially to information coding. Mathematical definition and properties of information. Source coding theorem, lossless data compression and optimal lossless coding. Structural properties of natural languages. Information characteristics of images. Cryptography, data encryption. Noisy communication channels, channel coding theorem, multiple access channels. Error detection and error correction. Cyclic, binary block and convolutional codes, capacity-approaching codes. Gaussian noise, time-varying channels. Unified theory of information with applications to other sciences.

General Competencies

After finishing this course students will be able to understand and apply fundamentals of information theory. They will know principles of information coding as well as properties of communication channels. They will gain skills required for modelling and analysis of optimal, error-detecting and error-correcting codes. They will develop learning skills necessary to continue to undertake further study in the field of information theory and coding theory.

Learning Outcomes

  1. identify information, coding and communication problems
  2. explain coding and compression methods and information limits
  3. apply accepted knowledge to real systems analysis
  4. analyze complex information and communication systems
  5. explain phenomens in different area of science
  6. estimate performances of different information and communication systems

Forms of Teaching

Lectures

First cycle (seven weeks): lectures then Midterm exam, and Second cycle (six veeks): lectures and Final exam. Lecture duration: 3 hours per week.

Exams

Midterm exam: 8th week; Final exam: 15th week.

Laboratory Work

Programming and testing basic coding algorithms.

Consultations

Consultation every week in terms defined by lecturer.

Acquisition of Skills

Programming skills: students write programs for different coding algorithms.

Grading Method

Continuous Assessment Exam
Type Threshold Percent of Grade Comment: Percent of Grade
Mid Term Exam: Written 5 % 50 % 0 %
Final Exam: Written 5 % 50 %
Exam: Written 0 % 100 %
Comment:

Although laboratory exercises and homeworks have 0% of a total share on this course, students are obliged to successfully complete both of these activities. In other words, they are precondition, i.e. it is impossible to pass the final exam or any further exam without having successfully completed laboratory exercises and homeworks.

Week by Week Schedule

  1. Introduction. Information, communication and processing. Model of communication system.
  2. Discrete communication system, probability distributions and measures of information, entropy and mutual information. Communication channels, discrete memoryless noisy communication channels, channel capacity.
  3. Information sources, information content of discret information source, information redundancy, data compression and optimal coding.
  4. Sources with memory. Shannon-Fano, Huff,man, aritmetic coding and dictionary methods (Lempel-Ziv algorithms).
  5. Source coding: quantization, undersampling, transform methods, prediction method.
  6. Introduction to block codes: Hamming distance, perfect codes, parity check coding.
  7. Linear binary block codes: parity check matrix, syndrome decoding.
  8. Midterm examination.
  9. Hamming and cyclic codes. Linear BCH block codes. R-S codes.
  10. Convolutional codes. Viterbi algorithm. Turbo codes.
  11. Signals: deterministic, random, noise, spectar. Continous channel.
  12. Sampling and quantization.
  13. Capacity of band-limited channel.
  14. Introduction to media coding.
  15. Final examination.

Study Programmes

Control Engineering and Automation -> Electrical Engineering and Information Technology (Module)

Electrical Power Engineering -> Electrical Engineering and Information Technology (Module)

Electronic and Computer Engineering -> Electrical Engineering and Information Technology (Module)

Electronics -> Electrical Engineering and Information Technology (Module)

Wireless Technologies -> Electrical Engineering and Information Technology (Module)

Information Processing -> Computing (Module)

Software Engineering and Information Systems -> Computing (Module)

Computer Engineering -> Computing (Module)

Computer Science -> Computing (Module)

Telecommunication and Informatics -> Computing (Module)

Literature

I.S. Pandžić, A. Bažant, Ž. Ilić, Z. Vrdoljak, M. Kos, V. Sinković (2009.), Uvod u teoriju informacije i kodiranje, 2. izd., Element, Zagreb
V. Sinković (1997.), Informacija, simbolika i semantika, Školska knjiga
R.E. Hamming (1986.), Coding and Information Theory. 2nd ed., Prentice-Hall. Englewood Cliffs, New Jersey
R. Togneri, C.J.S. deSilva (2003.), Fundamentals of Information Theory and Coding Design, Cjapman & Hall/CRC
F.M. Reza (1994.), An Introduction to Information Theory, Dover, New York
(.), Pandžić, I. S. Bažant, A. Ilić, Ž. Vrdoljak, Z. Kos, M. Sinković, V. Uvod u teoriju informacije i kodiranje. Element, 2007.,

Grading System

4 ECTS
L0 English Level
L1 e-Learning
45 Lecturers
0 Exercises
15 Laboratory exercises

Grading

85 Excellent
70 Very Good
55 Good
40 Acceptable