Information Theory
Course Description
General Competencies
After finishing this course students will be able to understand and apply fundamentals of information theory. They will know principles of information coding as well as properties of communication channels. They will gain skills required for modelling and analysis of optimal, error-detecting and error-correcting codes. They will develop learning skills necessary to continue to undertake further study in the field of information theory and coding theory.
Learning Outcomes
- identify information, coding and communication problems
- explain coding and compression methods and information limits
- apply accepted knowledge to real systems analysis
- analyze complex information and communication systems
- explain phenomens in different area of science
- estimate performances of different information and communication systems
Forms of Teaching
First cycle (seven weeks): lectures then Midterm exam, and Second cycle (six veeks): lectures and Final exam. Lecture duration: 3 hours per week.
ExamsMidterm exam: 8th week; Final exam: 15th week.
Laboratory WorkProgramming and testing basic coding algorithms.
ConsultationsConsultation every week in terms defined by lecturer.
Acquisition of SkillsProgramming skills: students write programs for different coding algorithms.
Grading Method
| Continuous Assessment | Exam | |||||
|---|---|---|---|---|---|---|
| Type | Threshold | Percent of Grade | Threshold | Percent of Grade | ||
| Mid Term Exam: Written | 5 % | 50 % | 0 % | |||
| Final Exam: Written | 5 % | 50 % | ||||
| Exam: Written | 0 % | 100 % | ||||
Comment:
Although laboratory exercises and homeworks have 0% of a total share on this course, students are obliged to successfully complete both of these activities. In other words, they are precondition, i.e. it is impossible to pass the final exam or any further exam without having successfully completed laboratory exercises and homeworks.
Week by Week Schedule
- Introduction. Information, communication and processing. Model of communication system.
- Discrete communication system, probability distributions and measures of information, entropy and mutual information. Communication channels, discrete memoryless noisy communication channels, channel capacity.
- Information sources, information content of discret information source, information redundancy, data compression and optimal coding.
- Sources with memory. Shannon-Fano, Huff,man, aritmetic coding and dictionary methods (Lempel-Ziv algorithms).
- Source coding: quantization, undersampling, transform methods, prediction method.
- Introduction to block codes: Hamming distance, perfect codes, parity check coding.
- Linear binary block codes: parity check matrix, syndrome decoding.
- Midterm examination.
- Hamming and cyclic codes. Linear BCH block codes. R-S codes.
- Convolutional codes. Viterbi algorithm. Turbo codes.
- Signals: deterministic, random, noise, spectar. Continous channel.
- Sampling and quantization.
- Capacity of band-limited channel.
- Introduction to media coding.
- Final examination.
Study Programmes
University undergraduate
Prerequisites
Literature
I.S. Pandžić, A. Bažant, Ž. Ilić, Z. Vrdoljak, M. Kos, V. Sinković (2009.), Uvod u teoriju informacije i kodiranje, 2. izd., Element, Zagreb
R.E. Hamming (1986.), Coding and Information Theory. 2nd ed., Prentice-Hall. Englewood Cliffs, New Jersey
R. Togneri, C.J.S. deSilva (2003.), Fundamentals of Information Theory and Coding Design, Cjapman & Hall/CRC
F.M. Reza (1994.), An Introduction to Information Theory, Dover, New York
