**ECE 534: Elements of Information Theory**

University of Illinois at Chicago, ECE

Fall 2018

CRN 26574**Instructor:** Natasha Devroye, devroye@uic.edu **Course coordinates:** Monday, Wednesday, Friday from 2-2:50pm in Lincoln Hall (LH) 210**Office hours:** Mondays and Wednesdays from 3-4pm and Fridays 10-11am in SEO 1039, or by appointment (time permitting)

Welcome to ECE 534! This course is a graduate-level introduction to information theory, an elegant mathematical theory with direct impact on our life in the ``information age.'' Specifically, information theory allows us to quantify the fundamental limits of information communication and compression (to what size can we compress that image with and without distortion, how fast can we communicate ``reliably'' over a noisy channel?). Information theory has found its applications in many areas including statistics, computer science, biology, economics, as well as electrical engineering. We will touch upon many aspects of information theory, but the focus of this course will be on the direct applications of information theory in digital communications. Nonetheless, information theory teaches one a new and intuitive way of reasoning about problems, rendering it useful even to non-communication-engineering majors.

**Course Textbook:** Elements of Information Theory, by T. Cover and J. Thomas, Wiley 2006 **2nd edition!**) Free PDFs may or may not be available online though a search.

Other useful references:

Shannon's original 1948 paper, A Mathematical Theory of Communication.

A great list of papers related to network information theory are here.

The IEEE Information Theory Society web-page.

Recent (often just submitted for publication) results are found on arXiv.

Excellent notes on network / multi-user information theory

Bob Gray's free online book called Entropy and Information Theory.

David MacKay 's free online book called Information Theory, Pattern Recognition and Neural Networks.

Costello and Forney's excellent survey ``Channel Coding: The Road to Channel Capacity''

**Topics:** Entropy and differential entropy, mutual information; data compression; channel capacity, the Gaussian channel; rate-distortion; universal-source-coding; network information theory. Contemporary examples and research topics.

**Grading:** Project: 25%, Exam 1: 25%, Exam 2: 25%, Exam 3: 25%. I will use Blackboard to report your grades (for you to see and make sure I entered them correctly as well)

**Homework:**

Will NOT be graded. This is a graduate level class and one of my goals is to encourage self-motivation, self-discipline and self-study. I will post homeworks each Wednesday, due the next Wednesday, when I will post solutions online. Please check your own solution. I **strongly** encourage you to carefully attempt the homework on your own as if it were graded. This will provide excellent practice for the three exams. As practice for your project, I encourage you to do some of your homeworks in Latex. This is excellent practice for typsetting math, which I promise you you will find useful in the future. No more battling equations in Word ever again! Here is a template you can use for your homeworks. Note that the project must be done in latex, so might as well start practicing early.

**Exams:** There will be 3 midterms of equal grade weight (the last one on the last day of class during class time, NO final during finals week). For midterm 1, you may have one 8.5x11 double-sided sheet which you can fill with anything you like. For midterm 2, you may have two 8.5x11 double-sided sheets, and for midterm 3 you may have 3 such sheets. No other books, notes or calculators.

Past exams (I only taught it from 2009 onwards). Solutions are provided. Spring 2006, Spring 2007, Fall 2009, Fall 2009 midterm 2, Fall 2010, Fall 2010 midterm 2
Fall 2013, Fall 2013 midterm 2,
Fall 2017 midterm 1,
Fall 2017 midterm 2
Fall 2018 midterm 1
Fall 2018 midterm 2
Fall 2017 final exam (no solutions)
Fall 2017 final exam (with solutions)

**Project:** The project, to be done individually, will consist of 1) a short, professional, well-written report and 2) a set of 20 accompanying slides. The goal will be to explore contemporary research topics in the area of information theory that are not covered in class. Each student will be asked to select one paper from the list of Best Paper Award winners of the IEEE Transactions on Information Theory, read it in detail, and summarize it in a 1) 5 page survey and 2) a set of 20 slides. Both the 5 page summary and the 20 slides should be written / made with your fellow students as the target audience -- i.e. someone with some knowledge of the basics of information theory but not an expert on the subject of your paper. Usually papers are written with experts in mind, so the challenge in this project will be to first understand the paper (this may take considerable effort), and present it in a way that is accessible to a student of information theory. Each student will be asked to read the work and slides of two other students and to turn in a "graded" copy of the two assigned papers -- you will review the paper and write 1 paragraph summary of the other student's work, give 3 strengths (and why), 3 weaknesses (and why), and any other general comments you have. Feel free to write directly on a printed copy of the paper, and remember to also write your own name on each paper that you graded! Please see Additional materials for details on the project.

**Religious holidays.** Students who wish to observe religious holidays should notify the instructor by the tenth day of the semester of the date when they will be absent unless the religious holiday is observed on or before the tenth day of the semester. In such cases, the students should notify the instructor at least five days in advance of the date when he/she will be absent. Every reasonable effort will be made to honor the request.

**Plagiarism and student dishonesty.** Plagiarism in the project will not be tolerated. Cheating of any form during the three exams will not be tolerated. All forms of dishonesty will be dealt with according to UIC's Student Disciplinary policy here.

Topic revision: r37 - 2018-08-21 - 21:15:52 - Main.devroye

Copyright 2016 The Board of Trustees of the University of Illinois.webmaster@cs.uic.edu |
WISEST Helping Women Faculty Advance Funded by NSF |