**ECE 534: Elements of Information Theory**

University of Illinois at Chicago, ECE

Fall 2017

CRN 26574**Instructor:** Natasha Devroye, devroye@uic.edu **Course coordinates:** Monday, Wednesday from 4-5:15pm in Lincoln Hall (LH) 104**Office hours:** Tuesdays 1-2pm, and Fridays 10-11am in SEO 1039, or by appointment (time permitting)

Welcome to ECE 534! This course is a graduate-level introduction to information theory, an elegant mathematical theory with direct impact on our life in the ``information age.'' Specifically, information theory allows us to quantify the fundamental limits of information communication and compression (to what size can we compress that image with and without distortion, how fast can we communicate ``reliably'' over a noisy channel?). Information theory has found its applications in many areas including statistics, computer science, biology, economics, as well as electrical engineering. We will touch upon many aspects of information theory, but the focus of this course will be on the direct applications of information theory in digital communications. Nonetheless, information theory teaches one a new and intuitive way of reasoning about problems, rendering it useful even to non-communication-engineering majors.

**Course Textbook:** Elements of Information Theory, by T. Cover and J. Thomas, Wiley 2006 **2nd edition!**) Free PDFs may or may not be available online though a search.

Other useful references:

Shannon's original 1948 paper, A Mathematical Theory of Communication.

A great list of papers related to network information theory are here.

The IEEE Information Theory Society web-page.

Recent (often just submitted for publication) results are found on arXiv.

Excellent notes on network / multi-user information theory

Bob Gray's free online book called Entropy and Information Theory.

David MacKay 's free online book called Information Theory, Pattern Recognition and Neural Networks.

Costello and Forney's excellent survey ``Channel Coding: The Road to Channel Capacity''

**Topics:** Entropy and differential entropy, mutual information; data compression; channel capacity, the Gaussian channel; rate-distortion; universal-source-coding; network information theory. Contemporary examples and research topics.

**Grading:** Project: 25%, Exam 1: 25%, Exam 2: 25%, Exam 3: 25%. I will use Blackboard to report your grades (for you to see and make sure I entered them correctly as well)

**Homework:**

Will NOT be graded. This is a graduate level class and one of my goals is to encourage self-motivation, self-discipline and self-study. I will post homeworks each Wednesday, due the next Wednesday, when I will post solutions online. Please check your own solution. I **strongly** encourage you to carefully attempt the homework on your own as if it were graded. This will provide excellent practice for the three exams. As practice for your project, I ecnourage you to do some of your homeworks in Latex. This is excellent practice for typsetting math, which I promise you you will find useful in the future. No more battling equations in Word ever again! Here is a template you can use for your homeworks. Note that the project must be done in latex, so might as well start practicing early.

**Exams:** There will be 3 midterms of equal grade weight (the last one on the last day of class during class time, NO final during finals week). For midterm 1, you may have one 8.5x11 double-sided sheet which you can fill with anything you like. For midterm 2, you may have two 8.5x11 double-sided sheets, and for midterm 3 you may have 3 such sheets. No other books, notes or calculators.

Past exams (I only taught it from 2009 onwards). Solutions for 2009 and 2010 and provided. Spring 2006, Spring 2007, Fall 2009, Fall 2009 midterm 2, Fall 2010, Fall 2010 midterm 2
Fall 2013, Fall 2013 midterm 2,
Fall 2017 midterm 1,
Fall 2017 midterm 2

**Project:** The project, to be done individually, will consist of a short, professional, well-written report. The goal will be to explore contemporary research topics in the area of information theory that are not covered in class. Pick (or suggest) a topic of interest to you and provide a comprehensive treatment of it: introduce the problem/topic, survey what has been done by whom on the topic (we expect many citations to relevant journal and conference papers. The most relevant journal is the IEEE Transactions on Information Theory, while some relevant conferences are the International Symposium on Information Theory (ISIT, held once a year), Allerton (held in UIUC), Information Theory Workshop (held twice a year). ) After summarizing past work on the topic in your own words (plagiarism will NOT be tolerated), state some open problems in the area. Each student will be asked to read the work of two other students and to turn in a "graded" copy of the two assigned papers -- you will review the paper and write 1 paragraph summary of the other student's work, give 3 strengths (and why), 3 weaknesses (and why), and any other general comments you have. Feel free to write directly on a printed copy of the paper, and remember to also write your own name on each paper that you graded! Please see Additional materials for details on the project.

**Religious holidays.** Students who wish to observe religious holidays should notify the instructor by the tenth day of the semester of the date when they will be absent unless the religious holiday is observed on or before the tenth day of the semester. In such cases, the students should notify the instructor at least five days in advance of the date when he/she will be absent. Every reasonable effort will be made to honor the request.

**Plagiarism and student dishonesty.** Plagiarism in the project will not be tolerated. Cheating of any form during the three exams will not be tolerated. All forms of dishonesty will be dealt with according to UIC's Student Disciplinary policy here.

Topic revision: r36 - 2017-11-03 - 03:09:33 - Main.devroye

Copyright 2016 The Board of Trustees of the University of Illinois.webmaster@cs.uic.edu |
WISEST Helping Women Faculty Advance Funded by NSF |