CS 505 - Computability and Complexity Theory

Welcome to the class! I’m excited to have you. Throughout this website, you’ll find all the relevant information needed for the course.

On this page, I’ll post important announcements, as well as a changelog. If you have any questions, please feel free to reach out to me.

Announcements

• [May 8, 2025] All grades except for the Final Project write-up have been posted to Blackboard. You also should have received feedback in Blackboard for your project presentations (peer feedback and my own feedback). If you have any questions, contact me as soon as possible. Sample solutions for Homework 5 have been posted. Lecture 20 typed notes posted.
• [April 28, 2025] Final Project write-ups are now required to be submitted via Blackboard. This has been noted on the Final Project page, as well as the PDF included there. Typed notes for Lecture 19 posted.
• [April 26, 2025] Handwritten notes for Lecture 23 and Lecture 24 posted. Additional resources for the PCP theorem posted; see Resources.
• [April 21, 2025] Homework 5 posted. Sample solutions for Homework 3 and Homework 4 posted. Make-up office hours will be held this week: April 23, 2025, at 2:00pm (in-person or via Zoom).
• [April 11, 2025] Lecture 21 and Lecture 22 handwritten notes posted. Recorded lectures for Crypto and Complexity Theory, along with handwritten notes for those lectures, posted. Office Hours next week are cancelled due to travel on my end; I will still be available via email or Piazza.
• [April 7, 2025] Lecture 19 and Lecture 20 handwritten notes posted. Schedule updated. Group orderings for the Final Project presentations posted. Please see the webpage for the schedule (i.e., when your group is presenting), along with how the ordering was decided.
• [March 31, 2025] Homework 4 posted. Grades for Homework 3 posted on Gradescope. Schedule updated; no class the week of April 14, instead there will be recorded lectures.
• [March 30, 2025] Typed notes for Lecture 17 and Lecture 18 posted.
• [March 26, 2025] Handwritten notes for Lecture 17 and Lecture 18 posted. Typed notes for Lecture 16 posted.
• [March 17, 2025] Typed notes for Lecture 15 posted.
• [March 16, 2025] Handwritten notes for Lecture 15 and Lecture 16 posted. Typed notes for Lecture 14 posted. Midterm grades posted to Gradescope.
• [March 11, 2025] Final Project information posted. Midterm sample solutions posted to Piazza. Schedule updated with Final Project information.
• [March 3, 2025] Homework 3 posted. Homework 2 sample solutions posted.
• [March 1, 2025] Typed notes for Lecture 13 posted.
• [February 28, 2025] Lecture 11 and Lecture 12 typed notes posted. Handwritten notes for Lecture 13 and Lecture 14 posted.
• [February 23, 2025] Lecture 9 and Lecture 10 typed notes posted. Handwritten notes for Lecture 11 and Lecture 12 posted.
• [February 18, 2025] Schedule updated.
• [February 17, 2025] Lecture 9 and Lecture 10 handwritten notes posted. Homework 1 grades released on Gradescope. Sample solutions for Homework 1 posted.
• [February 13, 2025] Homework 2 posted.
• [February 10, 2025] Lecture 7 and Lecture 8 handwritten notes posted.
• [February 04, 2025] Lecture 5 and Lecture 6 posted.
• [February 03, 2025] Handwritten notes for Lectures 2-6 posted (there are no notes for Lecture 1 since it was written on the board). Lecture 5 and Lecture 6 typed notes will be posted tonight or tomorrow. Zoom link added to office hours (see Important Info below and the Syllabus).
• [January 31, 2025] Homework 1 updated to reflect schedule changes. Problem 5 has been changed. If you do not see this change, you may need to force refresh the course website in your browser, or open it in an incognito window. Schedule updated.
• [January 27, 2025] Lecture 3 and Lecture 4 posted.
• [January 21, 2025] Homework 1 posted. Homework collaboration policy posted.
• [January 20, 2025] In-class lecture on January 21, 2025, is cancelled due to the weather. Class will be held on Zoom. Please check your email for the Zoom link.
• [January 20, 2025] Since today is a holiday and the weather is very cold, in-person office hours are optional. A Zoom link will be sent for office hours today (check your email).
• [January 19, 2025] Lecture 1 and Lecture 2 have been posted. Office hours have been updated (see above or the syllabus).

Important Info

Instructor: Alexander R. Block
Email: arblock [at] uic [dot] edu

Drop-in Office Hours

• Time: Mondays, 2–3pm (or by appointment)
• Location: SEO 1216 or Zoom
  • Zoom link (UIC login required): https://uic.zoom.us/j/81305503904?pwd=w4N9MLSrL4M7sny3zQmYyWT6MQgkiG.1

Course Modality and Schedule: In-person only, BSB 289, 2:00 pm - 3:15 pm, Tuesday & Thursday.

Important Links

• Blackboard
• Gradescope
• Piazza
• Syllabus
• Schedule

Changelog

• [January 20, 2025] Announcements moved to the top of this page. Links to lecture notes added to the schedule and resources.
• [January 19, 2025] Added important info to this page. Announcement on this day posted.
• [January 13, 2025] Syllabus updated.
• [January 06, 2025] Website available.

Syllabus

Instructor and Course Details

Instructor: Alexander R. Block
Email: arblock [at] uic [dot] edu

Drop-in Office Hours

• Time: Mondays, 2–3pm (or by appointment)
• Location: SEO 1216 or Zoom
  • Zoom link (UIC login required): https://uic.zoom.us/j/81305503904?pwd=w4N9MLSrL4M7sny3zQmYyWT6MQgkiG.1

Course Modality and Schedule: In-person only, BSB 289, 2:00 pm - 3:15 pm, Tuesday & Thursday.

Blackboard: https://uic.blackboard.com/ultra/courses/_279721_1/cl/outline

Gradescope: https://www.gradescope.com/courses/942742

Piazza: https://piazza.com/uic/spring2025/cs505

Course Announcements

Course information will primarily be conveyed using this website (see here). Course discussion will happen on Piazza. All course assignments and grades will be collected and returned through Gradescope. I will also send email notifications to the class with announcements.

You are responsible for checking this website and emails for any and all updates and information regarding the course, including homework assignments and schedule changes. You are also responsible for keeping up to date on Piazza for any corrections and/or clarifications regarding assignments, or other important information.

Blackboard will be used sparingly in this course, primarily for Homework, Midterm, Final Project, and Final Grades. For all technical questions about Blackboard, email the Learning Technology Solutions team at LTS@uic.edu.

Communication Expectations

Students are responsible for all information instructors send to your UIC email. Faculty messages should be regularly monitored and read in a timely fashion.

Please use Piazza private messages shared with the instructors (not just the professor or TA by name) if you wish to communicate with us directly. Please only use email for something that explicitly should be kept private only to that person.

Please email me if you face an unexpected situation that may impede your attendance, participation in class and exam sessions, or timely completion of assignments.

Course Information

CS 505 is a graduate-level introductory course to Computability and Complexity Theory. You will be expected to read, understand, and write formal (i.e., mathematic) proofs.

Prerequisites: For UIC students, CS 305 is listed as a prerequisite. However, most (if not all) topics covered will be self-contained in this course. As stated above (but in another way), you will need mathematical maturity to succeed in this course. That is, you should be comfortable answering questions of the form “prove or disprove the following statements.” If you are able to do this, then this course is for you; if you struggle with these types of problem, then this course may not be for you.

Brief list of topics to be covered (subject to change)

• Turing machines and their equivalent computational models
• Languages and Decidability
• The class $\mathbf{P}$
• The class $\mathbf{NP}$, and $\mathbf{NP}$-Completeness
• Randomized Computations
• Space Complexity
• Interactive proofs; $\mathbf{IP}=\mathbf{PSPACE}$
• (Time-dependent) Cryptography and Complexity Theory

Required and Recommended Course Material: No textbook is required for this course. Lectures will have all relevant information.

I will be closely following the book Computational Complexity: A Modern Approach by Aurora and Barak. I will additionally use material from Introduction to the Theory of Computation by Michael Sipser, Mathematics and Computation by Avi Widgerson, and Proofs, Arguments, and Zero-Knowledge by Justin Thaler.

I will give suggested additional readings for each lecture in relevant material freely available online.

Course Copyright: Please protect the copyright integrity of all course materials and content. Please do not upload course materials not created by you onto third-party websites or share content with anyone not enrolled in our course.

Course Policies and Classroom Expectations

Grading Policies & Point Breakdown

Grades will be curved based on an aggregate course score and are not defined ahead of time. The score cut-offs for A, B, C, etc., will be set after the end of the course.

The course will have the following grade breakdown:

Final Grade Assignments

My goal is to ensure that the assessment of your learning in this course is comprehensive, fair, and equitable. Your grade in the class will be based on the number of points you earn out of the total number of points possible, and is not based on your rank relative to other students. There are no set limits to the number of grades given (e.g., everyone can get an A if everyone does well). If the class average is at least 75%, then assigned letter grades will be based on a straight scale with the following thresholds:

If the class mean is less than 75%, then this scale will be adjusted to compensate.

Under no circumstances will grades be adjusted down (except in cases of course policy violation). You can use this straight grading scale as an indicator of your minimum grade in the course at any time during the course. You should keep track of your own points so that at any time during the semester you may calculate your minimum grade based on the total number of points possible at that particular time. If and when, for any reason, you have concerns about your grade in the course, please email me to schedule a time for you to speak with me so that we can discuss study techniques or alternative strategies to help you.

Regrade Policy

You are allowed to request one single regrade per homework assignment/non-final exam. Moreover, with every regrade request, you must submit the following information:

• Which problems you are requesting a regrade for; and
• The exact reason you are requesting a regrade.

I will be strict with this policy to ensure there are no frivolous regrade requests; i.e., do not request a regrade to try and argue for more points. You must have a specific and articulate reason for why you believe something was graded incorrectly. Finally, note that any regrade request can result in a score reduction if additional errors are discovered.

Homework Late Policy

All homework assignments are due by the beginning of class (2:00pm Central Time) on the day they are due. You may submit homework late, with a 25% point reduction per day of being late. On the fourth day of being late, you will receive zero points on the assignment, but it will still be graded in order to give you feedback.

In-class Participation

All course material will be primarily given through in-class lectures. However, as this is a graduate course, I will not require you to attend lecture, but it will be highly encouraged. You are responsible for submitting all assignments, taking the midterm exam, and completing the final project.

Note that though you are not required to attend lecture, I will not answer questions in office hours of the form “can you explain X?” when X was explained in class. By not attending lecture, you do not then get to ask me to teach you the lecture material in office hours.

Evaluation

Homework

There will be 4-5 homework assignments in this course, depending on our progress through the semester. Each homework assignment will be weighted equally when calculating your final grade. See here for more information about completing homework assignments.

Note that your lowest homework score will be automatically dropped from your final grade.

Midterm Exam

There will be an in-class midterm exam, covering topics from (roughly) the first half of the course. The tentative date for the midterm exam is Thursday, March 6, 2025, with an in-class review planned for the previous lecture on Tuesday, March 4, 2025. Please plan to attend class on this day for the midterm exam. I will notify all students once the midterm exam date is finalized, and will do so as soon as possible.

Final Project

In place of a final exam, there will instead be a final project. It will consist of both a written portion and an in-class presentation. Your grade will be assigned based on the written portion, the in-class presentation, and peer-evaluations as well. I will give more details about the final project as we get closer to the midpoint of the semester.

Academic Integrity

Consulting with your classmates on assignments is encouraged, except where noted. However, turn-ins are individual, and copying proofs from your classmates is considered plagiarism. You should never look at someone else’s writing, or show someone else your writing. Either of these actions are considered academic dishonesty (cheating) and will be prosecuted as such.

To avoid suspicion of plagiarism, you must specify your sources together with all turned-in materials. List classmates you discussed your homework with and webpages/resources from which you got inspiration and help. Plagiarism and cheating, as in copying the work of others, paying others to do your work, etc., is obviously prohibited, and will be reported (this includes asking questions and copying answers from forums such as Stack Overflow and Reddit).

I report all suspected academic integrity violations to the dean of students. If it is your first time, the dean of students may provide the option to informally resolve the case – this means the student agrees that my description of what happened is accurate, and the only repercussions on an institutional level are that it is noted that this happened in your internal, UIC files (i.e., the dean of students can see that this happened, but no professors or other people can, and it is not in your transcript). If this is not your first academic integrity violation in any of your classes, a formal hearing is held and the dean of students decides on the institutional consequences. After multiple instances of academic integrity violations, students may be suspended or expelled. For all cases, the student has the option to go through a formal hearing if they believe that they did not actually violate the academic integrity policy. If the dean of students agrees that they did not, then I revert their grade back to the original grade, and the matter is resolved.

If you are found responsible for violating the academic integrity policy, the penalty can range from receiving a zero on the assignment in question, receiving a grade deduction, or receiving an F in the class, depending on the severity of the violation.

As a student and member of the UIC community, you are expected to adhere to the Community Standards of academic integrity, accountability, and respect. Please review the UIC Student Disciplinary Policy for additional information.

GenAI

Since this course will be rigorous in formal (i.e., mathematical proofs), the use of GenAI is NOT allowed. As we will see, there are computational tasks that are impossible in any computational model, thus it is highly likely (and, indeed, expectd) that GenAI would answer any such questions incorrectly.

Failure to adhere to this policy will result in the following consequences:

• First use: You will lose 50% of the points available on the assignment.
• Second use: You will fail the assignment.
• Third use: You will fail the course.

Accommodations

Disability Accommodation Procedures

UIC is committed to full inclusion and participation of people with disabilities in all aspects of university life. If you face or anticipate disability-related barriers while at UIC, please connect with the Disability Resource Center (DRC) at drc.uic.edu, via email at drc@uic.edu, or call (312) 413-2183 to create a plan for reasonable accommodations. To receive accommodations, you will need to disclose the disability to the DRC, complete an interactive registration process with the DRC, and provide me with a Letter of Accommodation (LOA). Upon receipt of an LOA, I will gladly work with you and the DRC to implement approved accommodations.

Religious Accommodations

Following campus policy, if you wish to observe religious holidays, you must notify me by the tenth day of the semester. If the religious holiday is observed on or before the tenth day of the semester, you must notify me at least five days before you will be absent. Please submit this form by email with the subject heading: “[CS 505] YOUR NAME: Requesting Religious Accommodation.”

Classroom Environment

Inclusive Community

UIC values diversity and inclusion. Regardless of age, disability, ethnicity, race, gender, gender identity, sexual orientation, socioeconomic status, geographic background, religion, political ideology, language, or culture, we expect all members of this class to contribute to a respectful, welcoming, and inclusive environment for every other member of our class. If aspects of this course result in barriers to your inclusion, engagement, accurate assessment, or achievement, please notify me as soon as possible.

Name and Pronoun Use

If your name does not match the name on my class roster, please let me know as soon as possible. My pronouns are [she/her; he/him; they/them]. I welcome your pronouns if you would like to share them with me. For more information about pronouns, see this page: https://www.mypronouns.org/what-and-why.

Community Agreement/Classroom Conduct Policy

• Be present by removing yourself from distractions, whether they be phone notifications, entire devices, conversations, or anything else.
• Be respectful of the learning space and community. For example, no side conversations or unnecessary disruptions.
• Use preferred names and gender pronouns.
• Assume goodwill in all interactions, even in disagreement.
• Facilitate dialogue and value the free and safe exchange of ideas.
• Try not to make assumptions, have an open mind, seek to understand, and not judge.
• Approach discussion, challenges, and different perspectives as an opportunity to “think out loud,” learn something new, and understand the concepts or experiences that guide other people’s thinking.
• Debate the concepts, not the person.
• Be gracious and open to change when your ideas, arguments, or positions do not work or are proven wrong.
• Be willing to work together and share helpful study strategies.
• Be mindful of one another’s privacy, and do not invite outsiders into our classroom.

Furthermore, our class (in person and online) will follow the CS Code of Conduct. If you are not adhering to our course norms, a case of behavior misconduct will be submitted to the Dean of Students and to the Director of Undergraduate Studies in the department of Computer Science. If you are not adhering to our course norms, you will not get full credit for your work in this class. For extreme cases of violating the course norms, credit for the course will not be given.

Student Parents

I know well how exhausting balancing school, childcare, and work can be. I would like to help support you and accommodate your family’s needs, so please don’t keep me in the dark. I hope you will feel safe disclosing your student-parent status to me so that I can help you anticipate and solve problems in a way that makes you feel supported. Unforeseen disruptions in childcare often put parents in the position of having to choose between missing classes to stay home with a child or leaving them with a less desirable backup arrangement. While this is not meant to be a long-term childcare solution, occasionally bringing a child to class in order to cover gaps in care is perfectly acceptable. If your baby or young child comes to class with you, please plan to sit close to the door so that you can step outside without disrupting learning for other students if your child needs special attention. Non-parents in the class, please reserve seats near the door for your parenting classmates or others who may need to step out briefly.

Academic Success, Wellness, and Safety

We all need the help and the support of our UIC community. Please visit my drop-in hours for course consultation and other academic or research topics. For additional assistance, please contact your assigned college advisor and visit the support services available to all UIC students.

Academic Success

• UIC Tutoring Resources
• College of Engineering tutoring program
• Equity and Inclusion in Engineering Program
• UIC Library and UIC Library Research Guides.
• Offices supporting the UIC Undergraduate Experience and Academic Programs.
• Student Guide for Information Technology
• First-at-LAS Academic Success Program, focusing on LAS first-generation students.

Wellness

• Counseling Services : You may seek free and confidential services from the Counseling Center at https://counseling.uic.edu/.

• Access U&I Care Program for assistance with personal hardships.

• Campus Advocacy Network : Under Title IX, you have the right to an education that is free from any form of gender-based violence or discrimination. To make a report, email TitleIX@uic.edu. For more information or confidential victim services and advocacy, visit UIC’s Campus Advocacy Network at http://can.uic.edu/.

Safety

• UIC Safe App—PLEASE DOWNLOAD FOR YOUR SAFETY!
• UIC Safety Tips and Resources
• Night Ride
• Emergency Communications: By dialing 5-5555 from a campus phone, you can summon the Police or Fire for any on-campus emergency. You may also set up the complete number, (312) 355-5555, on speed dial on your cell phone.

Schedule

This schedule is tentative and subject to change. Any changes will be announced.

Resources

This page will be updated throughout the semester with new resources as I find them. If you have found a particularly useful resource, feel free to let me know and I will gladly add them to the resources below.

Lecture Notes

• Lecture 1
• Lecture 2
• Lecture 3
• Lecture 4
• Lecture 5
• Lecture 6
• Lecture 7
• Lecture 8
• Lecture 9
• Lecture 10
• Lecture 11
• Lecture 12
• Lecture 13
• Lecture 14
• Lecture 15
• Lecture 16
• Lecture 17
• Lecture 18
• Lecture 19
• Lecture 20
• Lecture 21
• Lecture 22
• Lecture 23
• Lecture 24
• Crypto and Complexity Theory

Books

• Computational Complexity: A Modern Approach (Draft) by Sanjeev Arora and Boaz Barak
• Introduction to the Theory of Computation¹ by Michael Sipser
• Proofs, Arguments, and Zero-knowledge by Justin Thaler
• Mathematics and Computation by Avi Widgerson

Other Lecture Notes and Videos

• Michael Sipser’s Theory of Computation Course at MIT

PCP Theorem

• Notes from Jon Katz: first lecture and second lecture. All his other notes from this course: website.
• Dana Moshkovitz’s Tale of the PCP Theorem.
• Irit Dinur’s alternative proof of the PCP Theorem: The PCP Theorem by Gap Amplification.
• PCP Theorem Course by Irit Dinur and Dana Moshkovitz: Probabilistically Checkable Proofs.

1. You may be able to find the book online for free. I cannot confirm or deny the availability. Supplemental readings may be given from this book and will be paired with the appropriate video lecture from Michael Sipser’s course below. ↩

Homework

This page will be updated with homework assignments as they become available throughout the semester. All homework assignments will be due before the start of class. That is, by no later than 2:00pm Central US Time.

All homework is required to be typeset in $\text{LATEX}$. Included with each assignment is a .pdf file of the assignment, along with a .zip folder with the $\text{LATEX}$ source for you to use to complete the assignment.

All homework is required to be submitted through Gradescope. For submission, you will simply need to upload the .pdf file of your assignment.

You will likely not be able to answer all the questions in a given homework when it is released. My intention is to spread out the material you need to complete each homework over several lectures. This will allow you to either (a) do the homework problems incrementally as we cover the relevant material in class; (b) wait until all lectures are completed then do the homework all at once; or (c) finish the homework as soon as it’s assigned by reading ahead in other resources.

Homework 1

Date Assigned: January 21, 2025.
Due Date: February 6, 2025, no later than 2:00pm Central Time.
Updated: January 31, 2025 (see Announcements).
PDF file: homework-1.pdf (SHA256: bfcb1bb09bcb433c2e22b462bd1a9ea62d227767936db3099261afcfa3124a37)
Source files: homework-1.zip (SHA256: 73fedc2f3141d71314fe451f2732bf6a7c1dcbf14df8d14c0de05d24289d69d4)
Sample solutions: homework-1-solutions.pdf

Homework 2

Date Assigned: February 13, 2025.
Due Date: February 27, 2025, no later than 2:00pm Central Time.
PDF file: homework-2.pdf (SHA256: 38ade5b4a4b5abd0070c33953861e201253efbcd5fc32dc904abc1314a4a1e54)
Source files: homework-2.zip (SHA256: 5e55f481dcc61aa5489875fc1e00084b1b02081c98fb6657d74551c543276b8b)
Sample solutions: homework-2-solutions.pdf

Homework 3

Date Assigned: March 4, 2025.
Due Date: March 20, 2025, no later than 2:00pm Central Time.
PDF file: homework-3.pdf (SHA256: 276a2dea7f34c264b355968bc90c7052889816082c7c1c179566c9b25537d980)
Source files: homework-3.zip (SHA256: 6e9e6b22715ba055dd1a884bdc18078851629992db7a628051c5ccc41d263635)
Sample solutions: homework-3-solutions.pdf

Homework 4

Date Assigned: April 1, 2025
Due Date: April 15, 2025, no later than 2:00pm Central Time.
PDF file: homework-4.pdf (SHA256: c1d1204ea77d579a379c6d25efb81fffcfc769317e754046a9b2e49ab69a1f31)
Source files: homework-4.zip (SHA256: 75ad92909df75544821d88a3a529bf07d6c7748c31b0114c9ba2df1104bcf7ed)
Sample solutions: homework-4-solutions.pdf

Homework 5

Date Assigned: April 22, 2025
Due Date: May 6, 2025, no later than 2:00pm Central Time.
PDF file: homework-5.pdf (SHA256: 01a81895717cab5d20fe83c7506f7293a02586af0fa53084c2865c5c6d1cf40c)
Source files: homework-5.zip (SHA 256: 19aeb9c23d39e0827c41b72213d6f64851bfa3ee5644e3477a50800565e00b9e)
Sample solutions: homework-5-solutions.pdf

Final Project

PDF Version

Final Project Group Presentation Schedule

I have randomly generated the Final Project Presentation schedule, as outlined below. The methodology is as follows.

1. List all groups in alphabetical order by first name, with each group being sorted alphabetically by first name internally.
2. Randomly generate a permutation to shuffle this alphabetical ordering.
3. Output the shuffled ordering. First 3 groups present on Tuesday, April 29, 2025; last 2 groups present on Thursday, May 1, 2025.

Groups

1. Ali
2. Brian & Victoria
3. Cameron
4. Javed & Nathan
5. Mohsen

Random permutation sampled: [4, 1, 3, 5, 2].

Final Ordering

Tuesday, April 29, 2025

1. Javed & Nathan
2. Ali
3. Cameron

Thursday, May 1, 2025

4. Mohsen
5. Brian & Victoria

Sagemath Code used to generate this result.

groups = ["Ali", "Brian & Victoria", "Cameron", "Javed & Nathan", "Mohsen"]
P = Permutations(5) # set of all permutations on [1,2,3,4,5]
p = P.random_element() # samples a random permutation of the list [1,2,3,4,5]
p # prints the sampled permutatation
shuffled_groups = [groups[i-1] for i in p]
shuffled_groups # prints the new ordering for 'groups'

Code screenshot below.

Project Description

The project you choose can be related to your research area, or completely unrelated. You may work in teams of up to 3 people. However, whatever your project is and however many people you have on your team, the project will consist of two major components—an In-Class Presentation, and a Written Report—as well as two minor components—Project Proposals and Peer Evaluations. There are no rigid page limits for the report; anything between 4–20 pages can work. For the In-Class presentation, you are expected to give a 15–18 minute presentation using the presentation materials of your choice (e.g., PowerPoint, Keynote, $\text{LATEX}$, Google Slides, a Board Talk, etc.).

The project can be a survey of a problem or topic of your choice, or a novel analysis of a problem that you like. In the first case, if you read some papers and summarize them in a survey, give the reader the required background (which may be covered only briefly in some conference papers) together with the main results and their proofs and open questions. For the second scenario, if you try to solve a problem that you are interested in, explain the connections with previous work; in case you don’t arrive to a solution by the end of the term, show what approaches you tried and what didn’t work.

The goal of the project is to understand a problem as much as possible, and to give you experience with complexity theory research. Give the reader the background and necessary explanations to make the problem very clear, understand the contribution of the paper, and the approaches used. You do not have to summarize every theorem in the paper (in either write-up or presentation). Pick instead one or a couple of results in the paper(s) you are reading and focus on those, while trying to answer questions such as: What is the idea of the proof? What techniques are the authors using? Where are the difficulties? What are the remaining open questions?

Recent proceedings of good conferences that publish theoretical work are a possible starting point. Some examples are STOC/FOCS/ITCS/SODA/APPROX-RANDOM, CRYPTO/TCC, EC (Economics and Computation), COLT, PODC.

Project Timeline

• Saturday, March 22, 2025, by 11:59pm CDT. Submit your project proposals to me. This includes at least one paragraph about your project, along with at least one (or, ideally, a few) papers you plan to read for the project. The easiest way to do this is to email me, cc your team members, and include the relevant information in the email.

• Tuesday, April 29, 2025, and Thursday, May 1, 2025. The last week/last two classes, we will hold the in-class presentations. Attendance will be required except for special cases. Part of this project is to give you experience presenting material you may not be very familiar with or an expert in to an audience who will be even less familiar with your topic. Also, you will be giving peer evaluations to your fellow students as well, and everyone will be required to submit these peer evaluations.

• Saturday, May 10, 2025, by 11:59pm CDT. Your written reports are due the day after finals. I want to give you as much time as possible for the written reports, but I will still need to read and grade them before grades are due. Early submissions (e.g., before finals) are also fine. You will submit your written reports via Gradescope Blackboard.

Grade Brakedown

• (5%) Project Proposals
• (10%) Peer Evaluations
• (40%) In-Class Presentation
• (45%) Written Report

Lecture 1

Review

Math Notation

Here are some common math notations we will be using throughout the semester. Any other notation introduced in lecture outside what is presented here will be explained. Please do not be afraid to ask questions about notation in class.

We let $\mathbb{Z}:=\{0,\pm 1,\pm 2,\dots ,\}$ denote the set of all integers and let $\mathbb{N}:=\{0,1,2,\dots ,\}$ denote the set of natural numbers (i.e., non-negative integers). The set ${\mathbb{Z}}^{+}:=\{1,2,\dots ,\}$ denotes the set of all positive integers. We let $\mathbb{R}$ denote the set of all real numbers, with $\lceil x\rceil$ and $\lfloor x\rfloor$ defined to be the ceiling and floor of $x\in \mathbb{R}$, respectively (i.e., round up or round down to the nearest integer). All logarithms are base 2 unless otherwise stated; i.e., $\log (x):={\log }_2(x)$. When need, we let $\ln (x):={\log }_{\mathrm{e}}(x)$ denote the natural logarithm.

Sets and Strings

Let $S$ be a finite set. We say that $s$ is a string with alphabet $S$ if it is a finite ordered tuple with elements in $S$. That is, if $s$ has length $n$, then $s\in S^n$ (i.e., it is a vector). We let $S^{\ast }$ denote the set of all finite length strings with alphabet $S$ (i.e., the set of all finite length vectors with elements in $S$). Given two strings $u$ and $v$, we let $u\Vert v$ denote their concatenation; sometimes, we also use $u\odot v$ or $(u,v)$ to denote their concatenation as well. Finally, given a string/vector $s$, we let $|s|$ denote the length of the string (i.e., number of elements in $s$). When needed, we let $|s{|}_2$ denote the bit-length of $s$; that is, the number of bits needed to represent $s$. If $|s|=n$, then $|s{|}_2=n\cdot \lceil \log (|S|)\rceil$.

Languages

Much of this course will be concerned with the idea of a language. A language is simply a set $L\subseteq \{0,1{\}}^{\ast }$. A language can be finite or infinite. While the above definition doesn’t really mean much, we’ll see later in the course how we define languages in more meaningful ways.

Function Notation

Let $A$ and $B$ be arbitrary sets (not necessarily finite). Then we let $f:A\to B$ denote a function from $A$ to $B$.

Big-Oh Notation

Let $f,g:\mathbb{N}\to \mathbb{N}$ be two functions. We way that $f$ is big-oh of $g$ if there exists $N\in \mathbb{N}$ and constant $c>0$ such that for all $n\ge N$, we have $f(n)\le c\cdot g(n)$. We denote this as $f(n)=O(g(n))$ or $f(n)\in O(g(n))$. Similarly, we say that $f$ is big-omega of $g$ if $g=O(f(n))$; we denote this as $f(n)=\Omega (g(n))$. We say that $f$ is theta of $g$ if $f(n)=O(g(n))$ and $f(n)=\Omega (g(n))$.

We say that $f$ is little-oh of $g$ if for all constants $c>0$, there exists $N\in \mathbb{N}$ such that for all $n\ge N$, it holds that $f(n)<c\cdot g(n)$. We denote this as $f(n)=o(g(n))$. Similarly, $f$ is litte-omeaga of $g$ if $g(n)=o(f(n))$; equivalently, for all constants $c>0$, there exists $N\in \mathbb{N}$ such that for all $n\ge N$, it holds that $f(n)>c\cdot g(n)$. We denote this as $f(n)=\omega (g(n))$.

Turning Machines

The goal of complexity theory is to quantify and measure computational efficiency. How can we do this? It is first necessary to establish a concrete model of computation. But this seems impossible—surely, there are infinite computational models one can cook up to get the job done, right?

Fortunately, we can focus on a single model of computation: the Turning Machine. For (nearly)¹ every physically realizable system we have been able to come up with, the Turing machine can efficiently simulate this other model. This gives us a single model with which we can try to understand and quantify computational efficiency.

Turing Machines, Informally

Turing machines, introduced by Alan Turing in 1948, are an attepmt to formalize the idea of computation as people have understood it for centuries. Intuitively, when someone asks you to compute the answer to a problem, we as people seem to follow a basic formula for realizing this:

1. Get the problem, along with the inputs to the problem.
2. Get a piece of scratch paper to work on.
3. Apply a set of rules to the inputs and make decisions according to those rules.
4. Arrive at an answer to the problem, state the answer, and stop working.

As a concrete example, consider the problem of multiplying two integers. Take $x=451$ and $y=127$ and suppose we want to compute their product $z=x\cdot y$. There is a simple (so-called “grade-school”) algorithm for computing $z$, which we show in the figure below.

Turing machines attempt to formalize the above intuitive process, but in a very restricted capacity (i.e., Turing machines are incredibly simple and, in a word, stupid).

Turing Machines, Formally

A $k$-tape Turing machine, which we denote by $M$, is a machine with $k$ tapes that are infinitely long in one direction (i.e., they are represented by the set $\mathbb{N}$). The machine $M$ has a single read-only input tape, a single read-write output tape, and $(k-2)$ read-write work tapes. $M$ also contains a register which tracks the machine’s state. Each head can be moved independently. More formally, a $k$-tape Turing machine $M$ is described by a tuple $(\Sigma ,\Gamma ,Q,\delta )$ with the following properties:

1. $\Sigma$ is a finite set, which we call the input alphabet;
2. $\Gamma =\Sigma \cup \{▹,\square \}$ is called the tape alphabet, and it contains $\Sigma$ along with two special symbols that are not in $\Sigma$. $▹\notin \Sigma$ is the start symbol and $\square \notin \Sigma$ is the blank symbol.
3. $Q$ is a finite set of states which can be held in the register. $Q$ always contains two special states: $q_{\mathrm{start}}$ (the start state) and $q_{\mathrm{halt}}$ (the halt state).
4. A transition function $\delta :Q\times {\Gamma }^k\to Q\times {\Gamma }^{k-1}\times \{L,R,S{\}}^k$ describing the rules $M$ follows during a computation.

We say that a tuple $(q,{\gamma }_1,\dots ,{\gamma }_k)\in Q\times {\Gamma }^k$ is a configuration of the Turing machine $M$. In this light, the transition function $\delta$ maps configurations of the Turing machine to new configurations as follows:

• $q$ represents the current state of the machine $M$ stored in its register;
• ${\gamma }_i\in \Gamma$ represents the current symbol under tape $i$’s head, for $i\in [k]$;
• $\delta$ reads the current state and the contents of the $k$ tape heads. It then outputs a new configuration $(q^{\prime },{\gamma }_2^{\prime },\dots ,{\gamma }_k^{\prime },d_1,\dots ,d_k)$, where
  • $q^{\prime }$ is the new state stored in the register;
  • ${\gamma }_i^{\prime }\in \Gamma$ is a new symbol that is written under the $i$th tape head for $i\ge 2$ (i.e., it excludes the input tape); and
  • $d_i\in \{L,R,S\}$ specifies moving tape head $i\in [n]$ one space Left, one space Right, or telling the tape head to Stay.²
• If the Turing machine is ever in the state $q_{mathrmhalt}$, it stops executing the transition function and halts.

Additionally, all Turing machines satisfy the following.

• All $k$ tapes are initially set with $\square$ in every location.
• The first index of every tape is then initialized to $▹$. All tape heads begin here.
• On input $x$, all Turing machines $M$ will:
  • Move the input head $R$, then write $x$ to the input tape.
  • Move the input head $L$ until it reaches $▹$.
  • Set the initial state in the register to $q_{\mathrm{start}}$. The Turing machine is now ready to begin its computation.

We call this the initial configuration of the Turing machine.

A graphical example of a $3$-tape Turing machine is presented below.

Turing machine example: Palindromes

Let’s see an example of a Turing machine in action. We will be using Turing machines to compute functions. Let $f:\{0,1{\}}^{\ast }\to \{0,1\}$ be a function such that $f(x)=1$ if and only if $x\in \{0,1{\}}^{\ast }$ is a palindrome. That is, $x=\mathrm{reverse}(x)$; equivalently, if $n=|x|$, then $x_i=x_{n-i+1}$ for all $i\in [n]$. Let’s design a Turing machine for computing $f$.

High-level TM Specification. Let $M_f$ be a $3$-tape Turing machine (1 input, 1 output, 1 work tape). On input $x\in \{0,1{\}}^n$ for any $n\in \mathbb{N}$, $M_f$ will do the following.

1. Copy the input $x$ to the work tape.

2. Move the input head to the start of the input tape (with $▹$ under the head), leave the work head at the last position (with $\square$ under the head).

3. Move the input head one position right (with $x_1$ under the head) and move the work head one position left (with $x_n$ under the head).

4. Read the symbols under the input head and the work head.

   (a) If the symbol under the input head is $\square$ and the symbol under the work head is $▹$, then write $1$ to the output tape and halt.

   (b) Else if the symbols under the input head and work head are not equal, write $0$ to the output tape and halt.

   (c) Else (i.e., the symbols are equal) move the input head one step right and move the work head one step left.

Formal TM Specification of above. We now formalize thie above process. To do so, we specify (1) the input alphabet; (2) the set of states for $M_f$; and (3) the transition function for $M_f$.

• The input alphabet $\Sigma$ is simply $\{0,1\}$. This tells us our tape alphabet is $\Gamma =\{0,1,▹,\square \}$.
• The set of states will be $Q=\{q_{\mathrm{start}},q_{\mathrm{halt}},q_{\mathrm{copy}},q_{\mathrm{left}},q_{\mathrm{test}}\}$.
• The transition function $\delta :Q\times {\Gamma }^3\to Q\times {\Gamma }^2\times \{L,R,S{\}}^3$ is defined as follows. Let $(q,{\gamma }_1,{\gamma }_2,{\gamma }_3)$ be a configuration given as input to $\delta$.
  • If $q=q_{\mathrm{start}}$, move both the input head and work head right and change the state to $q_{\mathrm{copy}}$.
  • If $q=q_{\mathrm{copy}}$, then read the symbol under the input head (i.e., read ${\gamma }_1$).
    • If ${\gamma }_1\ne \square$, write ${\gamma }_1$ to the current position of the work tape. Then move both the input head and work head one step right. Keep the state as $q_{\mathrm{copy}}$. In this case, the output of the transition function is $(q_{\mathrm{copy}},{\gamma }_1,{\gamma }_3,R,R,S)$.
    • If ${\gamma }_1=\square$, then move the input head left and change the state to $q_{\mathrm{left}}$. In this case, the output of the transition function is $(q_{\mathrm{left}},{\gamma }_2,{\gamma }_3,L,S,S)$.
  • If $q=q_{\mathrm{left}}$, read the symbol under the input head (i.e., ${\gamma }_1$).
    • If ${\gamma }_1\ne ▹$, then move the input head left and keep the state as $q_{\mathrm{left}}$. In this case, the output of the transition function is $(q_{\mathrm{left}},{\gamma }_2,{\gamma }_3,L,S,S)$.
    • If ${\gamma }_1=▹$, then move the input head right, move the work head left, and change the state to $q_{\mathrm{test}}$. In this case, the output of the transition function is $(q_{\mathrm{test}},{\gamma }_2,{\gamma }_3,R,L,S)$.
  • If $q=q_{\mathrm{test}}$, read what’s under the input and work heads (i.e., ${\gamma }_1$ and ${\gamma }_2$).
    • If ${\gamma }_1=\square$ and ${\gamma }_2=▹$, then write $1$ on the output tape and change the state to $q_{\mathrm{halt}}$. In this case, the transition function outputs $(q_{\mathrm{halt}},{\gamma }_2,1,S,S,S)$.
    • Else if ${\gamma }_1\ne {\gamma }_2$, then write $0$ on the output tape and change the state to $q_{\mathrm{halt}}$. In this case, the transition function outputs $(q_{\mathrm{halt}},{\gamma }_2,0,S,S,S)$.
    • Else (i.e., ${\gamma }_1={\gamma }_2$), then move the input head right and the work head left, keeping the state as $q_{\mathrm{test}}$. In this case, the transition function outputs $(q_{\mathrm{test}},{\gamma }_2,{\gamma }_3,R,L,S)$.

Turing Machine Equivalences

At first, the Turing machine model seems like a very restrictive model that cannot compute many things, especially real-life computers. However, as we will see, this restrictive computational model is (roughly) equivalent to nearly every other computational model people have thought of over the years. This includes:

• Random access machines;
• Turing machines with write-only output tapes;
• $\lambda$-calculus;
• Single-tape Turing machines;
• Turing machines with bidirectional infinite tapes;
• Pointer and Counter machines;
• Turing machines with only binary input alphabets;
• Oblivious Turing machines.

In the next lecture or two, we will quantify what we mean by Turing machines being equivalent to the above notions.

1. Quantum computers are the one (almost) physically realizable computational model that we have which does not seem to admit efficient simulation on Turing machines. ↩

2. If the Turing machine specifies that a tape head to move left, but it’s at the start of the tape, the head simply stays in the same place. ↩

Lecture 2

In-class notes: CS 505 Spring 2025 Lecture 2

Measuring Runtime of Turing Machines

With the definition of Turing machines established, we can turn towards quantifying the run-time of Turing machines. Informally, the run-time of a Turing machine computing some function $f$ is the maximum amount of time needed to compute $f$ on all inputs (of a fixed length), where our measure of time corresponds to how many executions of the transition function $M$ needs to utilize to compute $f$. First, we need to actually define what it means for a Turing machine to compute a function $f$.

Definition (Turing Machine Computation). Let $f:\{0,1{\}}^{\ast }\to \{0,1{\}}^{\ast }$ be a function and let $M$ be a Turing machine. We say that $M$ computes $f$ if for all $x\in \{0,1{\}}^{\ast }$, when $M$ is initialized with input $x,$ it halts with $f(x)$ on the output take. We denote this as $M(x)=f(x)$.

Now we can define the run-time of a Turing machine computing a function $f$.

Definition (Turing Machine Run-time). Let $f$ be a function and let $M_f$ be a Turing machine which computes $f$. Furthermore, let $T:\mathbb{N}\to \mathbb{N}$ be a function. We say that $M_f$ computes $f$ in time $T$ if $M_f(x)=f(x)$ in at most $T(|x|)$ steps for all $x\in \{0,1{\}}^{\ast }$ (i.e., $O(T(|x|))$ steps). Here, a step of the Turing machine $M_f$ is a single execution of its transition function $\delta$.

In essence, executing one step of the transition function of a Turing machine is the atomic Turing machine operation.

Time Constructible Functions

An important concept is the idea of time constructible functions, which we will use to quantify and show equivalences among different Turing machine models. It will also be used in later topics (e.g., time hierarchy theorems).

Definition (Time Constructible Function). Let $T:\mathbb{N}\to \mathbb{N}$ be a function. Then we say that $T$ is time construictible if and only if (1) $T(n)⩾n$; and (2) there exists a Turing machine $M$ such that for all $x\in \{0,1{\}}^{\ast }$, we have $M(x)=⟨T(|x|){⟩}_2$ in time $O(T(|x|))$. Here, $⟨T(|x|){⟩}_2$ denotes the binary representation of $T(|x|)$.

Examples of time constructible functions include $$T(n)=\{\begin{array}{l}n\\ n\log (n)\\ n^2\\ 2^n\end{array}$$ Notably, $T(n)=\log (n)$, or any $\omega (1)⩽T(n)⩽o(n)$, are not time constructible.

The above examples of non-time constructible functions highlight a key idea behind time constructibility: a Turing machine (usually) needs to read its entire input in order to compute a function. The stipulation $T(n)⩾n$ allows for a Turing machine to at least read the entire input before computing $T(|x|)$. If this restriction is removed, then $T(n)=\Theta (1)$ is still time constructible (you simply ignore all inputs and write the constant to the output tape), but $\omega (1)⩽T(n)⩽o(n)$ remain non-time constructible since the Turing machine is expected to compute $T(|x|)$ in less time than it takes to read $x$!

Turing Machine Equivalences

A function $T$ being time constructible turns out to be a key factor in how we define equivalences among Turing machines (and other models as well).¹ Informally, we say that a computational model $\mathcal{A}$ is equivalent to a computational model $\mathcal{B}$ if any for any computation capable of being performed in $B$ can be performed in $A$ (with at most polynomial time overhead). In the context of Turing machines, we say that a computational model $\mathcal{A}$ is equivalent to the Turing machine model if any problem solvable in time $T$ in model $\mathcal{A}$ can be solved by a Turing machine running in time $O(T^c)$ for constant $c>0$. Intuitively, if $P$ is a program in computational model $\mathcal{A}$ running in time $T$, then a Turing machine will simulate model $\mathcal{A}$ in order to run program $P$ (e.g., similar to modern interpreted programming languages like Python). If $P$ runs in time $T$, and $T$ is not time constructible, then this simulation will not meet our requirements; i.e., it will not be an efficient simulation.

As mentioned in Lecture 1, the $k$-tape Turing machine model we have been working with is equivalent to many other Turing machine (and non-Turing machine) models. We state these relations formally below.

First, recall that it is sufficient to consider a Turing machine which only uses a binary alphabet.

Lemma 2.1. For every $f:\{0,1{\}}^{\ast }\to \{0,1\}$ and time constructible $T$, if $f$ is computable in time $T$ by a Turing machine $M_f$ with tape alphabet $\Gamma$, then it is computable in time $O(T\cdot \log (|\Gamma |))$ by a Turing machine ${\hat{M}}_f$ with tape alphabet $\hat{\Gamma }=\{0,1,▹,\square \}$.

Proof Sketch. The main idea is to encode the (non-start and non-blank) symbols of $\Gamma$ using bits. This requires roughly $\lceil \log (|\Gamma |)\rceil$ bits to uniquely encode $\Gamma$ in binary. Then the new Turing machine ${\hat{M}}_f$ simply encodes each symbol from $\Gamma$ on its tapes in binary. To simulate a single step of $M_f$, the machine ${\hat{M}}_f$ must read $\log (|\Gamma |)$ bits from each tape, translate the symbol read into its current state, then execute $M_f$’s transition function. $\square$

Next, it turns out that any $k$-tape Turing machine can be readily simulated by a single-tape Turing machine (which many of you may have seen before).

Lemma 2.2. For every $f:\{0,1{\}}^{\ast }\to \{0,1\}$ and time constructible $T$, if $f$ is computable in time $T$ by a $k$-tape Turing machine, then it is computable in time $O(k\cdot T^2)$ by a single-tape Turing machine.

Proof Idea. The proof idea is to stagger the $k$ tapes onto the single-tape machine. Notably, since each of the $k$ tapes is infinite, if you try to write them side-by-side on a single-tape machine, you would inevitably run into a situation where you reach the end of an allocation for a work tape, so you’d have to shift the entire contents of the remaining tapes right one space. This would blow-up the time to simulate. So instead, you stagger the $k$ tapes. Consider tape $i$. Then positions $1,2,3,\dots ,j,\dots$ of tape $i$ would be written to positions $i,i+k,i+2k,\dots ,i+(j-1)k,\dots$ on the single-tape machine. $\square$

It also turns out that having tapes which are infinite in both directions does not buy you much in terms of computational efficiency.

Lemma 2.3. For every $f:\{0,1{\}}^{\ast }\to \{0,1\}$ and time constructible $T$, if $f$ is computable in time $T$ by a $k$-bidirectional tape Turing machine (i.e., every tape is infinite in both directions), then $f$ is computable in time $4\cdot T(n)$ by a standard $k$-tape Turing machine (i.e., tapes that are infinite in one direction).

Proof Idea. You can approach this two different ways.

1. Cut each bidirectional tape in half, then stagger this tape onto a single tape (similar to Lemma 2.2 above).

2. If the bidirectional Turing machine has tape alphabet $\Gamma$, let the standard Turing machine have tape alphabet $\hat{\Gamma }=\Gamma \times \Gamma$. Then you can encode the bidirectional tape onto the single tape using $\hat{\Gamma }$. $\square$

Universal Turing Machines

We’ve discussed how our $k$-tape Turing machine is equivalent to many other Turing machine models. Next, we will see that we can simulate any Turing machine (in any equivalent model). Much like how the modern computer can run any computation you give it, we will see there is a universal Turing machine which can simulate any Turing machine you give it as input.

Turing Machines are (Binary) Strings

We’ve focused our attention on Turing machines $M$ which compute some function $f:\{0,1{\}}^{\ast }\to \{0,1{\}}^{\ast }$, and we haven’t given much thought to how we write down the machine $M$. It turns out that we can conveniently describe Turing machines simply as binary strings. We’ll let $⟨M{⟩}_2\in \{0,1{\}}^{\ast }$ denote the binary string which represents the Turing machine $M$. Note: there are an infinite number of strings $x\in \{0,1{\}}^{\ast }$ which represent a single Turing machine $M$.

For any $\alpha \in \{0,1{\}}^{\ast }$, we will let $M_{\alpha }$ denote the Turing machine specified by the string $\alpha$. In this light, notice that

• We’ve always talked about Turing machines computing some function $f:\{0,1{\}}^{\ast }\to \{0,1{\}}^{\ast }$;
• Turing machines themselves are such a function; and
• Turing machines can also be inputs to these functions!

So there must be a Turing machine which can take Turing machines as input and compute the function that this Turing machine would have computed! This is the universal Turing machine.

Theorem 2.4 (Hennie & Stearns, 1966). There exists a Turing machine $\mathcal{U}$ such that for all $\alpha ,x\in \{0,1{\}}^{\ast }$, $\mathcal{U}(\alpha ,x)=M_{\alpha }(x)$. That is, $\mathcal{U}(\alpha ,x)$ computes the output of $M_{\alpha }$ when run with input $x$. Moreover, if $M_{\alpha }$ halts within $T$ steps on any input for time constructible $T$, then $\mathcal{U}(\alpha ,x)$ halts in $O(T\cdot \log (T))$ steps, where the hidden constant only depends on $M_{\alpha }$’s alphabet size, number of states, and number of tapes.

The proof of the above theorem can be found below (see Proof of Theorem 2.4). Here, we’ll give the proof of the above with $O(T\cdot \log (T))$ replaced with $O(T^2)$.

Proof with time bound $O(T^2)$. Suppose that $\alpha ,x\in \{0,1{\}}^{\ast }$. Without loss of generality, we can assume that the Turing machine $M_{\alpha }$ has tape alphabet $\{0,1,▹,\square \}$ and has a single work tape (i.e., it is a $3$-tape Turing machine). If not, then $\mathcal{U}$ can transform $M_{\alpha }$ into an equivalent Turing machine, denoted as ${\hat{M}}_{\alpha }$, with these properties by Lemmas 2.1 and 2.2.² In this case, if $M_{\alpha }$ runs in time $T$, then the resulting equivalent Turing machine ${\hat{M}}_{\alpha }$ runs in time $O(T^2)$ (ignoring the $\log |\Gamma |$ factors since $|\Gamma |$ is fixed).

The universal machine $\mathcal{U}$ will be a $5$-tape Turing machine; i.e., one input tape, one output tape, and 3 work tapes. $\mathcal{U}$ has alphabet ${\Gamma }_{\mathcal{U}}=\{0,1,▹,\square \}$. Now $\mathcal{U}$ will simulate ${\hat{M}}_{\alpha }$ as follows.

• $\mathcal{U}$ uses its input, output, and first work tape to identically copy the operations ${\hat{M}}_{\alpha }$ performs on these tapes (recall ${\hat{M}}_{\alpha }$ has $3$ tapes).
• $\mathcal{U}$ encodes the state space $\hat{Q}$ of ${\hat{M}}_{\alpha }$ on its second work tape.
• $\mathcal{U}$ encodes the transition function $\delta :\hat{Q}\times {\Gamma }^3\to \hat{Q}\times {\Gamma }^2\times \{L,R,S{\}}^3$ of ${\hat{M}}_{\alpha }$ on its third work tape. The transition function is simply encoded as a table of key-value pairs.

In order to simulate a single step of ${\hat{M}}_{\alpha }$’s computation, the machine $\mathcal{U}$ does the following.

1. Read the current symbols under the input tape, output tape, and first work tape. This identically matches what ${\hat{M}}_{\alpha }$ does and takes constant time.
2. Read the current state of ${\hat{M}}_{\alpha }$ from the second work tape. Since the tape alphabet is binary, the states of ${\hat{M}}_{\alpha }$ take $\lceil \log |\hat{Q}|\rceil$ bits to encode, so reading the current state takes $O(\log |\hat{Q}|)$ time steps (i.e., move to the end of the current state, go back to the start of the work tape).
3. Let $q_{\mathrm{current}}$ be the current state, and let ${\gamma }_1,{\gamma }_2,{\gamma }_3$ be the symbols read from the input, output, and first work tapes, respectively. Scan the third work tape for the key $(q_{\mathrm{current}},{\gamma }_1,{\gamma }_2,{\gamma }_3)$.
4. Once this key is found, read the value from the corresponding table entry. The value is exactly $(q^{\prime },{\gamma }_2^{\prime },{\gamma }_3^{\prime },d_1,d_2,d_3)=\delta (q_{\mathrm{current}},{\gamma }_1,{\gamma }_2,{\gamma }_3)$, where $d_i\in \{L,R,S\}$ for $i\in [3]$.
5. Execute the transition function $\delta$ of ${\hat{M}}_{\alpha }$.
   1. Write ${\gamma }_2^{\prime }$ to the output head and ${\gamma }_3^{\prime }$ to the head of work tape 1. This takes constant time.
   2. Write the new state $q^{\prime }$ to the second work tape and reset the tape head after. This take $O(\log |\hat{Q}|)$ time.
   3. Move tape head $i$ direction $d_i$ for $i\in [3]$. This takes constant time.
   4. Move the head of the third work tape back to the start.

Now, the time complexity of (3) and (5.4) above are the same. In particular, in the worst case, $\mathcal{U}$ must scan to the end of the table representing $\delta$ to find the correct state. There are $|\hat{Q}|\cdot |\Gamma {|}^3$ keys in this table, and each key has an entry in $\hat{Q}\times {\Gamma }^2\times \{L,R,S{\}}^3$. Since $|\Gamma |=4$ and because we can encode $\{L,R,S\}$ with only two more bits, we can conclude that each table entry (i.e., each key-value pair) has length $O(|\hat{Q}|)$. This means to write down a single entry, we need $O(\log |\hat{Q}|)$ bits. Moreover, there are a total of $O(|\hat{Q}|)$ entries in the table, so the total time of executing (3) or (5.4) is at most $O(|\hat{Q}|\log |\hat{Q}|)$ time.

Since $|\hat{Q}|$ is fixed, to simulate a single step of ${\hat{M}}_{\alpha }$ on $\mathcal{U}$ requires $O(1)$ time. So if ${\hat{M}}_{\alpha }$ runs in time $T^{\prime }$, then $\mathcal{U}$ runs in time $O(T^{\prime })$. Now $T^{\prime }=O(T^2)$ by the transformation we performed on $M_{\alpha }$ to obtain ${\hat{M}}_{\alpha }$. Thus, $\mathcal{U}$ simulates $M_{\alpha }$ in at most $O(T^2)$ time. $\square$

Turing Machines and Languages

We’ve spent most of our time discussing Turing machines and how they compute functions. We’ll now shift to mostly talking about Turing machines in the context of deciding languages.

Recall that a language $L$ is simply a subset of $\{0,1{\}}^{\ast }$. Notably, we can define a function $f_L:\{0,1{\}}^{\ast }\to \{0,1\}$ as $f(x)=1$ if and only if $x\in L$; this immediately implies that $f(x)=0$ if and only if $x\notin L$. So there is a natural correspondence to computing functions and deciding set membership in a language $L$.

Key to our later dealings with complexity classes will be the idea of Turing decidability. We’ll build up to this idea by first introducing Turing recognizability.

Definition (Turing Recognizable Language). A language $L$ is said to be Turing recognizable if there exists a Turing machine $M_L$ such that for all $x\in L$, $M_L(x)=1$. In particular, $M_L(x)$ always halts and outputs $1$ if $x\in L$.

Recognizability only requires that the Turing machine halt on any valid member of the language. If, however, one hands this Turing machine $x\notin L$, its behavior is undefined and not guaranteed! We’d like to strengthen this to make sure our Turing machine always halts, whether or not its input is in the language. This gives us decidability.

Definition (Turing Decidable Language). A language $L$ is said to be Turing decidable if there exists a Turing machine $M_L$ such that the following hold for any $x\in \{0,1{\}}^{\ast }$:

• $M_L(x)=1$ if and only if $x\in L$; and
• $M_L(x)=0$ if and only if $x\notin L$.

Notice that the above definition immediately means that $M_L$ halts on all possible inputs. This is because, equivalently stated, if $x\notin L$ then $x\in \overline{L}$, where $\overline{L}$ is the complement of $L$, which is defined as $\overline{L}=\{0,1{\}}^{\ast }\setminus L$ (i.e., everything in $\{0,1{\}}^{\ast }$ but not in $L$).

An equivalent definition of decidability states that both the language and its complement are recognizable.

Lemma 2.5. A language $L$ is Turing decidable if and only if both $L$ and $\overline{L}$ are Turing recognizable.

Undecidability

Unfortunately, there are many (interesting) languages that are undecidable; that is, there does not exist any Turing machine which decides the language. We’ll begin by showing the existence of at least one undecidable language.

Theorem 2.6. There exists a language $L_{\mathrm{UC}}$ that not Turing decidable (i.e., it is undecidable).

Proof. First define a language $\hat{L}=\{\alpha |M_{\alpha }(\alpha )=1\}$; i.e., $\hat{L}$ is the set of all strings $\alpha$ such that the Turing machine $M_{\alpha }$, when given input its own description $\alpha$, halts and outputs $1$. Now define the complement language $L_{\mathrm{UC}}=\{\alpha |\alpha \notin \hat{L}\}$. We claim that $L_{\mathrm{UC}}$ is undecidable.

We show this via a proof by contradiction. So towards contradiction, assume that $L_{\mathrm{UC}}$ is decidable. Then there exist a Turing machine $M_{\mathrm{UC}}$ which decides this language. This implies that for any $\alpha \in \{0,1{\}}^{\ast }$, $M_{\mathrm{UC}}(\alpha )=1$ if and only if $\alpha \in L_{\mathrm{UC}}$ and $M_{\mathrm{UC}}(\alpha )=0$ if and only if $\alpha \notin L_{\mathrm{UC}}$.

Consider $M_{\mathrm{UC}}(⟨M_{\mathrm{UC}}{⟩}_2)$. We have that

$$\begin{array}{rl}& M_{\mathrm{UC}}(⟨M_{\mathrm{UC}}{⟩}_2)=1\\ ⟺& ⟨M_{\mathrm{UC}}{⟩}_2\in L_{\mathrm{UC}}\\ ⟺& ⟨M_{\mathrm{UC}}{⟩}_2\notin \hat{L}\\ ⟺& M_{\mathrm{UC}}(⟨M_{\mathrm{UC}}{⟩}_2)=0\text{ or never halts. }\end{array}$$ Thus, we have a contradiction as $1\ne 0$. This implies that $L_{\mathrm{UC}}$ is undecidable. $\square$

Notably, the above proof technique is known as diagonalization. We’ll use it later when we discuss time hierarchy theorems.

One may argue that the language $L_{\mathrm{UC}}$ is not a very interesting language class, and may not come up in the real world. However, we’ll take one-step up and consider a more interesting language that would be great for us if it were decidable! Unfortunately, it is not decidable.

The Halting Problem

The Halting problem asks the following simple question: given a Turing machine $M$, does it halt on input $x$? More formally, it is specified by the following language: $$L_H=\{(\alpha ,x)|M_{\alpha }(x)\text{ halts in a finite number of steps}\}.$$

Theorem 2.7. $L_H$ is undecidable.

We’ll give the proof of this theorem in Lecture 3.

Proof of Theorem 2.4

This proof is taken directly from Arora & Barak’s book with the following notes:

• Theorem 1.9 in the proof corresponds to Theorem 2.3 in these lecture notes;
• Claim 1.6 in the proof corresponds to Lemma 2.2 in these lecture notes; and
• Claim 1.5 in the proof corresponds to Lemma 2.1 in these lecture notes.

The proof can be found in the following pdf: Proof of Theorem 2.4

1. “Key” here meaing it makes proofs much simpler. ↩

2. Lemma 2.2 tells us that a $k$-tape Turing machine can be simulated by a one-tape Turing machine with quadratic overhead. The same proof can be applied to reduce $k$-tapes to $3$-tapes, with a single input, output, and work tape (i.e., transform the $k-2$ work tapes into a single work tape, keep the input/output tapes the same). ↩

Lecture 3

In-class notes: CS 505 Spring 2025 Lecture 3

Undecidability Wrap-up

We begin by wrapping up our discussion of undecidability.

The Halting Problem

From last time, we’ll finish proving that the halting problem is undecidable. First, recall the definition of the halting problem.

$$L_H=\{(\alpha ,x)|M_{\alpha }(x)\text{ halts in a finite number of steps}\}.$$

Theorem 2.7. $L_H$ is undecidable.

Proof. We’ll prove this via a reduction to the language $L_{\mathrm{UC}}$ from last lecture, defined as

$$L_{\mathrm{UC}}=\{\alpha |M_{\alpha }(\alpha )=0\text{ or never halts}\}.$$

Our proof will be by contradiction. In particular, this means we’ll assume that $L_H$ is decidable, then derive our contradiction by giving a decider for $L_{\mathrm{UC}}$.

Thus assume that $L_H$ is decidable. This means there is a Turing machine $M_H$ which decides $L_H$. This tells us that for every pair $(\alpha ,x)$, we have $M_H(\alpha ,x)=1$ if and only if $M_{\alpha }(x)$ halts, and $M_H(\alpha ,x)=0$ if and only if $M_{\alpha }(x)$ does not halt.

We’ll use $M_H$ to build a Turing machine $M_{\mathrm{UC}}$ which decides $L_{\mathrm{UC}}$. For any $\alpha$, define $M_{\mathrm{UC}}$ as follows.

• $M_{\mathrm{UC}}(\alpha ):$
  • Set $b=M_H(\alpha ,\alpha )$.
  • If $b=0$, then output 1.
  • If $b=1$, then set $b^{\prime }=M_{\alpha }(\alpha )$.
    • If $b^{\prime }=0$, output 1.
    • If $b^{\prime }=1$, output 0.

Since $M_H$ is a decider, it halts on all possible inputs $(\alpha ,\alpha )$. Now, if $M_H(\alpha ,\alpha )=0$, we know that $M_{\alpha }(\alpha )$ does not halt, which implies that $\alpha \in L_{\mathrm{UC}}$. So we set $M_{\mathrm{UC}}(\alpha )=1$ in this case. Next, if $M_H(\alpha ,\alpha )=1$, we know that $M_{\alpha }(\alpha )$ does halt. We then test the output of $M_{\alpha }(\alpha )$ by running it. If $M_{\alpha }(\alpha )=0$, then again we know $\alpha \in L_{\mathrm{UC}}$, so we set $M_{\mathrm{UC}}(\alpha )=1$. Otherwise, $M_{\alpha }(\alpha )=1$, and thus $\alpha \notin L_{\mathrm{UC}}$, so we set $M_{\mathrm{UC}}(\alpha )=0$.

Thus, $M_{\mathrm{UC}}$ halts on all possible inputs, and clearly $M_{\mathrm{UC}}$ decides $L_{\mathrm{UC}}$. This contradicts our previous result that $L_{\mathrm{UC}}$ is undecidable. Therefore, $L_H$ is undecidable. $\square$

Final Remarks on Undecidability

Rice’s Theorem

It would be great if the halting problem were decidable, as it would give us an efficient way to check if programs halt on all possible inputs. However, one may be wondering if there are other properties about programs we can efficiently decide/test. For example, “does this program have at least 5 for-loops?” or “does this program have a switch statement, followed by an if-then-else?” Unfortunately, these are also undecidable problems.

This is a result known as Rice’s Theorem, which informally states that it is impossible to determine if a computer program has any non-trivial property $P$. I.e., the language $$L_P=\{\text{all programs with non-trivial property P}\}$$ is undecidable. Here, a non-trivial property $P$ is a property which is not true or false for every program (i.e., there are some programs that satisfy $P$, and some which do not).

Mathematical Incompleteness

The idea of undecidability (and uncomputability) is closely related to (and inspired by) Gödel’s incompleteness theorem. In the early 1900’s, there was a large push to establish a set of mathematical aximoms from which you can prove or disprove any mathematical property. However, Gödel proved this is impossible. He showed that no matter what set of axioms you choose, there will always be theorems you cannot prove or disprove. This actually inspired the results on undecidability/uncomputability, and is closely related to these ideas.

Time-Efficient Computations

We’ll now turn our focus to a central topic in complexity theory: defining classes of efficient computations. This leads us to defining and discussing various complexity classes. Informally, a complexity class is simply a set of languages which are decidable (resp., computable) within some resource bound. Example resource bounds include running in linear time, running in logarithmic space, etc.

Deterministic Time

Building towards what we as computer scientists consider efficient, we turn to time bounds. We’ll define the notion of deterministic time.

Definition. Let $T:\mathbb{N}\to \mathbb{N}$ be a function. A language $L$ is in the class DTIME$(T)$ if and only if $L$ is decidable by a (deterministic) Turing machine in time $O(T)$.

All Turing machines we’ve discussed and defined so far have been deterministic. These machines all have straight-line computations: they execute their transition function, which simply outputs the next state. Later, we’ll see non-deterministic Turing machines, where the transition function can output a set of possible states and the Turing machine non-deterministically decides which state to pick next.

The Complexity Class P

Given the definition above, we can now define the set of (what we consider to be) all efficient computations. This is the complexity class P (which stands for polynomial).

Definition (P) $$\mathbf{P}=\bigcup _{c\in \mathbb{N}}\mathbf{DTIME}(n^c)$$

We consider anything computed in polynomial time (with respect to the input length) to be efficient. Examples of problems/languages in P include:

• Graph connectivity
• Digraph path exists
• Checking if a graph is a tree
• Integer multiplication: does $x\cdot y=z$?
• Are the integers $x$ and $y$ relatively prime?
• Gaussian elimination over rational numbers: For matrix $A$ and vector $b$, does there exist $x$ such that $Ax=b$?

Discussions on P

Does the computational model matter?

We’ve defined P with respect to $k$-tape Turing machines. But, as we’ve seen, $k$-tape Turing machines are equivalent to all other Turing machine models we’ve seen, including RAM Turing machines which reasonably emulate real-life computers. Moreover, the “equivalence” here is that all machines can simulate all other ones with at most polynomial overhead in the runtime. This means all of these computations still fall within the class P.

In fact, many people believe that Turing machines can simulate any physically realizable computational model or system. This is known as the Church-Turing thesis.¹ Some people also believe in the strong Church-Turing thesis, which states that this simulation can be done with only polynomial overhead in the runtime. However, as we get closer to quantum computing being physically realizable, people may stop believing in this since, for now, we do not know of a way to simulate quantum computations on standard Turing machines with only polynomial overhead.

Why polynomial time?

It is certainly true that an algorithm running in time $n^{100}$ is impractical starting at $n=2$; yet this is a polynomial. Why do we consider all polynomial time algorithms to be “efficient?”

One reason is above: the Turing machine is polynomially-equivalent to pretty much every model we have thought of, so it makes sense that polynomial time should appear somewhere in what we consider to be an efficient computation. Polynomials also compose well, which emulates how we compose computer programs. Often, computer programs will run sub-routines, and will run routines one after another. If all these runtimes are polynomial, then the final runtime remains polynomial as well. This is since for two polynomials $p(X)$ and $q(X)$, the functions $p(X)+q(X)$, $p(X)\cdot q(X)$, and $p(q(X))$ or $q(p(X))$ are all still polynomials.

Another reason is historic and heuristic. Often in history, someone is able to solve a problem in polynomial time, but for some large polynomial like $n^{50}$. But this algorithm is later improved to a more reasonable polynomial, such as $n^5$ or $n^3$.

Finally, polynomial-time problems are roughly equivalent to most (if not all) problems that we can efficiently solve on modern computers.

Worst-case time complexity is too restrictive

If you have a problem where for $99\%$ of the inputs you have an $n^2$ algorithm, but for $1\%$ you have an $n^7$ algorithm, then we’d say the algorithm runs in time $O(n^7)$. In particular, we keep P as a worst-case class. Some argue that this is too restrictive, which is valid. However, often it is much simpler to construct an algorithm that can solve all problem inputs in some amount of time, rather than trying to enumerate (the possibly infinite amount of) the inputs which have better algorithms.

This criticism of P is also addressed in complexity theory itself via the introduction of alternative models and classes, including approxmiation algorithms and average-case complexity.

Decision problems are too limited

We’ve framed P as a class of decision problems, but often we actually want to find solutions to these problems. This is known as a search problem, where you are asked to find an answer rather than decide if something is true or false. An example of this is: instead of deciding if there exists an $x$ such that $Ax=b$, you just compute the solution $x$. It can also be difficult to frame search problems as decision problems in the first place.

However, most often it is the case that the difference between search and decision problems is, again, only polynomial. That is, we often can solve a search problem when given an algorithm that decides the equivalent decision problem, only costing us polynomial overhead in the runtime; the reverse is often true as well.

Time-Efficient Verification of Problems

Sometimes, we don’t want to solve problems, but would like to verify solutions when given an answer. Moreover, this verification should at least as efficient as solving the problem itself.

Efficiently Verifiable Languages

This gives us a new way to define languages: efficiently verifiable languages.

Definition. Let $L\subset \{0,1{\}}^{\ast }$ be a language. We say that $L$ is efficiently verifiable if there exists polynomials $p$ and $q$, and Turing machine $M_L$ running in time $q$ such that $$x\in L⟺\exists w\in \{0,1{\}}^{p(|x|)}\text{ s.t. }{M}_L(x,w)=1.$$

In the above definition, we call $M_L$ a verifier, $x$ the instance, and $w$ the certificate or witness.

The Class NP

The above new notion of languages gives us a new complexity class: NP.

Definition (NP). $$\mathbf{NP}=\{L:L\text{ is efficiently verifiable}\}.$$

P vs. NP

However, we do know one thing for certain.

Theorem 3.1. $\mathbf{P}\subseteq \mathbf{NP}$.

This is true since every problem in $\mathbf{P}$ can be decided in polynomial time with no witness/certificate. So it meets the definition of efficiently verifiable.

Non-deterministic Turing Machines and NP

There is an alternative definition of the class $\mathbf{NP}$, which utilizes non-deterministic Turing machines.

Definition. A non-deterministic $k$-tape Turing machine is identical to a (deterministic) $k$-tape Turing machine, except for the following modifications.

1. The transition function of the non-deterministic Turing machine is defined as $\delta :Q\times {\Gamma }^k\to \mathcal{P}(Q\times {\Gamma }^{k-1}\times \{L,R,S{\}}^k)$, where $\mathcal{P}$ denotes the power set operation.² During any step of the computation, the transition function outputs a (possibly empty) list of next possible Turing machine configurations.
2. Given a list of next possible configuration from the transition function, the non-deterministic Turing machine non-deterministically chooses the next configuration to execute from this list.³

Intuitively, deterministic Turing machines (the ones we defined in Lecture 1) are “straight-line”: every step of the computation proceeds directly from the previous one. For non-deterministic Turing machines (which we’ll denote as NTMs), they look more like “branching” programs: at every step of the computation, the Turing machine has a set of possible computational paths to head down, and non-deterministically chooses the path to proceed down.

How do we define decidability of a language with respect to NTMs? At first, it may seem difficult since there are many possible paths an NTM can do down during its computation. But, the answer turns out to be simple: we require all computational paths to halt, and there to be at least one accepting path (out of possibly exponential) which correctly outputs the decision.

Definition. A language $L$ is decidable in time $T$ by a non-deterministic Turing machine $M$ if

1. $x\in L$ if and only if there exists at least one execution path such that $M(x)=1$.
2. All execution branches halt in time at most $T(|x|)$ for any $x\in \{0,1{\}}^{\ast }$.

We can use this above definition to expand DTIME to NTIME.

Definition. Let $T:\mathbb{N}\to \mathbb{N}$ be a function. Then we define $\mathbf{NTIME}(T)$ to be the set of all languages $L$ decidable by an NTM running in time $O(T)$.

Alternative Definition of NP

Given NTMs and NTIME, we can now see the original formulation of the class NP.

Theorem 3.2. $$\mathbf{NP}=\bigcup _{c\in \mathbb{N}}\mathbf{NTIME}(n^c).$$

Note that this definition is equivalent to the efficiently verifiable language definition. At a high level, this is because of the following reduction.

• Let $w$ be a witness to the fact that $x\in L$ (i.e., $M_L(x,w)=1$ for efficient verifier $M_L$). Then, intuitively, $w$ correpsonds to some correct computational path on an NTM which decides $L$.
• Let $M$ be an NTM which decides $L$. Then we can specify a witness $w$ which is the computational path that $M$ takes to an accepting state. The deterministic machine $M_L$ takes this as input and simulates the NTM $M$ by following the computational path specified by $w$.

Solving NTIME in DTIME

Currently, until $\mathbf{P}$ vs. $\mathbf{NP}$ is resolved, the most efficient ways that we know of to solve problems in NTIME using only DTIME computations requires exponential time. Let $\mathbf{EXP}$ denote the class $$\mathbf{EXP}=\bigcup _{c\in \mathbb{N}}\mathbf{DTIME}(2^{n^c}).$$

Lemma 3.3. $\mathbf{NP}\subset \mathbf{EXP}$.

Proof. Enumerate all possible branches of the NTM deciding the language $L\in \mathbf{NP}$ (equivalently, enumerate all certificates/witnesses in the verifier definition). Then, run through this list until finding an accepting branch of the computation. If the original machine ran in time $T(n)$, then this procedure runs in time $O(2^{T(n)})$. By assumption, $T(n)$ is a polynomial, so we are done. $\square$

1. Note this is just a belief and not a formal theorem or conjecture. ↩

2. Given a set $S$, the power set of $S$, denoted as $\mathcal{P}(S)$, is the set of all possible subsets of $S$. Notably, $|\mathcal{P}(S)|=2^{|S|}$. ↩

3. Recall that non-determinism is not the same as behaving randomly. The choice of a non-deterministic machine is arbitrary and possibly not computable. ↩

Lecture 4

In-class notes: CS 505 Spring 2025 Lecture 4

Recall how we have the notion of a universal Turing machine: a machine that can simulate and solve any problem that any other Turing machine can solve. We’d like to now define a notion that is similar to this where, if you can solve one problem efficiently, then you can use that algorithm to solve a different problem (also efficiently). This leads us to the notion of reducibility.

Reducibility

As above, the idea of reducibility is that if I can solve problem $B$ (i.e., decide the language $B$), then I can use $B$ to solve (i.e., decide) a different language $A$. Moreover, this is efficient: there is only a polynomial overhead in using $B$ to solve $A$.

Definition (Polynomial-time Reducibility). Let $A$ and $B$ be languages. We say that $A$ is polynomial-time reducible to $B$, denoted as $A{\le }_{p}B$ if there exists a function $f$ that is computable in polynomial-time such that $$x\in A⟺f(x)\in B.$$

Lemma 4.1 (Reducibility is Transitive). Let $A$, $B$, and $C$ be languages. If $A{\le }_{p}B$ and $B{\le }_{p}C$, then $A{\le }_{p}C$.

Proof. By definition, there exist functions $f,g$ such that

$$\begin{array}{r}x\in A⟺f(x)\in B\\ y\in B⟺g(y)\in C\end{array}$$

This implies that $x\in A⟺g(f(x))\in C$. Note that both $f$ and $g$ are polynomial-time computable, so the function $g\circ f=g(f(\cdot ))$ is computable in polynomial time. $\square$

Since reducibility is efficient, it immediately tells us that if one of the problems is efficient, then the other is also efficient.

Theorem 4.2. If $A{\le }_{p}B$ and $B\in \mathbf{P}$, then $A\in \mathbf{P}$.

NP-Completeness

NP-Completeness captures the ideas and goals we’ve been building so far: problems in NP that if we can solve, then we can solve any other problem in NP.

Definition ($\mathbf{NP}$-Completeness). Let $L$ be a language. We say thta $L$ is $\mathbf{NP}$-complete if

1. The language is in $\mathbf{NP}$: $L\in \mathbf{NP}$; and
2. The language is $\mathbf{NP}$-hard: $\forall \hat{L}\in \mathbf{NP}$, we have $\hat{L}{\le }_{p}L$.

Notice that there can be languages $L\in \mathbf{EXP}$ such that $L$ is NP-hard, but this would not be NP-complete, unless $L\in \mathbf{NP}$ or $\mathbf{NP}=\mathbf{EXP}$ (which we don’t know is true or not). NP-completeness captures the intuition that if we can use a language to efficiently verify every other language in NP, then this language itself should be efficiently verifiable (otherwise we just verify the other languages directly).

Unhelpful/Useless NP-Complete Language

We’ll now see an example of an NP-complete language which is not helpful for solving problems. This is because, as we’ll see, it is intimately tied to the Turing machine.

Denote by $TMSAT$ the language of all satisfiable Turing machines, defined as $$TMSAT=\{(\alpha ,x,1^n,1^t):\exists w\in \{0,1{\}}^n\text{ s.t. }{M}_{\alpha }(x,w)=1\text{ in at most }t\text{ steps}\}.$$ Here, $1^n$ and $1^t$ denote a string of $n$ (resp., $t$) 1’s. This is a syntactic convention we use to ensure that any machine deciding $TMSAT$ runs in time that is polynomial in $n$ and $t$; whereas if we specified $n$ and $t$ in binary, then the machine would only run in polynomial time with respect to the bit-length of these numbers.

Lemma 4.3. $TMSAT$ is NP-complete.

Proof. Clearly $TMSAT\in \mathbf{NP}$ by definition. The NTM deciding $TMSAT$ takes the input $(\alpha ,x,1^n,1^t)$, guesses the string $w\in \{0,1{\}}^n$ and runs $M_{\alpha }(x,w)$. If $M_{\alpha }$ exceeds $t$ computational steps, output $0$; otherwise, output according to $M_{\alpha }(x,w)$ ($0$ if $0$, $1$ if $1$).

We now show that $TMSAT$ is NP-hard. That is, for any $L\in \mathbf{NP}$, we show $L{\le }_{p}TMSAT$. To do so, we define a function $f$ satisfying: $x\in L⟺f(x)\in TMSAT$. To being, let $p$ and $q$ be polynomials related to the Turing machine which verifies the language $L$. That is, $p$ and $q$ correspond to the Turing machine $M_L$ which on any input $x$ and witness $w\in \{0,1{\}}^{p(|x|)}$ runs in time at most $q(|x|+|w|)$.

Now we define $f(x)$ as follows for any $x\in \{0,1{\}}^{\ast }$.
$$f:x\mapsto (⟨M_L{⟩}_2,x,1^{p(|x|)},1^{q(|x|+p(|x|))}).$$ The tuple $f(x)$ is in $TMSAT$ if there exists $w\in \{0,1{\}}^{p(|x|)}$ such that $M_L(x,w)=1$ in at most $q(|x|+p(|x|))$ steps. Notice that this is trivially true by definition of the NP language $L$. Therefore we have $f(x)\in TMSAT⟺x\in L$. $\square$

This NP-complete language isn’t useful because it’s very definition makes it trivially NP-complete. Moreover, it is inherently tied to the definition of a Turing machine. Intuitively, this says that: if you can compute the Turing machine which verifies the language $L$, then you can compute the Turing machine which verifies the language $L$.

Ideally, we’d like a language that is NP-complete irrespective of the computational model we use. Intuitively, we want to show that the problem itself that is captured by the language is NP-complete, which would tell us that as long as we can solve this problem (and not the Turing machine tied to the problem), then we can solve other problems in NP.

Boolean Satisfiability

The problem we will examine as a candidate for NP-completeness in this light is Boolean Satisfiability. Recall the notion of Boolean variables or Boolean literals $x_1,\dots ,x_n$, which take on True/False values, where we use $1/0$ to denote these values, respectively. Similarly, recall Boolean operations: for example, $\vee$ (logical OR), $\wedge$ (logical AND), $\oplus$ (logical XOR), $\neg$ (logical NOT, denoted as $\overline{x}=\neg x$), etc. Then, a Boolean expression or Boolean formula is an expression involving Boolean variable and operations (e.g., $(\overline{x}\wedge y)\oplus (x\vee z)$). We define the length or size of a Boolean formula to be the number of non-$\neg$ operations in a formula.

For our purposes, we will only consider Boolean formulas which consist of AND, OR, and NOT. It is a well-known fact that these three operations are universal: any Boolean formula can be rewritten as an equivalent formula using only AND, OR, and NOT. Finally, we say that a Boolean formula $\varphi$ is satisfiable if there exists an assignment of the variable $x_1,\dots ,x_n$ such that $\varphi (x_1,\dots ,x_n)=1$.

Now, the language of Boolean Satisfiability is defined as follows. $$SAT=\{⟨\varphi {⟩}_2:\varphi \text{ is satisfiable}\}.$$

How powerful is $SAT$? One measure of its power is the collapse of P vs. NP if we find a polynomial-time algorithm for deciding $SAT$.

Theorem 4.4. $SAT\in \mathbf{P}⟺\mathbf{P}=\mathbf{NP}$.

Cook-Levin Theorem: SAT is NP-complete

In the 1970’s, Cook and Levin independently showed that $SAT$ is NP-complete. This means that if we can find a satisfying assignment for Boolean formulas, we can solve any problem in NP. We’ll begin proving this theorem, then wrap up the proof in the next lecture.

Theorem 4.5 (Cook-Levin). $SAT$ is NP-complete.

Proof. We must show that $SAT\in \mathbf{NP}$ and that $SAT$ is NP-hard. The first task is straightforward. For the second task, at a high level, we must construct a polynomial-time reduction from any language $L$ to an instance of $SAT$. This reduction must have the property that the $SAT$ instance is satisfiable if and only if membership in $L$ is true. Conceptually, we’ll construct a Boolean formula which encodes the correctness of the Turing machine deciding the language $L$. At a high-level, this is a simple task, but the devil is in the details with this reduction.

To begin, we show that $SAT\in \mathbf{NP}$. We give a simple NTM deciding $SAT$. Let $\varphi$ be a Boolean formula and suppose $\varphi$ has $n$ literals $x_1,\dots ,x_n$. Then, the machine $M_{SAT}$ on input $\varphi$ simply guesses a satisfying assignment for $x_1,\dots ,x_n$, checks if $\varphi$ evaluates to $1$ under this assignment, then accepts or rejects accordingly. Clearly, $M_{SAT}$ is a NTM which decides $\varphi$, and the running time of $M_{SAT}$ is clearly polynomial in the length of $\varphi$.

We now turn to showing that $SAT$ is NP-hard. Before doing this, we switch to the convention of single-tape non-deterministic Turing machines. That is, we’ll use the definition of NP languages where $L\in \mathbf{NP}$ if and only if there is a single-tape NTM which decides $L$ in polynomial time. Since, like deterministic machines, many-tape NTMs are (polynomially equivalent to) single-tape NTMs, everything remains in NP.

The idea behind the reduction is the following. Let $L\in \mathbf{NP}$ with single-tape NTM $M_L$ deciding $L$, and consider any $w\in \{0,1{\}}^{\ast }$.¹ The reduction (i.e., the function $f$) will first map the execution of $M_L(u)$ to a table representing this execution. Then, the reduction will specify a Boolean formula that is satisfiabile if and only if this table representing the execution is correct and accepts the input $x$; otherwise the formula will be unsatisfiable.

Assume that on inputs of length $n$, the machine $M_L$ runs in time $n^k$ for some constant $k$ (for convenience in the proof, we actually assume the runtime is $n^k-3$, but this is a minor detail). We’ll construct a table $T$ representing the computation of $M_L(u)$ of size $n^k\times n^k$. Every row of the table has the following properties:

1. The start and end of every row is filled with a special symbol $\#\notin \Gamma$, where $\Gamma$ is the tape alphabet of $M_L$. We’ll index the start of the row by $0$.²
2. For every row $i$, the cells between the start and end $\#$ symbols contain the contents of $M_L$’s single tape, plus its current state $q\in Q$. The current state $q$ is used to represent the current position of $M_L$’s single tape head.
   • If $q$ is at position $j$ in the row for $1\le j\le n^k-1$, then the tape head is reading from position $j+1$ in the table (which corresponds to the tape head being above position $j$ on $M_L$’s tape (here, we start indexing $M_L$’s tape at $1$).
3. The first row of the table (row $0$) always has the starting configuration of $M_L$. This corresponds to the tuple $(\#,q_0,w,\square ,\dots ,\square ,\#)$.

Since $M_L$ runs in time at most $n^k-3$, it can read/write to/from at most $n^k-3$ cells on its work tape. This is exactly the number of slots in a row of table which are dedicated to the work tape configuration, plus 2 slots for $\#$, and one more slot for the current state.

Our goal is to define a Boolean formula $\varphi$ capturing the correctness of the table representing $M_L(w)$. To do this, we first set up the alphabet of the table. Let $C=Q\cup \Gamma \cup \{\#\}$. We call $C$ the table alphabet. We let $cell[i,j]\in C$ denote a cell of the table for all $i,j\in \{0,1,\dots ,n^k-1\}$.

For every cell $cell[i,j]$ and every $s\in C$, we define a unique Boolean literal $x_{i,j,s}$. This literal represents the statement “$cell[i,j]=s$”. In particular, if $cell[i,j]=\square$, then we would set $x_{i,j,\square }=1$, and if $cell[i,j]\ne \square$, then we’d set $x_{i,j,\square }=0$. The reverse is also true; the literal being $1$ means the cell contains that element from $C$, and being $0$ means it does not.

Using these literals, we’ll now encode the correctness of the table for $M_L(w)$ into a Boolean formula $\varphi$. This formula is going to be the conjunction (i.e., logical AND) of 4 sub-formulas: $$\varphi ={\varphi }_{start}\wedge {\varphi }_{accept}\wedge {\varphi }_{cell}\wedge {\varphi }_{move}.$$

The formula ${\varphi }_{start}$ is simple: it will represent the correct starting configuration of the machine. This is a straightforward AND of many literals, shown below: $${\varphi }_{start}=x_{0,0,\#}\wedge x_{0,1,q_0}\wedge x_{0,2,w_1}\wedge \cdots \wedge x_{0,n+1,w_n}\wedge x_{0,n+2,\square }\wedge \cdots \wedge x_{0,n^k-2,\square }\wedge x_{0,n^k-1,\#}.$$

Next, the formula ${\varphi }_{accept}$ will check that the table is an accepting table. That is, it will check that there exists at least one accepting state $q_{accept}$ somwhere in the table. Note that we do not care where this accepting state is, nor if there is also a rejecting state, located in the table; we will handle these consistency checks with ${\varphi }_{move}$. Since all we care about is there is at least one accepting state, we can simply take a large OR of all the cells, yielding: $${\varphi }_{accept}=\underset{1\le i,j\le n^k-2}{\bigvee }{x}_{i,j,q_{accept}}.$$

The formula ${\varphi }_{cell}$ is going to make sure that every cell of the table only contains a single element of $C$. That is, we check to make sure that (1) every cell contains an element of $C$, and (2) every cell only contains a single element of $C$. For (1), we can check this with a simple OR. Let $0\le i,j\le n^k-1$. Then we can check if $cell[i,j]$ contains an element of $s$ using the expression $$\underset{s\in C}{\bigvee }{x}_{i,j,s}.$$ If this is true, we know that $cell[i,j]\in C$.

Now we ensure that $cell[i,j]$ only contains a single value from $C$. This is done by making sure that for all $s,t\in C$ such that $s\ne t$, the expression ${\overline{x}}_{i,j,s}\vee {\overline{x}}_{i,j,t}$ is true. This expression evaluates to false when $cell[i,j]$ contains both $s$ and $t$. If it contains at most one of $s$ or $t$ (including neither of them), then this expression is satisfied. Then we check that this holds over all $s\ne t$. Thus, (2) is captured by the formula= $$\underset{\begin{array}{c}s,t\in C\\ s\ne t\end{array}}{\bigwedge }({\overline{x}}_{i,j,s}\vee {\overline{x}}_{i,j,t}).$$

Therefore, a single cell $cell[i,j]$ is valid if both (1) and (2) hold. We then check that this condition holds for all possible cells, yielding our final expression $${\varphi }_{cell}=\underset{0\le i,j\le n^k-1}{\bigwedge }\left[\left(\underset{s\in C}{\bigvee }{x}_{i,j,s}\right)\wedge \left(\underset{\begin{array}{c}s,t\in C\\ s\ne t\end{array}}{\bigwedge }({\overline{x}}_{i,j,s}\vee {\overline{x}}_{i,j,t})\right)\right].$$