Lecture 14

In-class notes: CS 505 Spring 2025 Lecture 14

Randomized Computations

So far, we have only examined deterministic and non-deterministic Turing machines. Crucially, neither of these models for Turing machines utilize randomness. This is obvious for deterministic Turing machines, but it is very important to understand that non-determinism is not randomness, it is an idealized computation model that is not realistic.

Today, we’ll consider probabilistic Turing machines. These Turing machines will be allowed to sample uniformly and independently random bits $b\stackrel{\$}{\leftarrow }\{0,1\}$ and use these bits to help make decisions. Here, uniformly random means that $\mathrm{Pr}[b=0]=\mathrm{Pr}[b=1]=1/2$, and independent means that the probability $b$ is $0$ or $1$ does not depend on any previous bits sampled or decision made by the algorithm.

Definition. A probabilistic Turing machine (PTM) is a Turing machine with two transition functions $({\delta }_0,{\delta }_1)$. For all $x\in \{0,1{\}}^{\ast }$, a PTM $M$ on input $x$ does the following.

• For each step of the computation, $M(x)$ samples $b\stackrel{\$}{\leftarrow }\{0,1\}$.
• Execute ${\delta }_b$.

We let $M(x)$ denote the random variable corresponding to PTM $M$’s output on input $x$ given the random choices it makes during its execution. We say that $M$ runs in time $T$ if for all $x$, $M(x)$ halts in $O(T(|x|))$ steps for any set of random choices.

Deciding Languages: PTMs vs NTMs

As stated above, PTMs and NTMs are not the same. Recall that for NP, we say that an NTM $N$ accepts a string $x$ if there exists an execution of $N(x)$ such that $N(x)=1$. Similarly, for coNP, the NTM $N$ accepts string $x$ if all executions of $N(x)$ satisfy $N(x)=1$. Above, these definitions are quantified with respect to any possible non-deterministic choices the NTM $N$ make during its execution.

Now, for a PTM, say $M$, we can similarly define it with respect to the number of accepting paths. That is, we can say that $M$ accepts a string $x$ if some fraction of all executions $M(x)$ satisfy $M(x)=1$. For example, we can specify that if $M(x)=1$ for at least half of all computation paths (that is, looking at every possible set of computation paths $M$ could take using randomness), then we say $M$ accepts $x$. This gives us a natural definition for deciding a language with a PTM.

Definition. Let $L$ be a language and $T:\mathbb{N}\to \mathbb{N}$ be a function. We say that a PTM $M$ decides $L$ in time $T$ if $M(x)$ halts in time $O(T(|x|))$ for any $x$ and $$\mathrm{Pr}[M(x)=L(x)]\ge 2/3,$$ where the probability is taken over the random choices made by $M$ on input $x$, and $L(x)=1$ if $x\in L$ and $L(x)=0$ if $x\notin L$.

In the above definition, decidability with respect to a PTM has two-sided error, meaning that $M(x)$ outputs correctly with probability at least $2/3$ for both $x\in L$ and $x\notin L$. This leads us to our first probabilistic complexity class: BPP.

Definition. For a function $T:\mathbb{N}\to \mathbb{N}$, we say that a language $L\in \mathbf{BPTIME}(T)$ if there exists a PTM $M_L$ which decides $L$ in time $T$. The complexity class BPP is defined as $$\mathbf{BPP}=\bigcup _{c\in \mathbb{N}}\mathbf{BPTIME}(n^c).$$

Note that BPP is still a worst-case class since we require deciding languages on a PTM in strict polynomial time.

BPP vs. Other Classes

Intuitively, like how NP was the non-deterministic analogue of P, BPP is the randomized analogue of P. In fact, we know that $\mathbf{P}\subseteq \mathbf{BPP}\subseteq \mathbf{EXP}$. To see the first inequality, notice that every deterministic algorithm is a randomized algorithm that uses no randomness (e.g., you just pick ${\delta }_0={\delta }_1=\delta$, where $\delta$ is the transition function of the DTM). For the second inequality, suppose that $M$ is a PTM running in time $O(n^c)$. Then $M$ has $2^{O(n^c)}$ possible computation paths (each step picks 0/1 with equal probability). So we can construct a machine $M^{\prime }$ running in time $O(2^{n^c})$ which enumerates all possible computation paths of $M$ and outputs $1$ if and only if at least 2/3 of the paths are accepting.

Now, it is unknown whether $\mathbf{BPP}⊊\mathbf{NEXP}$ or if $\mathbf{BPP}=\mathbf{P}$. For the second statement, most complexity theorists actually believe that $\mathbf{BPP}=\mathbf{P}$, but this is a topic we will not cover in this course.

This is not the last time we will see how BPP relates to other classes; we will return to this topic in later lectures.

Alternate Definition of BPP

Taking inspiration from the above outline of $\mathbf{BPP}\subseteq \mathbf{EXP}$, we can give an alternative definition of BPP similar to the certificate definition of NP.

Definition. A language $L$ is in the class BPP if there exists a polynomial-time deterministic Turing machine $D$ and a polynomial $p$ such that for all $x$, $$\underset{r\stackrel{\$}{\leftarrow }\{0,1{\}}^{p(|x|)}}{\mathrm{Pr}}[D(x,r)=L(x)]\ge 2/3.$$

3 Examples of BPP Algorithms

We’ll now give 3 examples of randomized algorithms.

Finding the Median

Given a set of $n$ numbers $S=\{a_1,\dots ,a_n\}$, the median of $S$ is the number $x$ such that $x\ge s$ for $\lfloor n/2\rfloor$ elements $s\in S$ and $s\le t$ for $\lfloor n/2\rfloor$ elements $t\in S$. Stated as a decision problem, you’d be given $(S,m)$ and would need to decide if $m$ is the median of $S$.

There is an easy deterministic algorithm that requires $O(n\log (n))$ time to check.

1. Sort $S$ and obtain $S=(s_1,\dots ,s_n)$, where $s_i\le s_{i+1}$ for all $i$.
2. If $n$ is even, set $x=(s_{n/2}+s_{1+n/2})/2$ (more generally, any number in the range $[s_{n/2},s_{1+n/2}]$). If $n$ is odd, set $x=s_{\lceil n/2\rceil }$.

This takes $O(n\log (n))$ time since we sort $S$.

Now there is a $O(n)$ time deterministic algorithm for finding the median and, more generally, the $k$th smallest element. However, the algorithm is highly non-trivial.

To contrast, we will give an $O(n)$ time simple randomized algorithm to find the $k$th smallest element, where an element $t\in S$ is the $k$th smallest if $k-1$ $s\in S$ satisfy $s\le t$ and $n-k-1$ $s^{\prime }\in S$ satisfy $t\le s^{\prime }$. In particular, setting $k=\lfloor n/2\rfloor$ gives us the median problem. Let $S=\{a_1,\dots ,a_n\}$.

• Find$k$thElement$(k,S)$:
  1. Pick $i\stackrel{\$}{\leftarrow }[|S|]$. Set $x=a_i$.
  2. Scan $S$ and let $m$ denote the number of elements $s\in S$ such that $s\le x$.
  3. If $m=k$, output $x$.
  4. If $m>k$:
     • Create set $S^{\prime }=\{c\in S|c\le x\}$.
     • Run Find$k$thElement$(k,S^{\prime })$.
  5. If $m<k$:
     • Let $T=\{c\in S|c\ge z\}$.
     • Run Find$k$thElement$(k-m,T)$.

We now give the intuition on why this algorithm runs in $O(n)$ time for initial list size $n$.

1. The deterministic parts of the algorithm run in linear time.
2. The expected sizes of $S^{\prime }$ and $T$ can be shown to be at most $9n/10$. That is, with good/high probability over the random choice of $i$, $|S^{\prime }|\le 9n/10$ and $|T|\le 9n/10$ (where the “$n$” is updated every recursive call to mean $|S|$).
3. Together, this implies that if $T(n)$ is the runtime of the algorithm, then $T(n)=O(n)+T(9n/10)$, which can be shown to imply that $T(n)=O(n)$.

Polynomial Identity Testing

Polynomial identity testing is very common in many areas of theoretical computer science and cryptography. Consider an $n$ variate polynomial with integer coefficients $p(X_1,\dots ,X_n)\in \mathbb{Z}[X_1,\dots ,X_n]$ We define the degree of $p$ to be $\mathrm{deg}(p)={\sum }_{i=1}^n\mathrm{deg}(X_i,p)$, where $\mathrm{deg}(X_i,p)$ is the largest degree of $X_i$ in $p$. For example, if $p(X_1,X_2,X_3)=X_1^{100}+X_2^2{X}_3^{50}+X_1^3{X}_2^5{X}_3^7$, then $\mathrm{deg}(p)=100+5+50=155$.

The most natural identity to test is whether $p$ is the identically zero polynomial. That is, $p(y_1,\dots ,y_n)=0$ for all $y_1,\dots ,y_n\in \mathbb{Z}$. Note that this isn’t even checkable in polynomial time. One can give a $p$ with a small description but, once expanded, contain $O(2^n)$ terms! An example of one such polynomial is $p(X_1,\dots ,X_n)={\prod }_{i=1}^n(c_i-X_i)$ for some constants $c_i\in \mathbb{Z}$.

However, there is a probabilistic algorithm which can efficiently check if $p$ is identically zero, assuming that evaluating $p$ at any single point is efficient (i.e., polynomial-time).

$\mathbf{ZeroCheck}(p)$:

1. Sample $k\stackrel{\$}{\leftarrow }[2^{2n}]$.
2. Sample uniformly random ${\alpha }_1,\dots ,{\alpha }_n\stackrel{\$}{\leftarrow }[10\cdot 2^n]$.
3. Check if $p({\alpha }_1,\dots ,{\alpha }_n)\mathrm{mod}k=0$. Output $1$ if this check passes, and $0$ otherwise.

Observations. Let $y=p({\alpha }_1,\dots ,{\alpha }_n)$.

1. If $y=0$ then $y\mathrm{mod}k=0$ for any $k$.
2. Suppose that $y\ne 0$. We want to analyze the probability that ${\mathrm{Pr}}_k[y\mathrm{mod}k=0|y\ne 0]$. This is equal to the probability that $k$ divides $y$. Suppose that $y$ has prime factors $p_1,\dots ,p_{\ell }$. We can upper bound the probability that $k$ divides $y$ by the probability that $k$ is any one of these prime numbers. By the Prime Number Theorem, the number of primes in $[2^{2n}]$ is at least $2^{2n}/2n$. Now, one can show that $\log (y)\le 5n\cdot 2^n=o(2^{2n}/2n)$. This implies that the number of elements of $[2^{2n}]$ not equal to $p_1,\dots ,p_{\ell }$ is at least $2^{2n}/4n$. So, with probability at least $\delta =1/4n$, the randomly chosen $k$ will not be equal to any of $p_1,\dots ,p_{\ell }$. This gives us $$\underset{k}{\mathrm{Pr}}[y\mathrm{mod}k=0|y\ne 0]\le (1-1/4n).$$
3. Now suppose that $y=0$ but $p$ is not the identically zero polynomial. This means that ${\alpha }_1,\dots ,{\alpha }_n$ is a root of the polynomial $p$. By the Schwartz-Zippel lemma, this implies that $$\underset{{\alpha }_1,\dots ,{\alpha }_n}{\mathrm{Pr}}[y=0|p\ne 0]\le \frac{\mathrm{deg}(p)}{|[10\cdot 2^n]|}.$$

All together, these observations imply that $$\begin{array}{rl}\mathrm{Pr}[\mathbf{ZeroCheck}(p)=1|p\ne 0]& \le \underset{k}{\mathrm{Pr}}[y\mathrm{mod}k=0|y\ne 0]+\underset{{\alpha }_1,\dots ,{\alpha }_n}{\mathrm{Pr}}[y=0|p\ne 0]\\ & \le \left(1-\frac{1}{4n}\right)+\frac{\mathrm{deg}(p)}{10\cdot 2^n}.\end{array}$$

Verifying Matrix Multiplications

Let $p\in {\mathbb{Z}}^{+}$ be a prime number and let ${\mathbb{Z}}_p$ be the set of integers modulo $p$. Fix three $n\times n$ matrices $A,B,C\in {\mathbb{Z}}_p^{n\times n}$. We want to decide if $C=A\cdot B$.

The fastest known matrix multiplication algorithm runs in time $O(n^{2.37286})$ (and, I believe, is not possible to be run on a real computer because the hidden constant is enormous). There is also the trivial algorithm which takes $O(n^3)$ time; here, we’re actually counting the number of operations, so this algorithm takes roughly $O(n^3)$ multiplications and additions over ${\mathbb{Z}}_p$.

We can use randomness to get an $O(n^2)$-time algorithm to verify if $C=A\cdot B$. The algorithm operates as follows.

1. Sample $r\stackrel{\$}{\leftarrow }{\mathbb{Z}}_p$.
2. Create vector $x=(1,r,r^2,\dots ,r^{n-1}{)}^{\top }\in {\mathbb{Z}}_p^{n\times 1}$.
3. Check if $Cx=A(Bx)$. Output $1$ if and only if this check passes.

Notice that computing $Cx$ takes $O(n^2)$ time. Similarly, computing $y=Bx$ takes $O(n^2)$ time, and $z=Ay$ also takes $O(n^2)$ time, so the algorithm runs in $O(n^2)$ time.

Now, if $C=AB$, then $Cx=A(Bx)$ for any choice of $x$, and thus with probability $1$ if $C=AB$ then $Cx=A(Bx)$. What if $C\ne AB$? This basically reduces to $n$ polynomial equality checks (or, equivalently, $n$ zero-checks). Let $C_i$ be the $i$th row of $C$. Then, if we set $c=Cx$, we have that $c_i=⟨C_i,x⟩$. Recall that $x=(1,r,r^2,\dots ,r^{n-1})$, so this inner product looks like $c_i={\sum }_{j=1}^n{C}_{i,j}{r}^{j-1}$, which looks exactly like evaluating a degree $n-1$ polynomial, say $C_i(X)={\sum }_{j=1}^n{C}_{i,j}{X}^{j-1}$, at a random point $r$.

Now, if we set $M=A\cdot B$, suppose that in row $i$, we have $C_i\ne M_i$. Then, our random check $r$ is actually testing the following: $$\underset{r}{\mathrm{Pr}}[C_i(r)-M_i(r)=0|C_i\ne M_i]\le \frac{n-1}{p}.$$ In particular, the above is true if $C_i$ and $M_i$ differ on a single entry! So this shows that $$\underset{r}{\mathrm{Pr}}[Cx\ne A(Bx)|C\ne AB]\ge 1-\frac{n-1}{p}.$$

One-Sided Error

Interestingly, the above algorithms all have stronger guarantees than just BPP gives. Finding the median always outputs the correct answer, but runs in expected linear time. Both polynomial identity testing and verifying matrix multiplications will answer “YES” with probability $1$ when the polynomial given as input is identically zero and when $AB=C$ for the given matrices, but both of these algorithms will output “YES” with small probability (not equal to $0$) when the input does not have the correct form.

Building on the latter two algorithms, this lets define two “stronger” classes than BPP which have one-sided error.

Definition. The class $\mathbf{RTIME}(T)$ is the set of all languages $L$ that are decidable by a PTM $M$ running in time $T$ such that for all $x$ $$\begin{array}{rl}x\in L& ⟹\mathrm{Pr}[M(x)=1]\ge 2/3;\\ x\notin L& ⟹\mathrm{Pr}[M(x)=0]=1.\end{array}$$ We define the class RP as $$\mathbf{RP}=\bigcup _{c\in \mathbb{N}}\mathbf{RTIME}(n^c).$$

Naturally, we have the co-class of RP, called coRP defined as $\mathbf{coRP}=\{L:\overline{L}\in \mathbf{RP}\}$. Equivalently, it is also a one-sided error class with the opposite guarantees as RP.

Definition. The class $\mathbf{coRTIME}(T)$ is the set of all languages $L$ that are decidable by a PTM $M$ running in time $T$ such that for all $x$ $$\begin{array}{rl}x\in L& ⟹\mathrm{Pr}[M(x)=1]=1;\\ x\notin L& ⟹\mathrm{Pr}[M(x)=0]\ge 2/3.\end{array}$$ We define the class coRP as $$\mathbf{coRP}=\bigcup _{c\in \mathbb{N}}\mathbf{coRTIME}(n^c).$$