What is a random variable?

A random variable $X$ is nothing more than a tuple containing $p(x)$ distribution, and a range $\chi$. This $x$ in $p(x)$ indexes into $\chi$, but that’s only an indexing property. So…as long as you have some set $\chi$ and a valid probability distribution with the same cardinality, you’ve got yourself a random variable.

• Example

  As an example, we can have $q_i$ be the $\chi$ and $b_i/\sum b_j$ as the $p(x)$. Note how $b$ and $q$ can have no relation. As long as they have the same cardinality, it is possible to make an RV out of them.

If $X$ has more than one variable, it is a joint distribution. When we express things like $P(X)$, we are now talking about a distribution because $X$ isn’t an event. Therefore, we can also write stuff like $P(X | Y)$, which is a conditional probability distribution over two random variables. The chain rule and Bayes rule still applies

PMF, CDF, and PDF

• A PMF is $p(X = x_i)$, which is only a thing for discrete RV's

• A PDF is the PMF of continuous RV's, but this does NOT correspond to the probability\. Rather, it corresponds to the density of the distribution at that point.

• A CDF is $p(x_i < X < X_j)$, which is a summation for discrete RV's and an integral for continuous RV's.

The formal definition of density

So for continuous RV’s , we have the notion of density $f(x)$. When defining the density, we always do it in terms of the CDF $F$.

so this is a great first step when you’re dealing with proofs with density. And of course, remember that

and you can apply functions inside the $p$, like $p(\phi(k) < x) → p(k < \phi^{-1}(x))$.

Same RV, same parameters

Here’s a critical distinction. When we talk about some RV $X$, it has an identity as well as parameters. If we set $Y = X$, this means that every sample of $Y$ is the same as $X$.

However, there’s also the notion of sufficient statistics. If we have $p(Y = y) = P(X = y)$, then we have the same distribution, but the identity of $X, Y$ are not the same. As a consequence, if you draw a sample from X, it may not be the same as the sample from Y.

Distributions over likelihoods

So $X$ is a random variable, and $p(X)$ is also a random variable. This is because $p$ is just a function that takes in something and outputs a number between 0 and 1. Therefore, $p(X)$ is feeding the sample back into its own likelihood fuction. This is perfectly valid, and in fact, this is exactly what entropy does! $E[\log 1/p(X)]$.