dg-publish: true
202405201927
Status: #idea
Tags: Probability, NPTEL ~ Probability Foundation for Electrical Engineers

Discrete Probability Spaces

When we are dealing with a countable Sample Space $\Omega$, then we will be operating in the realm of discrete probability spaces. $\Omega$ could be infinite or finite.

They are nice because they are simple and intuitive to work with.

When dealing with those cases, it is possible to assign a probability to all elements of the sample space since $2^{\Omega }$ will be a countable set. Thanks to that all subsets of $\Omega$ are events, and therefore we can assign a probability to any element of $\Omega$. Therefore in discrete contexts, our Probability Spaces will defacto be defined as follows $(\Omega ,2^{\Omega })$ because WE CAN preserve the full richness of the space, we can make analysis as granular as we want, and therefore WE DO.

In the context of Continuous Probability Spaces, we resort to Borel Sets and Lebesgue Measures because we have to, not because we want to. After all keep in mind that the sample space represents the set of outcomes relevant to our target, why would we choose to drop valuable data if we do not have to? Exactly, we don't.

So, how do we accomplish this witchcraft?

By assigning probabilities to singleton elements (subsets containing single elements of the sample space) in such a way that:

In such an environment, what is $P(x_1,x_2,x_3)$? Well it is $P(x_1)+P(x_2)+P(x_3)$.

Note that these probabilities can be assigned as haphazardly as we want, I could make a bunch of $x_i$ have probability $0$ and give all the probability bandwidth to some arbitrary $x_{\alpha }$, or I could weight it in such a way that the bigger the index of $x_i$ is, the greater its assigned probability.

As long as it all sum to $1$ in the end, we are gucci.

In practice, unless there's a reason to do otherwise (will vary based on Random Experiment), we generally assume a Uniform Distribution, but this is by no mean an obligation.

Probability Mass Functions (PMF)

Recall how in measure theory a measure is nothing more than a function that maps elements of our $\sigma -$algebra (in this case the power set of $\Omega$) to the space $[0,1]$.

A Probability Mass Function based on the theory covered in Probability Spaces, is nothing more than the measure that maps those singleton subsets of the $\sigma -$algebra to the right values. We interpret this measure as the likelihood of our random experiment to yield a given event.

This is the name we give to a Probability Density Functions when operating in a discrete probability space.

Important Consideration

While at times due to how we write things, we can mistakenly think that we are assigning probabilities to elements of the Sample Space. This is categorically false. While an harmless error most of the time (and in fact the way everyone that doesn't learn the foundations of probability learns it,) probability are assigned to elements of the $\sigma -$algebra, not to element of $\Omega$. This might sound like a nitpick, but its not.

While not too big of an issue in Discrete Probability Spaces, it is crucial to understand that when we go to Continuous Probability Spaces, because in the latter it is generally not possible to assign a probability to all subsets of $\Omega$ and therefore all we can do is treat with the $\sigma -$algebra and its measurable sets.

This is true in the discrete case as well! But that subtlety can be missed if we're not careful.

Relevant Links

Lecture 7