202412110952
Status: #idea
Tags: Classification
State: #nascient

Bayes Classifier

This is provably the best classifier possible on data that will achieve the lowest classification rate short of literal divination. It makes use of the Bayes Theorem to compute the posterior probabilities of a class $k$ having generated an observation $X$. It's as good as we get, as a testament of that, the lowest possible classification error attainable is called the Bayes Error. It is optimal under perfect knowledge conditions and is thus often used as a reference of how well a model is performing on toy datasets. It could be seen as a counterpart of the Null Classifier which would always take the laziest/most naive approach to classification giving us the worst error rate that someone actively trying could get.

So what is the catch? Why is is that we have so many classifiers then?
Well, the issue is that the parameters needed to fit it are basically never known and so it serves as a pie in the sky goal that other models try to approximate either directly like in the case of Linear Discriminant Analysis (LDA), Quadratic Discriminant Analysis (QDA), or Naive Bayes, where we approximate both the likelihood and the prior and do something with it.

Or indirectly where we skip the Bayes theorem and try to estimate the posterior probability directly without computing priors like in the case of Logistic Regression, or Neural Networks, support vector machines as well.

Relevant Links