The Mathematical Intuition Behind the Curse of Dimensionality

For the longest time, I couldn’t get my head around the curse of dimensionality.

Almost every ML/DS professional I know has at least heard about it.

However, very few can explain the deeply rooted mathematical foundations behind it and its implications.

If you are one of them, I decided to cover it in the latest deep dive: A Mathematical Deep Dive Into the Curse of Dimensionality.

I vividly remember when I first heard about it, I was pretty confused and couldn’t grasp what it meant.

Also, dimensionality, in itself, didn’t seem like a bad thing at all. In fact, having more dimensions meant I had more features in my dataset, more data to train my model, and potentially more complex insights to extract.

Most resources/posts/tutorials never cared to dive into the mathematical details explaining it (which I always cared about) and instead assumed it as a given.

And I never understood why they did that.

Think about it.

When Richard Bellman, in 1961, first coined it, he had no idea about it, correct?

No one else in the machine learning community knew about it either.

Thus, I expected that an ideal answer should formulate it from scratch and with valid reasoning, replicating the exact thought process when it was first formulated.

As you will find out, the curse of dimensionality actually has solid mathematical reasoning behind why it is called so, due to the strange phenomena that appear in high dimensional dataset.

It affects a range of areas in machine learning and data science, from distance metrics to model generalization.

So, in this deep dive, I intend to leave you with an explanation, uncovering every piece of mathematical intuition I have ever learned about it.

This is a must-read if you are always curious to dive into the origin stories of popular ideas.

I have also seen this talked about in several DS interviews, so knowing the underlying details will be immensely useful to you.

Read here: A Mathematical Deep Dive Into the Curse of Dimensionality.

I hope you will learn something new today.

Have a good day!

Avi