The Right Way to Use Multiple Embedding Models

Imagine you have two different models (or sub-networks) in your whole ML pipeline. Both generate a representation/embedding of the input in the same dimensions (say, 200).

Here, many folks get tempted to make them interact. They would:

• compare these representations

• compute their Euclidean distance

• compute their cosine similarity, and more.

The rationale is that as the representations have the same dimensions, they can seamlessly interact.

However, that is NOT true, and you should NEVER do that.

Why?

This is because even though these embeddings have the same length (or dimensions), they are not in the same space, i.e., they are out of space.

Out of space means that their axes are not aligned.

To simplify, imagine both embeddings were in a 3D space.

Now, assume that their z-axes are aligned, but the x and y axes of the first is at an angle to the x and y axes of the second:

Now, of course, both embeddings have the same dimensions — 3.

But can you compare them?

No, right?

Similarly, comparing the embeddings from the two networks above would inherently assume that all axes are perfectly aligned.

But this is highly unlikely because there are infinitely many ways axes may orient relative to each other.

Thus, the representations can NEVER be compared, unless generated by the same model.

I vividly remember making this mistake once, and it caused serious trouble in my ML pipeline.

And I think if you are not aware of this, then it is something that can easily go unnoticed.

Instead, I have always found that concatenation is a much better way to leverage multiple embeddings.

The good thing is that concatenation works even if they have unequal dimensions.

👉 Over to you: How do you typically handle embeddings from multiple models?

Are you overwhelmed with the amount of information in ML/DS?

Every week, I publish no-fluff deep dives on topics that truly matter to your skills for ML/DS roles.

For instance:

• A Crash Course on Causality – Part 1

• A Crash Course on Causality – Part 2

• A Beginner-friendly Introduction to Kolmogorov Arnold Networks (KANs)

• Implementing KANs From Scratch Using PyTorch

• A Practical Guide to Scaling ML Model Training

• 5 Must-Know Ways to Test ML Models in Production (Implementation Included)

• A Beginner-Friendly Guide to Multi-GPU Model Training

• Understanding LoRA-derived Techniques for Optimal LLM Fine-tuning

• 8 Fatal (Yet Non-obvious) Pitfalls and Cautionary Measures in Data Science

• Implementing Parallelized CUDA Programs From Scratch Using CUDA Programming

• You Are Probably Building Inconsistent Classification Models Without Even Realizing

• And many many more.

Join below to unlock all full articles:

SPONSOR US

Get your product in front of 82,000 data scientists and other tech professionals.

Our newsletter puts your products and services directly in front of an audience that matters — thousands of leaders, senior data scientists, machine learning engineers, data analysts, etc., who have influence over significant tech decisions and big purchases.

To ensure your product reaches this influential audience, reserve your space here or reply to this email to ensure your product reaches this influential audience.