Accelerate Pandas 20x using FireDucks

Two of the biggest problems with Pandas is that:

• It always adheres to a single-core computation on a CPU.

• It creates bulky DataFrames.

Moreover, since Pandas follows an eager execution mode (every operation triggers immediate computation), it cannot prepare a smart execution plan that optimizes the entire sequence of operations.

FireDucks is a heavily optimized alternative to Pandas with exactly the same API as Pandas’ that addresses these limitations.

Let’s learn more about it today!

How to use it?

First, install the library:

Next, there are three ways to use it:

1. If you are using IPython or Jupyter Notebook, load the extension as follows:

1. Additionally, FireDucks also provides a pandas-like module (fireducks.pandas), which can be imported instead of using Pandas. Thus, to use FireDucks in an existing Pandas pipeline, replace the standard import statement with the one from FireDucks:

1. Lastly, if you have a Python script, executing it as shown below will automatically replace the Pandas import statement with FireDucks:

Done!

It’s that simple to use FireDucks.

The speedup is evident from the gif below from my personal experimentation:

I haven’t shown the full animation due to size limitations, but Pandas code ran in 12.3 seconds while the FireDucks code ran in 3.5 seconds, ~4x speedup:

As per FireDucks’ official benchmarks, it can be ~20x faster than Pandas and ~2x faster than Polars, as shown below on several queries:

Considering the above benchmarks, FireDucks outperforms Polars on 14 out of 22 benchmarks.

Moreover, another thing that stands in favor of FireDucks is that, unlike Polars, we don’t need to make any code changes.

How does it work?

Whenever %load_ext fireducks.pandas is executed, the “import pandas as pd” statement does not import the original Pandas library, which we use all the time.

Instead, it imports another library that contains accelerated and optimized implementations of all Pandas methods.

This is evident from the image below:

This alternative implementation preserves the entire syntax of Pandas. So, if you know Pandas, you already know how to use FireDucks.

Moreover, unlike Pandas, FireDucks is driven by lazy execution.

This means that transformations do not produce immediate results.

Instead, the computations are deferred until an action is triggered, such as:

• Viewing/printing the data.

• Writing the data to a storage source.

• Converting the data to Python lists, etc.

By lazily evaluating DataFrame transformations and executing them ONLY WHEN THEY ARE NEEDED, FireDucks can build a logical execution plan and apply possible optimizations.

For instance:

• In the following code, df2 is never used. Thus, FireDucks will never read from the CSV. Pandas, however, will load it anyway.

• In the following code, assume df contains 10 columns. However, we are only making use of two of them:

  • Pandas will perform the sort operation on the entire DataFrame, requiring more memory.

  • But since FireDucks is driven by lazy evaluation, it will build the following optimal plan instead to generate the result:

Of course, several other optimizations are involved, which I haven’t covered here, but I hope you get the point.

This way, FireDucks turns out to be much more optimal than Pandas.

That said, FireDucks does support an eager execution mode like Pandas if you prefer to use that. Here’s how to enable it:

One slight limitation is that it is currently available only for Linux on the x86_64 architecture. As per the official docs, Windows and MacOS versions are currently under development, and their respective beta versions will be released soon.

There is, however, a way to use it on Windows, which you can find here.

You can find the code here: Google Colab.

FireDucks documentation is available here: FireDucks docs.

🙌 A big thanks to FireDucks, who very kindly partnered with me on this post and let me share my thoughts openly.

👉 Over to you: What are some other ways to accelerate Pandas operations in general?

SPONSOR US

Get your product in front of 87,000 data scientists and other tech professionals.

Our newsletter puts your products and services directly in front of an audience that matters — thousands of leaders, senior data scientists, machine learning engineers, data analysts, etc., who have influence over significant tech decisions and big purchases.

To ensure your product reaches this influential audience, reserve your space here or reply to this email to ensure your product reaches this influential audience.