Two case studies that show underlying architecture keeps Python projects slow.
Python, a familiar and easy to implement language, is used to develop a huge proportion of emerging AI projects; projects in which speed means making up for limited capabilities by spending on infrastructure. Python’s dynamically typed, interpreted architecture is known to have performance problems compared to C, C++, and Rust.
The community has done its best to resolve Python’s speed problems through recent updates and the use of key libraries that move heavy processing into compiled languages. NumPy moved the math into C. TensorFlow and PyTorch moved it into CUDA. ONNX gave everyone a portable format so a model could be trained once and run anywhere. Every one of these was sold, correctly, as a performance fix.
Put them together and the story most teams tell themselves is that Python’s speed problem got solved somewhere in the last decade — the interpreter may still chug along, but most work is done on faster code. Haevek has run two recent benchmarks, both on live client data, which say otherwise; Python still has performance problems.
What “optimized” leaves out
NumPy, TensorFlow, and PyTorch all optimize the same thing: the arithmetic. Matrix multiplies, convolutions, tensor ops — all of it compiles down to C, C++, or CUDA kernels that run outside the Python interpreter. That part of the story is true, and it’s why “Python is basically as fast as C now” became a defensible thing to say out loud in an engineering review.
But Python is still in the loop, and its drawbacks are still dragging down workloads. Every call into a compiled kernel still has to be dispatched from Python. Data still has to move across the boundary between Python objects and native memory. Pre- and post-processing, control flow, orchestration between calls — all of that still runs one Python bytecode instruction at a time, on every request, in every pipeline. If you just measure the kernel in isolation — a matrix multiply, a forward pass, a single vectorized call — you will see a huge improvement in performance, but you will miss how much of that performance savings is then given back to the architecture.
Most benchmarks never catch this, because most benchmarks measure the kernel in isolation, not the full path a production system actually takes.
In two recent projects, Haevek had the chance to see just what a Rust-based solution can do on an identical workload to optimized Python. One was ONNX, one a math-intensive algorithm. Measuring the full path, the gap is glaring.
Benchmark one: an ONNX model, run two ways
We took a production ONNX model and ran it two ways on identical hardware: once through the standard PyTorch runtime, and once executed natively inside Falcon (built in Rust), where inference runs inline in the same compiled job as the rest of the pipeline rather than as a separate served step.
ONNX exists to solve exactly the problem described above: export a trained model once, in a portable graph format, and run it anywhere without dragging the training framework along. Teams treat the export step as the optimization — once trained, dump the model into a static graph and strip out whatever Python overhead existed during training.
“Running” that ONNX model in production, though, usually still means loading it back into a Python process and executing it through PyTorch’s runtime, with the surrounding pipeline — pre-processing input, batching requests, post-processing output — written in Python and dispatching each inference call one at a time.
When we ran the model in Falcon with the same weights, same hardware, and generating the same output, we got 1.2x the performance of PyTorch.
Falcon ran the same model 1.2x faster than PyTorch — same weights, same hardware, same output.
Why Falcon is faster
The model itself didn’t get any faster; it’s the identical graph in both cases. What changed is everything Python was still doing around it:
- Dispatch overhead of crossing into the native runtime,
- Python-side object creation and orchestration,
- Conversions and memory copies required by the surrounding pipeline,
- Invoking PyTorch inference (a distinct runtime operation not compiled into the job).
Falcon has AI inference compiled into the compute layer itself — a model runs in the same job that ingested and transformed the data, with no separate serving hop to cross.
With Falcon, the AI inference can be compiled into the job. That’s optimization.
Benchmark two: rewriting the math itself
Our second example is algorithm execution, not model serving. Python has to do twice the trips from memory to cache to complete the task, seriously dragging down the application.
The project is built around Fast Approximated Power Iteration (FAPI) subspace tracking, a signal processing technique for continuously tracking the dominant subspace of a streaming signal. It shows up in adaptive filtering, source localization, and frequency estimation. FAPI is used when a system needs to update its understanding of a signal in real time as new samples arrive without recomputing an eigen decomposition from scratch on every step.
The algorithm is recursive. It keeps a basis matrix and revises it at every timestep against new signal input, meaning each timestep and update of the basis matrix depends on the previously calculated state. A native Python approach would take the shape of a for loop over the matrix, applying the changes per row. The usual next step to increase Python performance is to use NumPy. NumPy can vectorize the operations applied to the matrix using BLAS (Basic Linear Algebra Subprograms) that are compiled low-level algorithms typically written in C or Fortran.
The speed comes down to memory traffic. In the benchmark, we evaluated a matrix that is 1 million dimensions and rank 10 utilizing 32-bit floating-point numbers. They require about 40 MB to store. A matrix this size does not fit into cache on most CPUs, so every iteration results in a full round trip to main memory. The operations applied to the whole matrix are equivalent to 2 multiply-adds. For a modern CPU, this is almost negligible. The runtime and efficiency are set by how many times that matrix needs to move across the memory bus.
The Python implementation makes two of those trips per update. One writes the correction from the previous step into the matrix. The other reads the matrix back and measures the new input signal against it.
The Rust implementation implements both operations in a single pass. It is a single loop over the rows and computes the corrected value, stores it, and immediately reuses it to accumulate the projection while the value is still in a register.
The Python implementation makes two of those trips per update. The Rust implementation implements both operations in a single pass.
NumPy’s app is a drag on performance. NumPy’s unit of work is the entire array. Each call takes an array, walks it end to end, and returns a new array. Python is simply unable to represent the fusing of operations the same way that Rust can. We tried several approaches to help NumPy vectorize operations, including Python’s Numba library that will JIT compile a function.
Adding cores does not close the gap either. A threaded BLAS kernel does in fact parallelize a single call by splitting rows across a thread pool, performing the operation in each thread, and then joining all values back together to hand back off to the single threaded Python interpreter. The parallel boundary is never wider than a single NumPy operation, which results in more memory contention and thread spawning and collapsing overhead.
Threading as a solution, with limited improvement
At a million dimensions and rank 10, Python takes 11.2ms per step on a single core. With 8 cores, it takes 10.1ms per step. When we parallelize the full Python implementation and shard the matrix across 8 cores, we were able to get the execution time down to 5.3ms per step. Falcon with a single core was 3.1ms per step and 0.86ms per step with 8 cores using the same sharded matrix approach. This is 3.6x faster core for core and 6x faster at 8 cores.
Falcon on a single core is still 1.7x faster than Python on 8 cores. That is a more useful number for anyone paying for the infrastructure: the same workload, faster, served on a fraction of the cores.
Benchmark of FAPI performance on production data
| Configuration | Milliseconds per step |
|---|---|
| Python, 1 core | 11.2 |
| Python, 8 cores | 10.1 |
| Python, 8 cores, sharded | 5.3 |
| Falcon, 1 core | 3.1 |
| Falcon, 8 cores, sharded | 0.86 |
Findings
-
Little advantage to adding cores for Python
-
Falcon on a single core is 2.3 times faster than Python on 8 cores and outperforms even sharded Python.
The gap is larger than the ONNX case for a specific reason: there, the fix was to stop paying Python's overhead around a kernel that was already in the right shape; here the kernel shape is the problem. A faster library call does not fix a kernel boundary in the wrong place. When the answer lives further down the stack – outside native Python’s reach and beyond the interfaces that NumPy exposes – “just vectorize it” has a ceiling. The mismatch stops being a rounding error and becomes most of the runtime.
Two different problems, one shared tax
These are two distinct mechanisms. In the ONNX case, Python is a thin wrapper around math that’s already optimized — the tax is dispatch and orchestration overhead layered on top of good arithmetic. In the algorithmic subspace tracking case, the workload itself can’t be handed off to a vectorized kernel at all — the tax is the interpreter running the actual computation, one step at a time.
Different causes, same result: hardware doing less useful work per cycle than it’s capable of, because Python is sitting between the hardware and the workload either way.
NumPy made the math fast. It didn’t make the loop around it disappear.
What this actually costs
A 1.2x or 6x implementation tax won’t show up on a performance dashboard as “Python is slow.” Instead, it shows as more instances provisioned to hit a latency target, more replicas added to an inference service under load, or more cores allocated to a streaming pipeline that’s falling behind its SLA. Each of these consequences is a real infrastructure line item, and in each case the team paying for it will describe the problem as needing more capacity — not as needing less interpreter overhead. The interpreter overhead tax is invisible until someone benchmarks the full path instead of the kernel.
That is the pattern across both examples here; the extra hardware isn’t buying more computation. The hardware compensates for delays built into the architecture — cycles spent on dispatch, object handling, and single-threaded iteration instead of on the workload itself. Teams are provisioning (and paying) for resources they wouldn’t need if the same job ran on an engine that didn’t have an interpreter in the critical path to begin with.
Where Falcon fits — without a rewrite
Neither of these benchmarks required re-platforming any upstream or downstream systems or migrating the data estate. Falcon reads the same inputs, writes the same outputs, and runs on the same Kubernetes clusters teams already operate. The key difference is that inference and algorithmic pipelines execute inline and compiled within the same job as everything else, rather than as Python code orchestrating calls into someone else’s runtime. That’s true whether the workload is a served ONNX model or a hand-written signal processing algorithm with no library to lean on. If it runs as a loop or a call graph today, it’s a candidate for the same comparison.
Get started
Both of these benchmarks were developed and run by Haevek on behalf of customer workloads, not synthetic microbenchmarks. We hold every benchmark we publish to a high standard. We ask, "does it represent a real-world benefit?" If you have an inference service or an algorithmic pipeline that’s provisioned larger than it feels like it should be, the Falcon Test Flight runs it side by side against your current stack, on your own hardware, so you can see the gap yourself before any commercial conversation. Contact Haevek to learn more or sign up for a Test Flight.