The Value Pillars of Haevek’s Falcon Compute · Pillar 04: Speed
Haevek Falcon’s modernization of the data compute platform opens new horizons for data-centric organizations.
- Falcon architecture overcomes performance chokepoints underlying Spark and Databricks.
- Typical migrations get 25x performance improvement when moving complex workloads to Falcon.
- Performance improvements don’t just meet SLAs; they open up new opportunities and save budget on compute.
- Proven on the most complex pipelines with the most demanding performance requirements.
“By integrating Falcon into our pipeline, we achieved substantial gains in performance and scalability while reducing overhead.”
Co-Founder · Everlign
The problem
Problem: Traditional architecture creates bottlenecks
Data compute performance targets can be expensive to reach. It would be cheaper to tune pipelines to get more performance, but the available headroom is limited. Spark and Databricks rely on the Java Virtual Machine (JVM) which is limited by two major forces: uncompiled code and the garbage collection method of freeing memory.
Spark, first deployed in 2012, uses the uncompiled code of the JVM and bytecode to allow for varied and changing data types, called dynamic typing. As a result, all their workloads have the same performance issues as the JVM. Configuration can only do so much for performance. It can’t overcome the architecture.
The performance issues lead to strategic roadblocks: lower performance, slower delivery, less frequently sampled data. The project is less competitive.
Configuration can only do so much for performance. It can’t overcome the architecture.
Garbage collection: unscheduled performance demands.
Along with the performance costs of uncompiled code, JVM uses garbage collection to manage memory. Whenever needed (often very inconveniently timed in the middle of a batch process or large streaming traffic) the garbage collection process takes up resources identifying unused variables and freeing up memory. The result is higher memory requirements and unexpected drags on performance when the garbage collection kicks in.
The peaks and valleys of garbage collection.
How we do it
Haevek built the Falcon Compute Data Platform to overcome the limitations of incumbent data pipelines. It overcomes those bottlenecks by using the compiled, modern Rust programming language with excellent type safety and memory management; exactly the problems JVM creates for Spark.
Improvements to memory management in Falcon
| Java and the JVM in Spark | Rust in Falcon |
|---|---|
| Memory allocation automated without specification by developer. | Memory allocation automated without specification by developer. |
| Unused memory remains reserved after variable no longer references it. | Memory immediately made available when a variable goes out of scope. |
| Potentially resource intensive garbage collection occurs beyond the control of the developer or operator. | No scheduled process. Memory is reallocated as soon as it frees up. |
Why Rust
In addition to its performance advantages, the choice of Rust as Falcon’s implementation language means that type errors, missing dependencies, and API misuse are caught at build time rather than at job submission or execution time. It is more efficient to catch those errors earlier in the process, before QA, staging, or production.
Haevek’s solution is the product of decades of domain-relevant experience and many person-years of dedicated engineering effort.
Performance = cost savings, cost savings = performance
For data pipelines, modern technology means that if you have the budget you can buy a certain amount of performance through scaling. But with consumption pricing like cloud infrastructure and as-a-service data solutions, getting faster can make the ROI disappear fast.
Falcon’s architecture provides higher performance, reducing the costs of staying competitive and performant.
Increasing performance means releasing huge amounts of compute budget.
Benchmarks tell the tale
In a recent benchmark test with identical workloads, Falcon was 2.8 times faster when used as a drop-in replacement for PySpark (an easy migration path) and 13.4 times faster in Falcon-native configuration.
1x
Apache Spark (PySpark)
2.8x
Falcon (PySpark drop-in)
13.4x
Falcon (optimized)
| Technology | Configuration | Runtime (seconds) | Relative Speed |
|---|---|---|---|
| Apache Spark (PySpark) | Production configuration | 161 | 1 |
| Falcon (PySpark drop-in) | Equivalent partition count | 58 | 2.8x |
| Falcon (optimized) | Falcon-native job configuration | 12 | 13.4x |
ETL + AI Inference benchmark results. All runs on identical fixed infrastructure.
A new vista
Moving workloads to Falcon opens up new pathways for innovation and budget to make it happen. You can provide rich outputs and run them fast.
New tools to be competitive.
We offer the 2-4 week Falcon Test Flight to pilot Falcon in your environment with your data. No data moved, no lock-in, training included.