Falcon: Modernizing the Platform for Data Applications

Faster, easier, and more affordable data processing

Too often, the promise of new technology comes with disruption to your tech stack, establishing the need for a steep learning curve and placing burden on stretched development teams already working against the clock and within tight budgetary constraints.

We believe that modern enterprise data applications should be able to take full advantage of modern compute infrastructure for cost effective scaling and management. Haevek's Falcon is built to coexist with your technology infrastructure, operating next to solutions like Spark, Big Query, Snowflake, and Redshift without interrupting existing production processes.

What is Falcon?

Haevek's Falcon Compute Platform is a Kubernetes-native distributed compute engine. It replaces Apache Spark and interacts cleanly with Databricks, Snowflake, and other familiar data processing solutions. Falcon can be spun up anywhere Kubernetes runs. Our users get next generation performance at a remarkably lower cost than they previously paid — all while enjoying familiar developer ergonomics.

Haevek out performs Apache Spark and Apache Nifi with a compiled, memory-safe code.

Maximizing the Kubernetes advantage

Spark and equivalent services can run on Kubernetes, but they do not take full advantage of the orchestration available. Spark views Kubernetes as a service provider and struggles to leverage capabilities that are now standard for modern enterprise orchestration. Native Kubernetes benefits like declarative reconciliation, custom resource operators, and workload topology are missing from Spark's open-source baseline. Spark's internal cluster management causes long cold-start times and execution bloat.

We built Falcon from the ground up to run on Kubernetes with native deployment and resource orchestration patterns. This deep integration unlocks a rich, standardized, and familiar infrastructure layer upon which organizations can easily implement observability, access control, governance, and GitOps with anything that plugs into the Kubernetes ecosystem.

Building into Kubernetes means that Falcon can run anywhere Kubernetes can run; in the cloud, on-prem in the data center, and even at the edge. Rather than forcing architects into a cloud or SaaS design, Falcon works within the requirements and constraints of any modern enterprise — even air gapped.

Avoiding the Two JVM Taxes

Spark was written in Java and Scala to run on the Java Virtual Machine (JVM). When this was novel in 2012, it helped Spark establish a new standard of performance and scaling — despite the drawbacks that are built into the JVM structure. In particular, the performance costs of not using a compiled language and the headaches that garbage collection can create for an application. Built in Rust, Falcon eliminates both.

Running compiled code to beat the JVM bytecode tax

The performance costs of JVM and bytecode over a compiled language are well known, but conventional thinking was that compiled languages couldn't provide for heterogeneous data types the way that JVM dynamic typing could. We established a way to do this flexibly with minimal end-user effort. Haevek's solution leverages a collection of essential inventions that form the backbone of a strong patent portfolio.

Spark uses dynamic typing to interpret and execute workloads. That means that type mismatches are surfaced when the application is processing data. Falcon's Dataset API is fully statically typed. Every data type and function signature is rendered and validated at compile time before deployment or execution. This eliminates runtime exceptions due to data typing and function interface errors; missing dependencies and API misuse are caught and solved during build-time, not in production once expensive infrastructure has already been scaled up.

Eliminating the garbage collection tax with rust memory management

Garbage collection (GC) is a type of memory management that requires a scheduled process to identify memory that is no longer referenced. GC does liberate programmers from specifying when to free up memory, but at the expense of additional runtime execution overhead. Garbage collection processes operate on their own schedule, inevitably triggering when those resources are needed for a high-performance task.

Garbage collection will spontaneously spike heap size and compute in Spark's JVM processes.

We used Rust to build Falcon in order to overcome the garbage tax. The life of a Rust object in memory is closely coupled with the variable definition in code; the compiler tracks these lifetimes explicitly. When the variable goes out of scope, the memory is freed up immediately for the next need. Falcon's Rust-based implementation means data processing has lower overhead, more freed up memory, and no surprise garbage collection process to gum up operations.

Flattening Steep Learning Curves

A developer familiar with Spark will recognize a lot of the style and syntax of a Falcon workload.

use falcon_stdlib::prelude::*;

pub struct WordCount;

impl FalconJob for WordCount {
    fn build(ctx: &JobBuildContext) -> impl Dataset {
        ctx.read_prefix("s3://bucket/corpus/")
            .lines()
            .flat_map(|line| line.split_whitespace().map(str::to_owned).collect::<Vec<_>>())
            .map(|word| (word, 1u64))
            .reduce_by_key(|a, b| a + b)
            .write_json("s3://bucket/wordcount-output/")
    }
}

An example word count workload in Rust.

Patterns like flat_map / map / reduce_by_key are identical to Spark projects, just implemented in Rust. Common requirements like ETL, streaming, machine learning, and analytics pipelines are served by Falcon's standard library. Workloads that do not fit into a tabular data model required by SQL query engines can be expressed natively without user-side schema definitions.

For even more ease of adoption, Falcon can operate in PySpark drop-in configuration (the "Falcon Connect" service) for workloads that are Spark Connect-compliant. It takes advantage of core optimizations in the Falcon Compute Engine. In a recent test the PySpark drop-in was nearly 3 times faster. By comparison, the optimized Falcon workload was nearly 14 times faster.

Falcon can be up and running at warp speed. One client went from the launch of a pilot to production workloads in 14 days.

Reducing the Cost of Performance, Easing Implementation

Applications with data-heavy requirements can cost millions per year to operate. Using consumption SaaS models, the cost of operation rises as production scales.

Falcon breaks this pattern and expectation. The objective is peak performance at a much lower cost. When one of our clients moved a workload from Databricks Photon to Falcon, the numbers were impressive.

Real-time streaming infrastructure comparison.

Falcon is faster and easier to set up than the traditional model. No matter how you measure it, Falcon optimizes performance and ease of use while significantly reducing costs.

Test Flight: Deploy faster, easier, and more cost-effectively with Falcon

In just two weeks, you can see how Falcon compares to your existing solution using real data. The Falcon Test Flight is your opportunity to see what implementing Falcon looks like in your environment.

Learn more and schedule a test flight at haevek.com/contact.