Compute and storage costs are easy to calculate. The challenge is justifying the spend, especially at scale when compute costs start outweighing the value created by the workloads.
Every data leader can spend money on compute. Very few can defend what that investment returned.
The issue is structural. As you move from testing to production, the compute and storage costs increase in tandem with workloads. The problem is exacerbated by massive AI workloads that can grow in perpetuity and cause costs to compound quarter over quarter.
Lenny Ardiles has spent more than twenty years building data pipelines, from four-person development teams to Amazon and Meta. On The Compute Layer he described the problem in terms most finance conversations never reach.
This article is for the data and AI infrastructure leader whose spend is now under finance scrutiny, and for the CFO or CTO being asked to approve a number nobody can forecast.
Workload costs are a simple function of what you spend on compute and storage.
You can see what your compute costs are, what your storage costs are.
Justifying the spend is the challenge. In Lenny's framing, the difficulty is determining what it will actually cost to build, maintain, and run a system that delivers on the business initiative it was built to achieve.
The business reason is the driver. Mapping that to all of the use cases becomes challenging.
Lenny is equally direct about the standard remedy. Tagging and labeling assets does help, but that becomes its own maintenance headache. Ultimately, it does not create the justification; tying costs to individual pipelines does not tell the story of where the spend is going. Real work is running through a series of interconnected and overlapping steps. That does not come through in the costs of individual pipelines; the attribution breaks exactly where the spend concentrates.
The consequence is not confusion. It is arguments lost because costs cannot be justified.
When spend is precise and value is soft, the data platform becomes the easiest line item to cut. Projects get trimmed to control a number that nobody can defend, and the team gets blamed for overruns it did not choose.
The pricing model makes this worse. Usage-based billing means every rerun, every new workload, and every spike lands on the invoice, so the number moves for reasons that have nothing to do with whether the workloads created value for the business in a given quarter. Finance cannot forecast it. Procurement stalls. Quarterly true-ups arrive as surprises.
Lenny adds a second cost most models leave out entirely. Comparing platforms, he notes that one gets a team running the same day with low startup cost and, by his understanding, the highest compute pricing. Another offers a complete toolset and deep fine-tuning to optimize spend, but teams rarely go live in a day and more engineers are typically involved.
With all of those costs, there needs to be more expertise on how to do that.
Optimization capacity is a staffing decision. It is almost never priced next to the platform, and it lands on the same lean team already responsible for keeping the estate performant.
The response when a workload slows is very often to add compute. That works, and it scales cost linearly with volume, which means the bill grows faster than the value the data returns.
A better option? Find a solution that runs the same workloads at a fraction of the compute costs. Efficiency is the lever.
Lenny advocates for proven efficiency measures that can reduce costs.
The solution for getting costs under control is, surprisingly, we keep rediscovering the same things that we've done for ten, fifteen, twenty years.
His examples are unglamorous. Batch and chunk work instead of round-tripping every row. Cache and reuse answers where questions bucket into a smaller set. Put quality checks at every step so an open-ended process narrows into a reliable one. None of that is novel; they are efficiency measures that fell off the radar when compute felt cheap.
He applies the same lens to AI spend directly.
Tokenmaxxing is just spending as many tokens as possible. That's almost by definition not efficient.
Lenny understands how we got here, unable to explain where costs are going and where value comes from. Teams and enterprises are still in the learning phase when it comes to the massive compute and storage costs created by AI-driven projects. He also expects organizations to become better able to measure the spend, with the cost of the AI as the denominator, against newly created value.
If efficiency is the argument, it has to be provable. Lenny's standard here is a key insight, and it comes from three years measuring marketing campaigns rather than infrastructure.
Anything that we couldn't measure and prove to have a positive value, even if it did have a positive value, we would put a zero benefit on it. And that made sure that we would do proper experimentation. We would test how it [the marketing campaign] works with it [the feature], how it works without it. So it's not a guess or a gut feeling.
That is a demanding validation process for marketing campaigns, and it is also the right one for technology projects. It explains why vendor benchmarks rarely move an enterprise decision. The number that matters is the one measured on your workloads, with and without the change.
If Lenny's argument holds, three things follow for the layer underneath the stack.
Haevek Falcon is lakehouse augmentation, not a migration. The warehouse, the governance, the catalog, the downstream targets, and the user experience all stay where they are. Only production compute changes.
Compute cost is not a question mark because the numbers are hidden. It is a question mark because the conversion from spend to business value was never built, and because the pricing model punishes you for running more.
The fix is not a bigger budget or a better spreadsheet. It is running the same work on less, pricing it like infrastructure, and proving the difference on your own workloads.
Pilot the Falcon Test Flight for two to four weeks in your environment, with your data, benchmarked side by side. No data moved, no lock-in. The results show exactly what the same workload costs on a fraction of the compute.