Many teams discover the cost of their data and AI workloads when they receive the invoice. CloudZero's Larry Advey explains how continuously expanding compute power drives up costs and slows innovation, and why efficiency has to be measured and designed in from day one.
Why data and AI efficiency has to be designed in
Data and AI costs continue to grow as more and more compute power is needed to sustain AI-powered applications. This creates a widening gap between what teams spend and the value realized from the dollars used.
The moment of truth often arrives the same way; the bloated invoice shows up and the CFO asks the CTO, “Why are we paying so much for Spark?” Then the CTO confronts the dev team with an even harder question, “What did we get for all that money?”
Larry Advey is the Director of AI Economics at CloudZero, a member of the FinOps Foundation's Technical Advisory Council, and a FOCUS maintainer. His background runs through engineering, architecture, data, FinOps, and infrastructure, and his work today centers on AI ROI and unit economics. On Haevek's series, The Compute Layer, he described a pattern he sees across companies of every size:
“At the outset of a project, cost is often a secondary concern and optimization efforts only begin after the cost overruns have already occurred.”
– Larry Advey, CloudZero
Data and AI infrastructure leaders, CTOs, and FinOps leaders who share the budget with them, seek methods for controlling costs without compromising strategic ambitions. This article paints a picture of what a better approach looks like.
Most teams do not anticipate elevated compute cost
Advey shares a blunt assessment of how most teams discover how much they have spent on compute.
"They spent a bunch of money on data, AI, infrastructure, whatever it is, and then the invoice arrives and then boom, there's sticker shock. 'Why did this cost so much money?' And then it follows with, 'Well, what do we actually get from this?'"
Part of the problem is visibility. Some mature organizations forecast deeply and understand their AI and data infrastructure costs. In Advey’s experience, teams deploying the workloads are usually the last to see the cost.
"The teams that are actually deploying generally don't see the cost detail as quickly as they should. They're just challenged to push product and push features and get things done. Cost is second thought usually."
This is a structural failure, not one of effort. Teams are measured on shipping, so they work hard to deliver. When cost data arrives after the work is done, it is too late to go back and optimize.
🚀 Bring your biggest pipeline. Run it on Falcon in your own environment with a free 2–4 week Test Flight, and see what the same workload costs on a fraction of the compute.
If optimization starts after the big invoices arrive, you are already falling behind
This reactive approach allows unnecessary sunk costs to build over time before anyone thinks to look for savings. Advey notes that when revenue looks strong and costs are acceptable, nobody spends time looking for cost savings. In Advey’s words, a lot of it "is being vibed." The reckoning comes when revenue plateaus or falls and costs keep rising with the need for more compute power.
"Then you get into, 'Well, gosh, I wish we would've known this from the beginning.' If we understood our cost per transaction or cost per action or user, and we could have been mindful of that as a KPI from day one."
Throwing more compute at the problem doesn't scale
When more compute power is needed to handle the data and workloads, the default response is to add capacity. Advey calls that the most common mistake teams make when scaling.
"People typically scale, I think incorrectly, by just throwing hardware at the problem."
The cloud was supposed to fix this. Teams no longer rack and stack servers, and they can provision more on demand. But Advey has seen high-scale environments where even the cloud doesn't have the capacity a team wants or needs, and the bill grows with every addition.
Data volume makes it worse.
"The more data you have out there, the bigger it is, the worse structure it is, the more cost you get. And just throwing infrastructure at it usually is not the right answer. It does not solve the problem you're trying to get to."
The same logic applies to AI adoption. Telling everyone to go spend a fixed monthly amount on AI is not a scaling plan.
"'Hey, everyone go out there and use $5,000 a month worth of AI.' Like, no, that's not great, because you're just effectively lighting money on fire. It's just a big experiment."
At the largest scale, he frames the question every leadership team should be asking. If you spend a billion or two billion dollars on AI, was that an investment, or just a big experiment?
📣 Wondering what your heaviest pipeline would cost on a more efficient compute layer? Run it side by side in your own environment with a free Haevek Test Flight.
The economics behind what gets shipped
This is where cost stops being only a finance problem and becomes a roadmap problem. Budgets, not ambition, define what can and cannot be accomplished.
Advey describes the roadmap question in economic terms: how many tokens are you willing to throw at a project, and how much are you willing to have your people spend to generate outcomes? The projects that matter most are often the ones that are too expensive or too slow to run at scale, so they wait.
He also points to how much compute gets wasted by asking AI to do work a program can do better. Asking a model to analyze data directly can return two different answers from the same data. Having AI write a program to run the analysis returns the same result every time.
"You have the AI write the program, then you get the same result every single time you run it. So you're spending less, you're not wasting as much data infrastructure to answer questions and get results out."
The goal is to architect so each unit of compute produces more, rather than pulling back on AI.
A better way to think about data and AI cost: efficiency is the lever
When asked how teams can lower compute cost while sustaining velocity and performance, Advey comes back to one idea.
"It's all about efficiency and measuring that efficiency and being able to do more with less."
In practice, Advey proposes the following methods:
- Make cost visible to the people building. Teams pushing product need cost detail early enough to change decisions, not weeks after the invoice.
- Track unit economics from day one. Cost per transaction, cost per user, or cost per report tells you whether a workload will scale with your pricing.
- Tie spend to a business case. Advey’s work at CloudZero starts with what a team is spending on technology, cloud, and AI, and what the business case is for that spend.
- Architect before you scale. Choosing the right approach for the use case, including deterministic programs over repeated inference, beats adding hardware later.
He adds a reminder about inputs. AI succeeds with the right amount of accurate data. Feed it the wrong data, like the wrong prompt, and it won't do what you expect. It has to be the right data at the right time.
Where the compute layer fits
Advey’s methodology aligns with how Haevek thinks about the compute layer: the part of the data and AI stack that decides what everything else costs and how fast it moves.
If efficiency is the key to cost effective outcomes, the engine running your workloads matters as much as how you monitor them. Most teams inherited their compute layer from an earlier era and spend their time tuning around it, adding compute whenever jobs slow down. That is the throw-hardware-at-it pattern Advey describes.
Haevek offers the ability to do the same job using a fraction of the compute at the fraction of the cost. The Falcon Compute Platform is a Kubernetes-native data compute platform written from the ground up in the type safe Rust language. It runs the same batch, streaming, and AI/ML workloads for up to ~90% lower compute TCO.
Falcon also takes aim at the sticker shock itself. It is priced on parallel-compute capacity, a fixed annual ceiling instead of a usage meter. The CTO knows their development roadmap won’t accidentally break the bank, and the CFO can plan against the cost instead of discovering it on the invoice. Falcon augments the existing lakehouse. The warehouse, governance, and downstream tools stay in place, and only the compute-heavy production workloads move.
The takeaway
Data and AI cost problems rarely start with the invoice. They start when cost is not top of mind at the outset, when visibility arrives late, and when the default answer to scale is more hardware.
Larry Advey's advice is to flip that order. Measure efficiency, make unit economics a KPI from day one, and architect for cost before you scale. The compute layer is where much of that efficiency is won or lost.
How much cost reduction can you see with Haevek?
While the typical workload sees a 75% decrease in cost, each project is different. To find out what your savings can be, we offer the 2-4 week Falcon Test Flight for you to pilot Falcon in your environment with your data. No data moved, no lock-in, training included. The benchmarks will show the performance increase and cost savings Falcon can provide.