As AI tools are becoming a bigger part of data work for many organizations, the priority should not simply be to make every pipeline faster; they should be tuned to match the business need. Søren Meincke Persson of Norlys puts a focus on connecting pipelines to trusted, well-contextualized data to ensure accelerated, AI-powered pipelines produce trustworthy results.
As Director of Data, Platform, and Governance at the telecommunications and energy company Norlys, Persson is responsible for ensuring both the integrity and accessibility of the company’s massive data lake. AI can make all that data more accessible, but it can also make inconsistencies harder to hide. A big part of Persson’s job is to ensure multiple pipelines, dashboards, and metrics did not provide different answers to the same question. Teams working within the organization need to be able to trust their system’s output.
We can’t have conflicting truths.
AI-fueled pipelines digest and analyze far more data than was previously possible, but without a sound foundation, these systems can end up reaching the wrong answer — albeit faster than before.
A pipeline built for one use case and one dashboard can work just fine with a smaller, more narrowly defined data pool. But when AI can scour exponentially more data from disparate sources to answer a question, conflicting definitions can yield inaccurate or incoherent results. With the stochastic nature of AI, it will even return different results in repeated queries. The answer depends on which source the system chooses, not necessarily the one the business considers authoritative.
If the same question can be answered from five different data sources, then you would get 20% right, I would guess.
The solution is to connect pipelines to the correct data model, define metrics clearly, and attach metadata that explains which data source applies to which use case. Lineage and testing help teams see how data moves and whether separate measures are intentionally different. It sometimes seems like an extraneous chore, but if you skip it the organization will have to deal with competing truths.
After a big merger, the Norlys data team had the opportunity to rebuild its data foundation. The immediate priority was making data available for business decisions. Persson had to struggle with the constraints of legacy, but kept his focus on providing valuable resources for business decisions. In the urgency of the moment, determining cost allocation for some of the work was put off, but Persson made sure the pipelines could be tagged so the company could eventually understand and distribute costs by workload.
While a team does not need to solve every future operational question on day one, it can make early design choices that preserve the ability to answer key business questions down the road. Workload context supports both trusted data products and a clearer view of who or what is driving spend.
📖 More on costs from Larry Advey: “Stop optimizing after cost overruns.”
Like anyone in his position, Persson is constantly considering the relationship between cost and performance. For pipelines, the budget impact of speed can be massive.
Everybody wants real-time, but do they need real-time?
Before increasing refresh frequency, teams should ask whether people will use the data more often and whether getting it sooner changes a decision.
A pipeline supporting customer operations or sales targets may need faster performance because people are acting on the data. A quarterly report, not so much. Persson recommends clear service-level expectations with the teams that depend on operational data, then tuning those pipelines to meet the actual need.
Every pipeline does not warrant the same amount of engineering attention. Persson says Norlys has spent months fine-tuning workloads that account for about 10% of its overall cost, while a small pipeline moving 10,000 rows may not justify weeks of work. The right level of effort depends on what the team is optimizing for and what the workload means to the business.
So don’t treat everything with the same lens.
If cost is the priority, focus on the largest cost drivers. If a pipeline supports a critical operation, performance and reliability deserve attention too. Making that distinction helps teams spend engineering time where it can reduce costs, improve service, or generate a desired business outcome—not simply where another round of tuning is possible.
As AI systems use more data, context should be available to the people consuming system output. Persson has his teams closely link definitions and context — the semantic layer — to the data. These functions had been built into end user dashboards before, creating more risk and making it harder to control. They integrate data governance into the platform where it can be consistently used. That context helps people recognize metrics that look similar but answer different questions. It also gives machines a better chance of returning an answer that matches the intended meaning. Data models, definitions, metadata, lineage, and tests are not administrative extras; they are part of making data products dependable as use cases grow.
A solid technical foundation is only part of the operating model. People also need to learn how to use AI tools. Persson’s team chose not to begin with restrictions. Instead of imposing budget constraints, they encouraged adoption, expecting some early mistakes.
First we need adoption. Then we need to make people used to using it .... they will make mistakes.
That does not mean ignoring cost or leaving AI unmanaged. Persson coaches users when their spending on AI utilization increases. He recognizes that more autonomous agents will eventually require stronger guardrails against cost overruns. He expects the controls to mature as the technology and use cases mature. In the meantime, teams can help people choose a model appropriate to the task instead of assuming the newest or largest option is always necessary.
Persson’s examples offer a practical order of operations: make data trustworthy, understand the work it supports, then build. Considering the cost and business need, choose the performance level, optimization effort, and AI controls that fit. Faster is valuable — when it improves an outcome. Create governance so that people and systems use the right data to generate output they can trust.
The data compute stack is not just infrastructure to Persson; it is an expression of business priorities. Business priorities shape the cost and speed of data and AI workloads. They guide the data foundation and operating choices around them.
Bringing the infrastructure cost of pipelines down opens up more performance to more applications.
Haevek’s Falcon Compute Platform is designed to run existing data and AI workloads with lower compute requirements while augmenting—rather than replacing—the lakehouse, governance, and downstream tools teams already use.
For teams evaluating their own stack, start with a real workload and the business outcome it needs to support. Haevek’s invitation is to bring your most challenging pipeline and explore it through a Test Flight.