Why AI gets decisions wrong

In this episode of the Trust the Process Podcast, we’re joined by Devavrat Shah, MIT professor and Chief Scientist of Enterprise AI at Celonis, to step into a small shoe store on New York’s Upper West Side.

An AI agent recommends ordering 30 pairs of the best-selling shoe, but it doesn’t know about the weekend promotion, the unreliable supplier, the nearly full warehouse or finance’s new spending limit.

We look at how AI uses probability to calculate possible futures, why Process Intelligence gives it the business context it’s missing, and how probabilistic AI and deterministic rules work together to reach the right call.

The following is a transcript of the podcast, edited and organized for readability. You can listen to the podcast on Apple, Spotify and wherever you get your podcasts.

https://delivery-p141552-e1488202.adobeaemcloud.com/adobe/assets/urn:aaid:aem:04ecbb4a-5ec0-41a0-ba98-f2c343362f08/original/as/trust_the_process_s2_ep1_video_thumbnail.png
https://celonis-videos.wistia.com/medias/7mcbu271rc

Ten minutes to place an order

It's Monday morning at our small shoe store on New York's Upper West Side. Our top brand supplier has just sent a reminder: place the order within the next ten minutes, or wait another week for the next delivery.

Our bestselling shoe is almost sold out. Order too few and we lose customers. Order too many and thousands of dollars stay trapped on our shelves. So we ask our new AI agent.

It analyzes years of sales data, calculates thousands of possible futures and delivers its answer: order 30 pairs. The probability says it's the right decision.

But the AI doesn't know that we have a weekend promotion planned. It doesn't know that our supplier recently missed two deliveries. It doesn't know that the warehouse is almost full, or that Finance has just introduced a new spending limit.

The AI has calculated the future without understanding the business. So with a few minutes left on the clock, how many shoes do we order? And if an AI can be mathematically brilliant but contextually blind, what does it really need before we allow it to make decisions for us?

To find out, we enter the mathematical brain of AI with Devavrat Shah, professor at MIT and co-founder of the Decision Intelligence company Ikigai Labs. Devavrat has spent more than two decades pioneering the probability, algorithms and graphical models that turn complex data into decisions.

In this episode, we look at how AI uses probability to calculate possible futures, the role business context plays, and how to combine probabilities with deterministic rules to decide what happens next.

Everything is probability

Order 30 pairs. But where did that number come from? To understand the recommendation, we first need to understand the language AI uses to express uncertainty.

Devavrat Shah: "Every field has language. When we think about language like English, language like German, there is a notion of letters, words, grammars, leading to sentences, to paragraphs and all that. Similarly, language for decision starts with extracting information from data, and probability is that language."

Probability lets us describe a world where we don't know exactly what will happen. Will ten customers visit our store today, or 50? Will warm weather bring more people into the city, or keep them away? We can't know any of this for certain. But with enough relevant data, we can estimate how likely different outcomes are.

That estimate is called a probability distribution.

Devavrat Shah: "What does probability distribution mean? Probability distribution at the simplest level means summary of data. We always summarize things like, what is the average of things, or what is the variation in the data? What is the maximum? What is the minimum? That style of information is coming out of what I would call probabilistic representation of data."

A probability distribution doesn't give us one certain future. It describes a range of possible outcomes and assigns a likelihood to each. For our shoe store, it might say that selling around 30 pairs is most likely, while selling fewer than ten or more than 50 is possible but less likely.

From connected variables to what-if decisions

Demand isn't the only uncertain variable in our shoe store. Sales may depend on the weather, our prices, a planned promotion, how many customers walk in and which sizes we have on the shelf. Now we're dealing with several variables whose possible values are connected. That's where graphical models come in.

Devavrat Shah: "Graphical models are a representation of data where they're representing data in a probability distribution manner. They're called graphical because they try to learn relationship between, say, every pair of data points, every triplet of data points and things like that."

Imagine writing each variable on a separate card: weather, promotion, customer traffic, demand, inventory, sales. The connections between the cards represent probabilistic dependencies. Knowing that a promotion is scheduled makes high demand more likely. Knowing that popular sizes are out of stock makes high sales less likely, even if the store is busy.

In scientific terms, a graphical model represents a joint probability distribution: the probabilities of several variables considered together. The graph shows which variables directly depend on one another, which makes a complicated distribution easier to learn, interpret and compute with. It doesn't prove that one variable causes another. It captures statistical relationships that carry useful information.

One quick side note: a graphical model is not the same as a graph database.

Devavrat Shah: "There's a very popular notion of graph database that people confuse with graphical models. When we say we are in a social network setting and you and I are two individuals who are connected with each other, that connection is represented explicitly in a database. That's what people call ontology. That is not what graphical model is. Graphical model is more like saying that things like sales and things like promotions have some relationship with each other, and let me capture that relationship implicitly."

A graph database might record that a particular customer bought a particular shoe. A graphical model asks what observing that purchase tells us about the probability of other events.

Say our model first estimates that we'll sell 24 pairs next week. Then we learn about the weekend promotion. Because promotion and demand are connected in the model, that new evidence shifts the probabilities for different levels of demand, and with them the probability that we'll run out of stock. Using observed information to update probabilities for unknown variables is called probabilistic inference. You already use it every time you open a navigation app.

Devavrat Shah: "The global positioning system that you and I use every time we use Google Maps to go from one place to another is a perfect example of a graphical model-driven decision support system. To evaluate what would be the delay and latency at each step of that path, it needs a probabilistic evaluation, some form of sequential simulation of path and latency."

But estimating travel times along each route doesn't yet tell the system which route to recommend.

Devavrat Shah: "As you're doing this simulation, you might have two or three different paths. Which one to choose, the best one? This is where decision optimization comes on top of these simulations."

So we ask our model to compare several possible actions.

Devavrat Shah: "Going back to that simple question: how many shoes to order? What if I ordered zero shoes today? What if I ordered 10 shoes today? What if I ordered 20, or 30? And if I ordered 30 shoes today, maybe seven days from now I will order 20 shoes, and 15 days from now I will order 45. So this is a big set of what-ifs that you're running on multiple trajectories."

Ordering more reduces the chance of running out but raises the risk of unsold inventory. Ordering less protects our cash but increases the chance of lost sales. And today's decision shapes tomorrow's options.

A probability distribution represents possible outcomes. A graphical model shows how uncertain variables connect. A forecast estimates what's likely to happen, and a simulation explores what could happen under different actions. Decision Intelligence then compares those alternatives against an objective. A decision is about trade-offs, not just predictions.

Our AI has done all of this. Its recommendation still reads: order 30 pairs. But did it include the business context in which those futures have to unfold?

Giving AI a business brain

Mathematically, 30 pairs may be an excellent answer. But our AI also needs to understand the business.

Master data identifies the relatively stable parts of our business: products, suppliers, stores and warehouses. Ontology knowledge defines how they relate, for example which supplier provides which shoe and where it's stored. Together they give AI a business vocabulary. What they don't necessarily tell it is what's happening right now. Has the promotion changed? Are deliveries still outstanding? Is the warehouse already full?

These are questions about events, states, sequences and variants. What happened, in which order, what's happening now, and how does that change what can happen next? This is process context.

Devavrat Shah: "When we as individuals in organizations do our role, we don't do our role in isolation. We constantly collaborate and work with people, and we are following certain processes, implicitly or explicitly. So these AI agents, when they are playing a role, cannot play their role in isolation. When AI models are trying to run a what-if simulation, they can't do that in isolation. They have to be respecting the process."

Our decision to order 30 pairs is connected to the weekend promotion, Finance's spending limit, the space left in the warehouse and the supplier's recent delivery record. If one of these changes, the value of our decision changes with it. A demand forecast can be statistically accurate on its own. So can an inventory calculation. A useful business decision has to connect them, and that's the job of a digital twin of the organization.

Devavrat Shah: "Processes have to be an integral part of the model at its core. It's not an afterthought or an add-on. It is an integral part of the modeling, an integral part of agent design and an integral part of decisioning."

To show why this matters, Devavrat takes us from the shoe store to a Formula One racetrack. A driver may have an extremely fast car and years of training, but the right decision still depends on the track.

Devavrat Shah: "If you are training and practicing to become a really good driver on the Las Vegas track versus maybe a track in Europe, those are two different tracks. On those tracks, you might want a very different strategy in terms of how you speed up and slow down at different stages. That itself is a context. To build those simulators, you actually have to include that context fully: the weather, the temperature, the rain, what the track looks like."

The car is the same. The objective is the same. The environment has changed. Back in our shoe store, the track is our business process.

Devavrat Shah: "In a similar sense, in business processes, what you do today in ordering more shoes must be dependent on what your demand is going to be, and must be dependent on what your marketing decisions are going to be."

Process Intelligence can reveal exactly that kind of context:

• which activities have already happened
• their sequence and timing
• the current state of relevant orders and deliveries
• how products, suppliers, warehouses and orders interact
• which process variant is unfolding
• where delays, changes or exceptions have appeared

Master data tells us which supplier we're dealing with and how much we usually order. The ontology tells us what role that supplier plays. Process Intelligence tells us which delivery variant we're in, whether deliveries are already on their way and whether the warehouse has room.

Devavrat Shah: "Either you have seen these kinds of scenarios in the past, and that means you are already trained and from historical learning you can make an automated decision. Or you say, I have not seen these things before, or I'm uncertain. Let me run a few what-if scenarios in real time first."

If the process follows a familiar path, the agent can recommend or automate the next step. If it follows an unusual variant, the agent can simulate the alternatives again or escalate the decision to a person. That's human-in-the-loop decision-making in practice.

So our agent recalculates. The promotion pushes expected demand up. The nearly full warehouse limits how much we can safely order. The unreliable supplier raises the risk of a late delivery. The spending limit caps what we're allowed to spend. The original 30 pairs no longer holds.

Decision Intelligence gives AI a mathematical brain. Process Intelligence gives it a business brain.

Where rules meet probability

One issue remains. With the weekend promotion factored in, the probability model now suggests that ordering 50 pairs gives us the best chance of meeting demand. But the warehouse only has space for 41. And Finance's spending limit doesn't allow more than 37.

Finance hasn't just suggested we spend less. It has imposed a limit. That's not another uncertain event. It's a rule our decision has to obey.

This is where deterministic rules meet probabilistic AI. Probabilistic AI estimates what's most likely to happen, based on patterns, data and uncertainty. Deterministic rules set boundaries that must not be crossed, however likely or attractive an outcome looks. Prediction tells you about the future. Rules tell you which futures are acceptable.

Devavrat Shah: "Processes impose constraints, guardrails, and we must satisfy them. And when those constraints are incorporated, you can typically do a much sharper what-if than not, because then you know that you're not going to do arbitrary things."

In AI research, these approaches come from different traditions. Early symbolic AI relied on explicit knowledge and logical rules. Probabilistic machine learning learns patterns and uncertainty from data. Enterprise AI increasingly needs both.

Devavrat Shah: "Classic AI was always deterministic. It was all decision rules. Then AI became generative, which is probabilistic. And really, the good place to be is in between. In my mind, when processes come together with the probabilistic view is where we should be going."

In mathematical optimization, the split is clear. The objective function describes what we want to achieve, such as maximizing expected profit while avoiding lost sales and excess stock. The constraints define which solutions are allowed at all.

Devavrat Shah: "Constraints are there to satisfy, and we should learn from them. And when there is nothing to be learned from, you learn from data."

Using process definitions to actively steer probabilistic and generative models is still an emerging frontier. It means bringing together the language of process and the language of probability.

Many agents, one process

Our shoe store now has an AI agent that understands business relationships and dynamics, and respects the rules. But much of that knowledge is domain-specific. So an organization is unlikely to rely on a single context-aware agent.

Devavrat Shah: "In my view, there won't be one all-aware agent. The best way to think about where agents should be is, again in the shoe store setting, what are the different functions or roles that are there? Those roles are done by different individuals in our organization, and those roles lead to one agent each."

The result is a multi-agent system: several agents observing information, making decisions and interacting in a shared environment. That creates a new problem. Each agent may pursue a goal that makes sense locally. Autonomy needs more than smart individual agents. It needs coordination.

Devavrat Shah: "These different agents and different roles might need to be aware of each other, might be handing off to each other. That's where process-style integration comes into play. We need roles to be supported with agents where agents are talking to each other, not alone. And for agents to talk to each other, they need to follow the organizational processes that we collectively follow right now as humans."

The process defines how individual decisions become one coordinated action. Process Intelligence feeds the digital twin's context to each agent, and coordinates their roles within it.

Back in our shoe store, the clock is still ticking. The demand model has shown what could happen. Warehouse capacity and the spending limit have defined what's allowed. And the process has coordinated Marketing, Inventory, Finance, Warehouse and Procurement to decide what happens next.

Order 37 pairs

The final recommendation appears: order 37 pairs. The procurement agent submits the purchase order before the supplier window closes.

We didn't blindly trust the original 30 pairs. Nor did we throw out the mathematics. We gave the AI the missing business context, applied the relevant rules and coordinated the agents through the process. That's the foundation of meaningful enterprise autonomy: not one all-knowing machine deciding in isolation, but specialized agents using probability, rules and process context to work toward a shared goal.

Devavrat Shah: "I think there is no future of AI-supported organizations without process integration."

Probabilities describe what could happen. Rules define what's allowed. Processes coordinate what should happen next.

Listen to Trust the Process Podcast

Join us next time on Trust the Process as we explore what processes mean for US juvenile justice and youth rehabilitation, together with the Evident Change project.

Because when processes work, decisions become clear.