Agentic Workflow: When a Model Should Decide a Step, and When It Should Not
An agentic workflow is one where at least one step is decided by a model at run time rather than by a rule written in advance. That is the whole definition, and the useful question is not what to call it but which steps deserve that treatment. The answer is: far fewer than the current enthusiasm suggests.
One test does most of the work. Can you enumerate the cases? If you can write down the inputs and what should happen for each, that step is a rule, and a rule is cheaper, faster, testable and explainable. Hand it to a model and you have bought ambiguity you then have to govern. If you genuinely cannot enumerate the cases — the input is open text, the categories keep moving, the judgement is contextual — a model earns its place.
There is a second test that overrides the first. The same input will not return the same answer twice. If a step has to be reproducible — because money moves, because a regulator will ask, because a downstream system depends on it being stable — it is not a model step, however hard it is to enumerate. Non-determinism is not a tuning problem you can configure away. It is what you bought.
The guardrails below come from building the AI layer inside Kroolo, an AI work operating system where generation and an agent catalogue sit inside the work itself rather than beside it. They are not best practices collected from reading. They are the four things that turned out to be build decisions rather than settings.
It is not multiple agents talking to each other. That is orchestration, it is a separate problem, and we have written about it separately. A single step inside one workflow, decided by a model, is already an agentic workflow and is where most teams should stop.
It is also not automation with a chat box on the front. Replacing a form with a prompt changes the interface, not the decision. If the rule behind it is unchanged, you have added latency and cost to something that worked.
And it is not a reason to rewrite the workflow. The useful move is almost always to keep the existing deterministic pipeline and replace one step, chosen because it was the one nobody could specify properly.
Apply the test to something concrete. Take inbound support: a message arrives, it gets categorised, it gets routed to a queue, and someone drafts a reply.
Categorisation looks like the obvious model step and usually is not. If your categories are a fixed list of twelve and most messages land cleanly, that is a rule with a lookup, and a rule will be right more consistently than a model while costing nothing and explaining itself. The honest reason teams reach for a model here is that the list has grown to forty overlapping categories nobody maintains — and the fix for that is to fix the list, not to hire a model to guess at it.
Routing is almost never a model step. It is a table: this category, this tier, this language, this queue. It has to be reproducible because people are measured on it, and it has to be explainable because someone will ask why their ticket sat in the wrong place for a day. Both of those rule a model out before cost or accuracy enter the conversation.
Drafting the reply is the model step, and it is the only one of the three that passes both tests. The input is open text that cannot be enumerated. The output is judged rather than checked. A poor draft costs a few seconds of an agent's time, and the agent reading it is the verification step, which means you get quality signal for free.
That ratio — one step in three, and the one everybody would have guessed last — is typical. The instinct is to point the model at the part that feels most intelligent. The discipline is to point it at the part that is genuinely unenumerable and cheaply verified.
Each of these is a decision taken while building, not a flag turned on afterwards. That distinction matters because every one of them is expensive to retrofit, for the same reason the data model in any system is expensive to retrofit: too many things already depend on the shape.
This is the one that keeps people awake, and it is specific to the moment you add retrieval to a multi-tenant product.
In a relational store, the tenant filter is one predicate the ORM will not let you forget. Move that data into a vector index and the boundary becomes a metadata filter — and a filter applied after the nearest-neighbour search is weaker than one applied inside it. The search has already crossed the boundary; you are discarding results rather than never retrieving them.
The second half is worse because it is easy to miss entirely. Embeddings are derived copies. When a row is deleted or a permission changes, that change has to reach the index, not just the row. A tenant can delete a document, see it gone from the interface, and still have it inform an answer generated for somebody else, because the vector it produced is still sitting in the index.
If you take one thing from this post into a vendor conversation, make it this: ask where the tenant filter is applied relative to the search, and what happens to the index on delete.
Per-seat pricing assumes cost per account stays roughly flat. Put a model call inside a feature people use all day and cost tracks usage instead, with a thin slice of accounts generating most of it. Your pricing model and your cost model stop being the same shape, and the gap grows with adoption, which punishes exactly the adoption you were aiming for.
Three things follow, and none of them is optional. You need metering at the unit the price is set on, so you can see margin per the thing you actually charge for. You need a budget enforceable mid request, not a daily report, because by the time a report arrives the spend has happened. And when the budget is hit, you need a degraded answer rather than a failure — a smaller model, a cached response, a plain non-generated path — because an error message is a worse outcome than a shorter answer.
There is a second-order effect worth planning for. Because cost follows behaviour, your heaviest users are your least profitable, which inverts the usual intuition that engagement is unambiguously good. On a per-seat plan the account that uses the AI feature forty times a day is the one to worry about, not the one to celebrate, and you will not know which accounts those are unless the metering exists from the start. Retrofitting usage metering onto a feature that has been live for a year means a year of invoices you cannot explain.
Latency belongs in the same conversation. A call inside a click has the click's budget, not the model's. A step that takes four seconds is not a slow feature, it is a broken interaction, and the fix is usually to move the call out of the interactive path rather than to find a faster model.
The same input will not return the same answer twice, so assertions stop being the safety net. This is the single biggest change to how the work is tested, and teams routinely underestimate it because the code looks testable.
What replaces assertions is a fixed evaluation set, scored before release, plus production signals: how often a draft gets edited, how often it gets regenerated, how often it gets abandoned. Those three are the honest measures of whether a generated step is helping, and they cost nothing to collect compared with what they tell you.
None of it is attributable unless the prompt and the model version are stored with the output. A quality change you cannot trace to a prompt edit or a model upgrade is just weather. This is the cheapest guardrail on the list and the one most often skipped, because it feels like logging rather than product.
The practical consequence is that you cannot safely upgrade a model without the eval set, and model upgrades are not optional: providers deprecate versions on their own schedule. A team without a scored evaluation set experiences a forced upgrade as a mystery, where output quality shifts in some direction nobody can measure and the only evidence is complaints. A team with one runs the set, sees the delta per case, and decides. That is the difference between operating the thing and being operated by it.
The word "agentic" implies the system does things rather than suggests them, and the moment it does things it needs an answer to a question nobody asks early enough: as whom?
An agent that creates tasks and edits documents needs its own identity, scoped to what the person invoking it could already do. Never a service account. A service account is convenient on day one and is a privilege escalation waiting to be discovered: an agent invoked by a junior user, running with permissions nobody granted that user, writing into records they could not otherwise touch.
Every action wants an audit row naming the agent and the human behind it. Both, not either. "The agent did it" is not an answer when someone asks why a document changed, and neither is a user id with no indication that a model was involved.
And whatever an agent does, a human has to be able to reverse. This is the constraint that should shape the feature from the start, because it rules out whole categories of action. If a step cannot be undone — a payment sent, a message delivered to a customer, a record purged — it either stays a human decision or it gets a confirmation step, and the confirmation step is not a failure of ambition. It is the design.
Honest gap, stated plainly: this post carries no inference cost figure and no failure rate, because we do not have measured ones we can publish.
What can be said without numbers is the shape. Inference cost is a function of three things you control — how often the step fires, how much context you send it, and which model answers — and of one you do not, which is provider pricing. The first of those moves the number most. Teams reach for a cheaper model when the real saving is usually in not calling the model at all for the forty percent of inputs a rule could have handled, or in trimming context that was attached out of caution rather than need.
The number you actually need is cost per unit of the thing you charge for, and it is specific to your workflow, your context sizes and your model choice. Anyone quoting a general figure for "what an agentic workflow costs" is quoting their own workload, not yours. Build the metering first and you will have your number within a week of going live; take someone else's and you will plan against a workload that is not yours.
The same applies to failure rates. A published rate from another product tells you about their prompt, their data and their evaluation set, which is to say nothing transferable at all.
Not in the model. Models are the commodity part now, and swapping one is a day of work if you stored the version.
They fail when a step that needed to be reproducible was handed to a model, and the business discovers it the first time two runs of the same input disagree in front of a customer. They fail when retrieval crosses a tenant boundary, which is a single incident with regulatory consequences rather than a bug with a fix. They fail on unit economics, when a feature everyone loves turns out to cost more per account than the account pays. And they fail quietly when nobody stored the prompt and version, so six months of drift cannot be attributed to anything and the team argues from memory.
Notice that three of those four are not AI problems. They are multi-tenancy, pricing and observability problems that a model made sharper. That is the honest summary of this whole category: the model is rarely the hard part, and the parts that are hard were already hard.
Start with a step that is genuinely unenumerable, cheap to get wrong, and verified by a human before it takes effect. Drafting falls into that category almost perfectly: a first draft of a summary, a reply, a project plan. The input is open, the output is judged rather than checked, a wrong answer costs a few seconds, and the human who reads it is the test.
Avoid, at least first, anything that moves money, anything a regulator reads, anything another system treats as a fact, and anything that cannot be undone. Those can come later, with a confirmation step, once you have the metering and the audit trail that the first step forced you to build.
There is one more selection criterion that gets missed: pick a step where you already know what good looks like. If nobody in the business can say why one summary is better than another, no evaluation set can be written, no production signal can be interpreted, and you will be shipping changes on taste. Steps with an existing quality bar — because a human already does the work and already has opinions about it — are the ones where the feedback loop closes.
The order matters more than the ambition. The first agentic step in a product is mostly an excuse to build the four guardrails above, and a team that ships drafting with metering, evals, scoped identity and a working undo has done the expensive part. The second step is then a configuration change rather than a project, which is the opposite of how most of these programmes are sequenced.
A workflow where at least one step is decided by a model at run time rather than by a rule written in advance. It does not require multiple agents, a chat interface or a rewrite of the existing pipeline. The common and sensible shape is an otherwise deterministic workflow with one step replaced, chosen because it was the step nobody could specify properly. Multiple agents coordinating with each other is orchestration, which is a separate problem with its own failure modes.
When you cannot enumerate the cases: the input is open text, the categories keep changing, or the judgement is contextual. If you can write down the inputs and the right response to each, keep the rule, because a rule is cheaper, faster, testable and explainable. There is an overriding second test: if the step must be reproducible, because money moves or a regulator will ask or a downstream system treats the output as a fact, it is not a model step regardless of how hard it is to enumerate.
Because the same input will not return the same answer twice, so assertions stop working as a safety net. What replaces them is a fixed evaluation set scored before release, plus production signals: how often a draft is edited, regenerated or abandoned. None of that is attributable unless the prompt and the model version are stored alongside the output, which is the cheapest guardrail to build and the one most often skipped because it feels like logging rather than product.
The tenant boundary stops being a database predicate and becomes a metadata filter on a vector index, and a filter applied after the nearest-neighbour search is weaker than one applied inside it. Separately, embeddings are derived copies, so deletions and permission changes have to reach the index and not just the source row. Otherwise a tenant can delete a document, watch it disappear from the interface, and still have it inform an answer generated for someone else.
Per-seat pricing assumes cost per account stays flat, and a model call inside a daily-use feature breaks that: cost tracks usage, with a small share of accounts generating most of it. You need metering at the unit you price on, a budget that can be enforced part-way through a request rather than reported the next day, and a degraded answer rather than an error when the budget is reached. The gap between your pricing shape and your cost shape widens with adoption.
Its own identity, scoped to exactly what the person invoking it could already do, and never a service account. A service account is convenient initially and becomes a privilege escalation: an agent invoked by a junior user writing to records that user could not otherwise touch. Every action should produce an audit row naming both the agent and the human behind it, and anything the agent does a human must be able to reverse. If an action cannot be undone, it keeps a confirmation step or stays with a person.
One that is genuinely unenumerable, cheap to get wrong, and checked by a human before it takes effect. Drafting fits almost exactly: open input, judged output, a wrong answer costs seconds, and the reader is the test. Leave anything that moves money, is read by a regulator, is treated as fact by another system, or cannot be undone until later. The first step's real value is that it forces you to build the metering, evals, scoped identity and undo that make the second step a configuration change rather than a project.
Vikas has around fifteen years of experience building software and now builds generative AI systems at Zyneto. His work covers retrieval augmented generation, agentic AI, knowledge graphs, AI memory, and the evaluation and guardrails that decide whether any of it is safe to put in front of customers. He has shipped enterprise copilots, document AI, chatbots and predictive analytics for e-commerce, fintech and marketing teams, and works day to day in Python, JavaScript and SQL. He follows multimodal models, business process automation and enterprise AI security closely, and mentors engineers moving into AI. He writes about architecture, inference cost and the failure modes that only show up at production scale.
Share your details and we will talk soon.
Be the first to access expert strategies, actionable tips, and the trends actually shaping the digital world. No fluff - just practical insights delivered straight to your inbox.
Dive into our blog and stay ahead of the curve with expert perspectives, future-ready trends, and tech tips written for decision-makers and doers alike.
