Adding AI to Existing Software: What It Costs and What It Breaks

11 min read
Adding AI to Existing Software: What It Costs and What It Breaks

Adding AI to software that already has customers costs more than building the same feature from scratch, and the extra is almost never model work.

The model is a few days. The promises are the project. Your contracts, your data processing agreement, your subprocessor list, your residency commitments and your existing permission model were all written before you had a model vendor, and every one of them constrains where the inference can happen, what can be stored, who has to be told and which parts of the vendor's platform you are allowed to use at all.

That last constraint is the one nobody budgets for, and it is published, in writing, by the vendors themselves. OpenAI's zero data retention is unavailable on Conversations, ChatKit threads, Assistants, Threads, Vector Stores, Files, audio Voices, Fine-tuning, Evals, Batches and Agents: on most of the stateful machinery that makes an AI feature quick to build. Chat Completions, Responses, Embeddings and the audio transcription, translation and speech endpoints are eligible. The conveniences are not.

So if you have told an enterprise customer that their content is never retained by a third party, the tutorial architecture is closed to you and you are building the state yourself: your own conversation store, your own retrieval index, your own evaluation records, in your own database, under your own retention policy.

Three questions set the cost, and none of them is about the model. What did you already promise about this data? Which of your existing permissions have to hold inside the feature? And what does it cost to run every month once it works? Everything else is ordinary product work on a codebase you already know.

How much does it cost to add AI to software that already has customers?

For a product already in production, budget the AI feature at roughly the cost of a substantial new module, not an add-on. Model calls are the smallest line. What dominates is rebuilding the vendor's stateful features inside your own boundary, threading your permission model through retrieval, and the contract work your agreements require before launch.

At our published rates of $40 to $100 an hour by role, blending to $60 to $70 for a mixed team, that lands most first AI features in the same bracket as any other serious module, and well above the figures a generic AI development cost estimate will give you for greenfield work. The variance between two superficially identical features is almost entirely the promise layer. A product with no enterprise customers, no data processing agreement and no residency commitment can use the vendor's stateful endpoints and ship in a fraction of the time. The same feature in a product with three signed enterprise agreements is a different build.

Which is why the first conversation is not about models. It is about reading your own paperwork.

Your data processing agreement already chose your architecture

Open the agreement you signed with your largest customer and find the subprocessor clause. That clause, not an architecture diagram, determines whether you can call a model API at all.

Under GDPR Article 28(2) the rule is explicit: "The processor shall not engage another processor without prior specific or general written authorisation of the controller." Where the authorisation is general, the processor "shall inform the controller of any intended changes concerning the addition or replacement of other processors, thereby giving the controller the opportunity to object to such changes."

A model vendor is another processor. Adding one is the addition the clause is about.

And the liability does not transfer with the work. Article 28(4) keeps the first processor "fully liable to the controller for the performance of that other processor's obligations". If your customer's data goes to a model vendor through your product, their retention behaviour is your contractual exposure, not the vendor's.

Two practical consequences for the plan. First, the notice period in your own agreement is now on the critical path, because the feature cannot go live for that customer until it has run. Second, the objection right has to be designed for: if a customer objects, you need the feature to be switchable off per tenant, which is a build requirement discovered in a contract rather than a backlog.

Zero retention is missing from exactly the endpoints you wanted

Here is where published vendor documentation beats any amount of architectural intuition.

OpenAI states that data sent to its API is not used to train or improve its models unless you explicitly opt in. Abuse monitoring logs, though, are generated for all API feature usage by default and retained for up to 30 days. Zero data retention removes that, and under it the store parameter on the Responses and Chat Completions endpoints is always treated as false.

But the eligibility list is the interesting part, and OpenAI's own data documentation publishes it endpoint by endpoint. Eligible: Chat Completions, Responses, Decisions, Embeddings, audio transcription, translation and speech, Images, Moderations, Completions, Realtime and Live sessions. Not eligible: Conversations, ChatKit threads, Assistants, Threads, Vector Stores, Files, audio Voices, Fine-tuning, Evals, Batches and Agents. One exception survives even zero retention: an image flagged by the CSAM classifier is kept for manual review.

Read the second list again as a build estimate. Threads and Conversations are your chat history. Vector Stores and Files are your retrieval layer. Evals are your quality measurement. Agents are the orchestration. A strict no-retention promise means you own all four of those, in your own infrastructure, with your own backup and deletion behaviour. That is the single largest difference between a two-week AI feature and a two-month one, and it is decided by a sentence in a contract signed eighteen months ago.

A comparison of the model API endpoints eligible for zero data retention against those that are not, showing chat history, retrieval, evaluation and orchestration all falling on the ineligible side.

Anthropic's position on training is similar in shape: by default it does not use inputs or outputs from its commercial products for training, with the exception of data you explicitly submit as feedback, where thumbs-up and thumbs-down submissions are stored for up to five years, de-linked from user and customer identifiers. Worth knowing before you ship a feedback widget next to the AI output, because that widget moves data into a different retention regime than the feature beside it.

A "global" deployment can break a residency promise while the storage stays put

Microsoft's documentation for models sold through Azure is unusually direct about something most teams discover late.

Prompts, completions, embeddings and training data are not available to other customers, not available to OpenAI or other model providers, and not used to improve their models or to train foundation models without your instruction. The models are stateless and prompts are not used to retrain the base models. So far, so reassuring.

Then the deployment type changes the geography. For a deployment labelled Global, "prompts and responses may be processed in any geography where the relevant model sold by Azure is deployed". For DataZone, processing stays within the specified zone: a DataZone deployment created in an EU member nation may be processed in that or any other EU member nation. Data at rest, including the abuse monitoring store, remains in the customer-designated geography either way.

So storage residency and processing residency are two different commitments, and the cheaper, more available deployment type satisfies only one of them. If your contract says customer data is processed in the EU, a Global deployment is a breach with good uptime. Nothing in the code review catches this. It is a dropdown.

Abuse monitoring deserves the same attention. A sample of prompts and completions may be selected for review when indicators of potential abuse are detected; review is automated by default, with human review as necessary, and human reviewers can access the data only once it has been flagged. For deployments in the European Economic Area, those reviewers are located in the EEA. Managed customers can apply to modify abuse monitoring, and where that application is approved the storage and human review do not happen, though automated review may still run. Note what that application requires: an account team. If you do not have one, plan on the default.

Do we have to tell existing customers before switching an AI feature on?

If you process personal data on their behalf, usually yes, and the trigger is contractual rather than optional. Article 28(2) requires that controllers be informed of an intended addition of a subprocessor and given a real opportunity to object before that processor starts handling their data. Your own agreement sets the notice period, and that period sits on the launch critical path.

The practical failure here is sequencing. Teams build the feature, then discover a thirty-day notice obligation, then ship to everyone except the customers who matter most. The fix costs nothing if it happens first: read the notice clauses before the sprint, send the notices while the feature is being built, and design the per-tenant switch at the same time as the feature rather than after the first objection.

There is a separate transparency question about telling end users they are talking to an AI system, which in the EU has its own timetable; the EU AI Act deadlines that apply to a SaaS product are worth checking against your launch date rather than assumed.

The permission model is the integration

This is the part that breaks retrofits, and it has nothing to do with contracts.

Your existing product already enforces who can see what. Roles, tenants, record ownership, field-level restrictions, the sales rep who can read their own accounts but not their colleague's. A retrieval layer does not inherit any of that. Embed your documents into one index and the model will cheerfully answer a question using a document the asker was never allowed to open, and the answer will be fluent, correct and a data breach.

Enforcing permissions at query time rather than filtering afterwards is the only version that holds. That means every chunk in the index carries the access metadata of its source record, the retrieval query is scoped by the caller's identity before the model sees anything, and re-indexing runs when permissions change rather than when content changes. The last one is routinely missed: an employee leaving a team changes what they may retrieve, and nothing about the documents changed at all.

A diagram showing a question and the caller's existing roles passing through an identity-scoped retrieval filter before the model sees anything, producing only permitted sources and an audit record.

We have argued before that permissions decide the choice between retrieval and fine-tuning before cost does, and a retrofit is where that bites hardest. A fine-tuned model cannot forget what one tenant was allowed to know. Retrieval can be scoped. In a multi-tenant product that is close to decisive.

Two further integration costs, both usually underestimated. Your audit log has to record AI-assisted actions in a form that satisfies whoever reads it, which is more than a request log and overlaps the ground covered in securing enterprise AI; tracing and observability for AI features is a separate piece of work with its own storage cost. And if you are exposing your own product to other people's agents, the Model Context Protocol is where that interface lives, and data sent to third-party servers follows those services' retention policies rather than yours.

An invitation to discuss adding an AI feature to a product already in production with the Zyneto team.

Can we train on our own customers' data?

Only if your agreements say you can, and most standard processing terms do not. Processing data to deliver a service to a customer is not the same permission as using it to improve a product every customer shares, and the second one usually needs its own legal basis and its own consent.

Even where the paperwork allows it, the engineering argues against it in a multi-tenant product. A model trained on tenant data has no mechanism for forgetting one tenant, which turns an ordinary deletion request into a retraining project. Our general position is that getting the data ready matters far more than training on it, and that retrieval scoped by permission gets you most of the benefit with none of the deletion problem.

What you can usually do, and should: measure. Keep your own evaluation set of real questions and correct answers from your own domain, inside your own boundary. That set is the asset. It is what tells you whether a model change helped, and it is the reason the ineligibility of the Evals endpoint for zero retention matters more than it looks.

What it costs to run, every month, forever

The retrofit has a tail the original feature estimate never includes.

Inference is a variable cost on a product whose price is probably fixed. A seat-priced product with an AI feature has just acquired a per-use cost against a per-seat revenue line, and the margin depends on behaviour you do not control. What AI costs to run at real volume deserves modelling before launch rather than discovery in the second month's invoice.

Then the parts that are not the model. The retrieval index has to be kept current, which is a pipeline with failures. Human review needs somebody's actual hours when output quality matters. Monitoring an AI feature means watching output quality, not just latency and errors. And model versions move underneath you, which means the evaluation set gets re-run on a schedule rather than once.

Support load changes shape too. An AI feature generates a category of ticket your team has not handled before: not "it is broken" but "it is wrong". Answering those requires someone who can inspect what was retrieved and why, which is a tooling requirement disguised as a staffing one.

Where we would start, and what we would refuse

Start with one feature, one tenant, one measurable outcome, inside the boundary your existing contracts already allow. Not the most impressive feature. The one whose data is least contested. The failure pattern on a live product is the same one that strands greenfield work, which is why the pilot that never reaches production is worth reading alongside this.

The sequence that works on a live product: read the contracts and list every promise touching the data; pick the feature whose data carries the fewest promises; build retrieval scoped by your existing permission model before building any model interaction; keep an evaluation set from day one; launch to one tenant who has agreed in writing; then widen. Build the per-tenant off switch in that first pass. Retrofitting one after a customer objects means touching every code path the feature reaches, at the worst possible moment, under a deadline somebody else set. As a flag added on day one it costs an afternoon. The feature everybody demos first, which reads across all customer data to answer anything, is the one with the most promises attached and it should be last.

An invitation to scope a first AI feature for an existing product with the Zyneto team.

What we would refuse to build on a live product. An AI feature over data governed by an agreement nobody has read. A retrieval index that answers before it checks who is asking. A model trained on pooled tenant data in a product that owes any customer a deletion right. And anything priced as a flat add-on when its cost scales with use and nobody has modelled the heaviest user.

The honest summary is that retrofitting AI is a governance project with a model inside it. The teams that find this cheap are the ones who read their own agreements in week one, and the teams who find it expensive are the ones who read them in week nine, after the architecture was already chosen for them.

FAQs

Building fresh is usually cheaper per feature, because a new product has made no promises yet. An existing product carries data processing agreements, subprocessor lists, residency commitments and a permission model, and all four constrain the architecture. The model work is roughly the same in both cases; the difference is the work of satisfying commitments that already exist. Where the existing product wins is distribution: the feature reaches paying users immediately, which is worth more than the saving.

Usually yes, but not silently. A model vendor is another processor under GDPR Article 28, so a general written authorisation requires that you inform the controller of the intended addition and give them a real opportunity to object before processing starts. Your own agreement sets the notice period. Build the feature with a per-tenant switch so an objection does not block the launch for everyone else, and remember Article 28(4) leaves you fully liable for the vendor's performance.

Not by default at the major vendors, and each states it in writing. OpenAI says API data is not used to train or improve its models unless you opt in. Anthropic says it does not use inputs or outputs from its commercial products for training, except data you explicitly submit as feedback. Microsoft says prompts and completions for models sold by Azure are not available to model providers and are not used to train foundation models without your instruction. Retention for abuse monitoring is a separate question from training.

Zero data retention stops the vendor storing your request and response data, including the abuse monitoring logs that are otherwise kept for up to 30 days. OpenAI publishes which endpoints are eligible: Chat Completions, Responses, Decisions, Embeddings, audio transcription, translation and speech, Images, Moderations, Completions, Realtime and Live. Conversations, ChatKit threads, Assistants, Threads, Vector Stores, Files, audio Voices, Fine-tuning, Evals, Batches and Agents are not eligible. If you need zero retention, you build conversation history, retrieval and evaluation storage yourself.

It will if retrieval is not scoped by identity. A single index over all documents lets the model answer from a record the asker could not open, and the answer looks authoritative. The working pattern is to filter at query time using the caller's identity, carry each source record's access metadata on every indexed chunk, and re-index when permissions change rather than only when content changes. Pooled fine-tuning is worse: a trained model has no way to forget one tenant.

Check the deployment type, not just the region. Microsoft states that a Global deployment may process prompts in any geography where the model is deployed, while a DataZone deployment keeps processing inside the specified zone, and data at rest stays in the designated geography in both cases. Processing residency and storage residency are separate promises. If a contract commits you to in-region processing, the deployment type is a contractual control, not a performance setting.

Four, and only one is the model. Inference is a variable cost against what is usually fixed revenue, so margin depends on usage you do not control. The retrieval index needs a pipeline that keeps it current. Quality monitoring and an evaluation set re-run on model changes are continuous, not one-off. And support gains a new ticket category, the answer that is wrong rather than broken, which needs tooling to inspect what was retrieved before anyone can respond.

Vikas Choudhary

Vikas Choudhary

Vikas has around fifteen years of experience building software and now builds generative AI systems at Zyneto. His work covers retrieval augmented generation, agentic AI, knowledge graphs, AI memory, and the evaluation and guardrails that decide whether any of it is safe to put in front of customers. He has shipped enterprise copilots, document AI, chatbots and predictive analytics for e-commerce, fintech and marketing teams, and works day to day in Python, JavaScript and SQL. He follows multimodal models, business process automation and enterprise AI security closely, and mentors engineers moving into AI. He writes about architecture, inference cost and the failure modes that only show up at production scale.

Let's make the next big thing together!

Share your details and we will talk soon.

Phone

We reply to every enquiry within one business day.

WhatsApp
Email
Book a Meeting