RAG vs Fine Tuning: Permissions Decide It Before Cost Does

Someone senior has read that fine tuning makes a model "know your business", and has asked why the assistant cannot simply be trained on the company wiki. It is a reasonable question and the honest answer takes about two minutes.

The RAG vs fine tuning decision in an enterprise is rarely settled by accuracy and rarely settled by cost. It is settled by a property that is easy to overlook until an audit finds it. Fine tuning dissolves your documents into model weights, and weights cannot be filtered per user. Retrieval can. If your content has an access model, that single fact does most of the deciding before anyone opens a spreadsheet.

Everything else in this piece is the detail underneath that sentence.

The two things fine tuning actually changes

Fine tuning is very good at two jobs and routinely bought for a third it cannot do.

Job one, format. Getting output in a shape you can parse every time. A specific JSON structure, a house tone, a fixed set of labels, a report layout your downstream system expects. If you have written eleven paragraphs of prompt trying to stop a model adding a friendly preamble, fine tuning solves that permanently and cheaply.

Job two, behaviour on a narrow task. Classification into your own taxonomy, extraction from documents whose layout you see a thousand times a week, routing decisions with your own edge cases. Anywhere a general model keeps making the same specific mistake, a few hundred good examples usually fix it.

The job it cannot do, knowledge. Training on the wiki does not give the model a copy of the wiki. It shifts the probability distribution so the model is more likely to produce text that resembles the wiki. Those are different things, and the gap between them shows up as confident, well formatted, wrong answers. There is no citation to check, because there is nothing to cite.

That distinction is the one to carry into the meeting. Fine tuning changes how the model speaks and behaves. Retrieval changes what it can see.

Permissions, and why this usually ends the argument

Now the part that decides most enterprise cases.

Take an ordinary situation. Your document set includes board minutes, salary bands, a customer contract with a confidentiality clause, and the general handbook. Four hundred people should see the handbook. Nine should see the salary bands. Two should see the contract.

With retrieval, the permission check happens at query time. The user's identity resolves to a set of allowed document ids, the vector search is filtered to that set, and the model only ever sees passages the user was already entitled to read. The access model is the one you already run, enforced at the point of retrieval. Adding a person to a group takes effect on their next question.

With fine tuning, all four documents were in the training data. The salary bands are now part of the parameters. There is no filter to apply, because there is no document to exclude. The model has one behaviour for everyone who queries it.

The workaround people reach for is a separate fine tuned model per permission tier. Try costing that. Four tiers means four training runs, four evaluation cycles, four deployments, four sets of drift, and a new one every time somebody invents a fifth tier. And it still fails the ordinary case, which is not tiers at all but per document access that changes weekly.

There is no clean fix. That is the point. If your content has an access model, retrieval is the only one of the two that can honour it, and this is worth resolving before comparing anything else.

A related point that lands the same way: the right to be forgotten. A document that must be deleted can be removed from an index in seconds and is genuinely gone from every future answer. Removing it from a set of weights means retraining.

Freshness is a workflow problem, not a budget line

Comparisons usually file freshness under cost. That understates it.

A policy document changes on a Tuesday afternoon. Under retrieval, someone saves the file, the pipeline re embeds the changed chunks, and the assistant is answering from the new version within minutes. No release, no approval, no engineer.

Under fine tuning, that same change enters a queue. It waits for enough other changes to justify a training run. Then a training run, an evaluation pass against your regression set, a deployment, and a rollback plan. Call it a fortnight in a well run team, and in most teams it quietly becomes a quarterly cadence.

Which means the real cost of fine tuning for knowledge is not the compute. It is that your assistant is confidently answering from a version of the truth that is on average six weeks old, and nobody can tell by looking at the answer.

Where fine tuning wins outright

This is not a piece against fine tuning. There are cases where retrieval is the wrong tool and we recommend against it.

1. High volume, narrow, stable tasks. Classifying 40,000 support tickets a day into your own 60 category taxonomy. Retrieval adds an embedding call, a search and a much larger prompt to every single one of those. A small fine tuned model does it faster, cheaper per call, and more consistently. Nothing needs citing because there is no source document, only a decision.

2. Strict output shape at scale. When every response must parse, every time, and a malformed response costs you a failed downstream job.

3. A house voice that prompt engineering keeps missing. Marketing and support copy where the tone is a real requirement and your prompt has grown to a page trying to describe it.

4. Latency floors. Retrieval costs you a search round trip and a longer prompt. Where you are inside a tight budget, removing the retrieval hop matters.

Notice the pattern. Every one of those is behaviour or format on data with no per user access model. That is the safe zone.

What each one costs, with the hours shown

The rate is $40 to $100 per hour by role. Pipeline and interface work sits near the floor, permission modelling and evaluation near the ceiling, and mixed teams blend to around $65.

A retrieval build.

Module	Hours	What it covers
Ingest, parsing and chunking	60 to 120	Format handling, chunk strategy, overlap, metadata capture
Embedding and vector store	40 to 90	Model choice, index build, incremental update path
Retrieval and reranking	60 to 130	Hybrid search, reranking, the tuning that turns 60 per cent into 90
Permission filter at query time	50 to 110	Identity to document set, enforced in the query rather than the UI
Evaluation set and scoring	50 to 100	Real questions, graded answers, a number that moves when quality moves
Answer surface with citations	60 to 120	The passage shown next to the claim, which is what earns trust

Worked example: 90 plus 65 plus 95 plus 80 plus 75 plus 90 equals 495 hours. That is $19,800 at $40, $49,500 at $100, about $32,175 at a $65 blend. The full span is 320 to 670 hours.

A fine tuning build.

Module	Hours	What it covers
Dataset construction and labelling	120 to 300	The dominant line, and the one always underestimated
Training runs and evaluation	60 to 140	Sweeps, regression set, the runs that do not work
Serving and version control	50 to 110	Deployment, rollback, knowing which version answered
Retraining cadence, per quarter	40 to 90	Recurring, not one off

Worked example: 200 plus 95 plus 75 plus 60 equals 430 hours, about $27,950 at the blend, with the fourth line repeating every quarter. The full span is 270 to 640 hours.

The headline numbers land close together, which is why cost is a poor tie breaker. Read the shapes instead. The retrieval build is mostly one off with a cheap update path. The fine tuning build carries a recurring line forever, and the dataset row is the one that doubles when the labelling turns out to be harder than the pitch suggested.

Excluded from both: inference spend, GPU time and any model licence. Those are usage, not build.

The combination that works, and the one that wastes money

Most published comparisons end with use both. That is true and unhelpfully vague, because there is a good version and a bad one.

The good combination. Retrieval supplies the facts. A small fine tuned model handles the format and the house behaviour on top. The knowledge stays in the index where it can be filtered, updated and cited, and the fine tuning does the two jobs it is genuinely good at. This is what a mature deployment usually looks like at eighteen months.

The wasteful combination. Fine tuning on the same corpus you are also retrieving from, in the hope that the model will "understand the domain better". You now maintain two copies of the same knowledge with different update cycles, and when they disagree, the weights win silently. We have been called in to unpick exactly this twice.

The order matters too. Build retrieval first, measure, and fine tune afterwards only against a specific failure you can name. Fine tuning first means you never find out how much of the problem retrieval alone would have solved, which is usually most of it.

Where retrieval builds actually fail, since that is the honest counterargument

Anyone who has run one of these will point out that retrieval projects fail too, and they are right. The failures are just different in kind, and all four are fixable in a way that a stale set of weights is not.

1] Chunking that cuts through meaning. The default advice is a fixed window with an overlap, and it works until the corpus contains tables, numbered clauses or anything with structure. A contract clause split across two chunks retrieves as two half answers, and neither half is wrong enough to look wrong. Structure aware chunking costs more in the ingest module and is the single highest return change we make on struggling systems.

2] Pure vector search on short, specific queries. Someone asks for an invoice number, a part code, or a policy reference. Embeddings are excellent at meaning and mediocre at exact strings. A hybrid of keyword and vector search, with the two result sets merged and reranked, typically moves a system from around 60 per cent useful answers to around 90. That reranking step is most of the 60 to 130 hours in the retrieval row and it is the line people cut first.

3] No evaluation set, so no idea. Without 80 to 150 graded questions, every change is argued rather than measured, and the team spends three weeks tuning a prompt that made things worse. Building the set takes two days with a subject matter expert and it is the cheapest insurance in the project.

4] Permissions bolted on afterwards. Covered above, and worth restating because it is the one that ends careers rather than sprints.

Compare that list to the fine tuning failure list, which is shorter and worse: the dataset was smaller or noisier than promised, and you find out after the training run.

What this means for the two thirds of cases that are mixed

Most real estates are not one thing. A single company has a support taxonomy that should be fine tuned, a policy library that must be retrieved with filters, and a set of product documents anyone may read.

Resist the urge to make one decision for all of it. Decide per workload, using the three questions, and accept that a mature deployment runs both.

What that looks like in practice, at the eighteen month mark, is usually three components. A retrieval service over the permissioned corpus, serving the assistant. A small fine tuned classifier doing high volume routing with no access model attached. And a shared evaluation set that both are scored against, so a change to one cannot quietly degrade the other.

The part teams get wrong is the third component. Two systems with separate quality measures drift apart, and the first sign is a user complaining that the assistant "used to be better", which is unfalsifiable without numbers.

A decision rule you can defend in a meeting

Three questions, in this order.

One. Does the content have an access model? If different people may see different documents, retrieval is required. This is not a preference and it is where the conversation should stop.

Two. Does the content change more often than quarterly? If yes, retrieval, because the retraining cadence will not keep up and nobody will notice it has fallen behind.

Three. Is the problem knowledge or behaviour? Knowledge means retrieval. Format, tone, classification or routing means fine tuning is a strong candidate, and probably a cheaper and faster one.

If all three point at fine tuning, fine tune, and do not let anyone talk you into a vector database you do not need.

What we would build first

For a first enterprise assistant over internal content, we would build retrieval, with the permission filter in the first release rather than the second. Retrofitting an access model onto a working assistant is the most expensive rework in this space, because by then people have seen answers they should not have seen and the conversation is no longer technical.

Then an evaluation set of 80 to 150 real questions with graded answers, before any tuning. Without it, every improvement is an opinion.

Then citations in the answer surface, because a cited answer that is 85 per cent right is more useful than an uncited one that is 92 per cent right. Users can check the first kind.

One more thing we would insist on in the first release, because retrofitting it is miserable: log the retrieved passages alongside every answer, keyed to the question and the user.

That log is worth more than it sounds. It is your evaluation set for free, growing daily from real usage rather than from what somebody guessed users would ask. It is the only way to answer the question a director will eventually ask, which is why the assistant said that on 14 March. And when someone reports a bad answer, you can see in ten seconds whether retrieval returned the wrong passages or the model mishandled the right ones. Those two failures look identical from the outside and have completely different fixes.

Teams that skip the log spend the first six months guessing, and the guesses are usually wrong in the same direction: they blame the model and tune the prompt, when the retrieval never surfaced the document at all.

Fine tuning comes after all of that, aimed at a named failure, and quite often it turns out not to be needed at all.