How I built and tuned a RAG agent for call transcripts

I once had to build a tool that could answer questions about sales calls. The problem was simple to state and hard to solve. There were hundreds of hours of call transcripts, long and messy, and the answer to a question like "why is this deal slipping" was scattered across many different conversations. No human had time to reread all of it. So I built a RAG agent to do the reading.

Here is how it worked, and how I made it actually reliable, in the plainest terms I can manage.

What a RAG agent is

RAG stands for retrieval augmented generation. The idea is simple. Instead of asking the AI to answer from its own memory, you first go and find the relevant pieces of text, and then you hand those pieces to the model and ask it to answer using only them. Two steps. First you retrieve, then you generate.

The second step, generating the answer, is the part everyone thinks about. The first step, retrieval, is really just a search problem. And that search step turned out to be where all the difficulty lived.

Why retrieval is the whole game

Here is the thing I learned fast. If the search step fails to find the piece of transcript that actually holds the answer, the model has nothing useful to work with. It will either admit it does not know, or worse, it will fill the gap with a confident guess. A brilliant model handed the wrong context still gives you a wrong answer.

So the quality of the whole tool came down to one question: does the search reliably find the right pieces of transcript? Everything else was secondary.

The basic build

The setup was straightforward.

First, I split each transcript into chunks. My first attempt used fixed size chunks, which was a mistake. It cut sentences in half and split a single point across two chunks, so the meaning got broken. I switched to splitting by speaker turn and by topic, so each chunk was one coherent piece of the conversation.

Second, I turned each chunk into an embedding, which is just a numeric fingerprint of its meaning, and stored all of them in a vector index. Chunks that mean similar things end up with similar fingerprints.

Third, I attached labels to every chunk: who was speaking, which call it came from, the date, and which deal it belonged to. This let me filter searches later.

At question time, the system turned the question into the same kind of fingerprint, pulled the closest chunks, and handed them to the model to write the answer.

The number that mattered: recall

Recall answers a very simple question. Of the transcript pieces that actually contained the answer, how many did the search bring back? If the answer lived in five chunks and my search surfaced four of them, recall was four out of five.

I cared about recall more than almost anything else, and here is why. A missed chunk is an invisible failure. The tool still produces an answer, so it looks like it worked. It just answered from incomplete information, and nobody can tell. High recall meant the model almost always had the context it needed. Low recall meant it was quietly flying blind.

How I measured it

You cannot improve what you do not measure, so I built a small evaluation set by hand. I wrote real questions, and for each one I marked which transcript chunks truly held the answer. That gave me a ground truth to check against.

Then I ran the search and looked at whether the chunks I had marked showed up in the results. I tracked recall at k, which just means recall when the search returns its top k results, for example the top five or the top ten. Now I had a real number to move.

How I pushed recall up

A handful of changes did most of the work.

Better chunking came first. Because each chunk was now one coherent piece of conversation, the answer usually sat inside a single chunk instead of being split across two, so the search could actually find it.

Then I added hybrid search. Pure meaning based search sometimes missed exact words, like a specific product name or a number. I ran a keyword search alongside the meaning based one and combined the results, which caught the cases where the exact term was what mattered.

Next I let the first search pull a wider set of candidate chunks, and then added a second, more careful step to reorder them and keep only the best. This raised the odds that the right chunk made it into the final set, without burying the model in noise.

I also had the system clean up the question before searching. People ask messy questions, so rewriting the question into a clearer search first helped a lot.

Finally, I used the labels. For a question about one deal or one time period, filtering by those labels first shrank the haystack, which made the right needle much easier to find.

The tradeoff I had to watch

Recall is easy to fake. You can always return more chunks and catch more of the right ones by luck. But if you hand the model fifty chunks and forty of them are irrelevant, two bad things happen. The answer gets noisier because the model has to wade through junk, and the whole thing costs more and runs slower.

So I balanced recall against precision, which is how many of the returned chunks were actually relevant. The reordering step was the key. It let me keep recall high while keeping the final set of chunks clean.

What it added up to

Once retrieval was reliable, the rest fell into place. The model got the right context most of the time, so its answers became trustworthy, and every answer could point back to the exact moment in the call it came from. That last part mattered as much as the accuracy did. People trust an answer they can go and verify for themselves.

What I would tell someone starting out

The temptation with RAG is to spend all your time on the model and the prompt. In my experience the model is rarely the bottleneck. The retrieval is. Build a small evaluation set early, measure recall honestly, and fix the search before you touch anything else. A simple model with great retrieval beats a great model with sloppy retrieval every single time.