RAG Pipeline Cost Calculator

RAG (Retrieval-Augmented Generation) has become one of the most widely deployed patterns for giving an LLM access to information beyond its training data, and understanding its real cost structure — which is genuinely different from a simple chatbot's — matters for budgeting it accurately rather than being surprised by the bill.

A RAG pipeline handles each incoming query in roughly three steps: embed the query into a vector representation, use that vector to retrieve the most relevant chunks of context from a document corpus (a vector database search), and then send the original query plus that retrieved context to an LLM to generate a final answer. Each of these steps carries its own cost, but they're wildly different in magnitude — query embedding is typically the cheapest step by a wide margin, since embedding models are priced far lower per token than generation models, and a single query is usually short. Vector database retrieval itself is often bundled into infrastructure cost rather than billed per-query directly. The generation step, by contrast, is where the real cost concentrates, precisely because it includes not just the user's short query but also however much retrieved context was pulled in to answer it.

This is exactly why retrieved context size is the single most important lever for controlling RAG cost at scale. A system retrieving 4 chunks of 500 tokens each (2,000 tokens of context) on every query, multiplied across tens of thousands of monthly queries, adds up to a genuinely significant recurring input-token cost — often larger than the cost of the query itself and the generated answer combined. Many RAG systems retrieve considerably more context than a task strictly requires, either from an overly generous default configuration or from not having tested whether a leaner retrieval setup maintains acceptable answer quality — testing this tradeoff directly, rather than assuming more retrieved context is unconditionally better, is one of the most effective cost-reduction levers available for an already-deployed RAG system.

It's worth being clear that this calculator's per-query pipeline cost is only part of a RAG system's total cost of ownership — the separate, often one-time-but-recurring cost of actually embedding and indexing your document corpus (covered by the Embedding Cost calculator) and the ongoing cost of storing that vector index (covered by the Vector Database Storage calculator) both add to the full picture. A RAG system with a large, frequently-updated document corpus can find corpus-side embedding and storage cost meaningfully rivaling or even exceeding its query-time generation cost, particularly for a corpus that's re-embedded regularly as underlying documents change — worth budgeting as a genuinely separate, ongoing line item rather than a one-time setup cost that's paid once and forgotten.