Building a Production RAG API with FastAPI and ChromaDB

Building a RAG REST API with FastAPI gives ShopMax India a production-ready endpoint that their mobile app, website, and internal tools can all call for product Q and A. Rather than embedding the RAG pipeline inside each client application, a central FastAPI service manages the document index, handles retrieval, and calls the LLM - ensuring all clients use the same retrieval logic and making upgrades seamless. The API accepts a query string and returns a structured JSON response with the answer and source documents.

A well-designed RAG API has three endpoints: POST /query (submit a question and get an answer), GET /health (check if the service and dependencies are up), and POST /index (add new documents to the knowledge base). FastAPI's async support allows multiple concurrent queries without blocking, and Pydantic models enforce request and response schemas. ChromaDB or a similar vector store runs as a dependency, and the LLM client is initialized once at startup and reused across requests.

The following example builds a complete FastAPI RAG service for ShopMax India. The API exposes a /query endpoint backed by BM25 retrieval and Claude for answer generation, with a health check and structured JSON responses.

It gives the following output,

# GET /health
{"status": "ok", "docs_indexed": 4}

# POST /query
# Request: {"question": "Which headphones are available in Mumbai?"}
{
  "answer": "The Sony WH-1000XM5 headphones are available in Mumbai at Rs 29,990 with 30-hour battery life and noise-cancelling.",
  "sources": [
    "Sony WH-1000XM5: 30-hour battery, noise-cancelling, Rs 29990, Mumbai and Bangalore."
  ],
  "query": "Which headphones are available in Mumbai?"
}

For ShopMax India in production, add request authentication using FastAPI's HTTPBearer dependency to ensure only authorized clients can query the API. Add rate limiting with slowapi to prevent abuse. Use async LLM calls (anthropic.AsyncAnthropic) and async BM25 retrieval to handle concurrent requests without blocking. Deploy on Cloud Run or Kubernetes with horizontal scaling - the RAG service is stateless since the index is loaded at startup, making it trivially scalable. Log all queries, retrieved sources, and response times to BigQuery for monitoring and continuous improvement.