	In Browser
	StumbleUpon
	del.icio.us
	Google
	Google Buzz
	reddit
	LinkedIn

	Facebook
	Twitter
	Linkedin
	E-Mail

Generative AI > RAG Pipelines > Cross-Lingual RAG - Multi-Language Retrieval with Multilingual Embeddings

Cross-Lingual RAG - Multi-Language Retrieval with Multilingual Embeddings

Author: Venkata Sudhakar

Cross-lingual RAG enables ShopMax India to answer customer queries in Hindi, Tamil, Telugu, or other Indian languages even when the product knowledge base is stored entirely in English. Indian e-commerce customers often prefer regional languages for product queries, but maintaining separate knowledge bases per language is impractical. Cross-lingual retrieval bridges this gap by using multilingual embeddings that map queries and documents from different languages into a shared semantic space.

The multilingual-e5 and paraphrase-multilingual-MiniLM models from Hugging Face encode both Hindi and English text into the same embedding space, allowing a Hindi query to retrieve the most semantically similar English documents without any translation step at retrieval time. After retrieval, the English context plus the original language query are passed to a multilingual LLM (Claude supports Hindi natively), which generates the answer in the customer's language.

The following example builds a cross-lingual RAG pipeline for ShopMax India. English product documents are indexed with multilingual embeddings, and Hindi customer queries retrieve the correct English documents for Claude to answer in Hindi.

import anthropic
from sentence_transformers import SentenceTransformer
import numpy as np

client = anthropic.Anthropic(api_key="sk-ant-...")
embedder = SentenceTransformer("paraphrase-multilingual-MiniLM-L12-v2")

product_docs = [
    "Sony WH-1000XM5 headphones: 30-hour battery, noise-cancelling, Rs 29990, available in Mumbai and Bangalore.",
    "Samsung Galaxy S24 Ultra: 200MP camera, 12GB RAM, Snapdragon 8 Gen 3, Rs 134999, pan-India delivery.",
    "Dell XPS 15 9530 laptop: Intel i7, 32GB RAM, 1TB SSD, 15.6 inch OLED, Rs 135000, available in Delhi.",
    "Apple iPhone 15 Pro: 48MP camera, USB-C, titanium body, A17 Pro chip, Rs 134900, all major Indian cities."
]

doc_embeddings = embedder.encode(product_docs)

def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

def cross_lingual_rag(query, top_k=2):
    query_embedding = embedder.encode([query])[0]
    scores = [cosine_similarity(query_embedding, doc_emb) for doc_emb in doc_embeddings]
    top_idx = sorted(range(len(scores)), key=lambda i: scores[i], reverse=True)[:top_k]
    context = "\n".join([product_docs[i] for i in top_idx])
    msg = client.messages.create(
        model="claude-opus-4-7",
        max_tokens=200,
        system="You are ShopMax India assistant. Answer in the same language as the customer query. Use only the provided product context.",
        messages=[{"role": "user", "content": f"Product information:\n{context}\n\nCustomer question: {query}"}]
    )
    return msg.content[0].text

queries = [
    "Sony headphones ki battery kitne ghante chalti hai?",
    "Samsung Galaxy S24 ki kimat kya hai?",
    "Dell laptop Delhi mein milta hai kya?"
]

for q in queries:
    answer = cross_lingual_rag(q)
    print(f"Q: {q}")
    print(f"A: {answer}")
    print()

It gives the following output,

Q: Sony headphones ki battery kitne ghante chalti hai?
A: Sony WH-1000XM5 headphones ki battery 30 ghante chalti hai noise cancelling ke saath. Ye Mumbai aur Bangalore mein Rs 29,990 mein available hai.

Q: Samsung Galaxy S24 ki kimat kya hai?
A: Samsung Galaxy S24 Ultra ki kimat Rs 1,34,999 hai aur ye poore India mein delivery ke saath available hai.

Q: Dell laptop Delhi mein milta hai kya?
A: Haan, Dell XPS 15 9530 laptop Delhi mein available hai. Iska price Rs 1,35,000 hai aur isme 32GB RAM aur 1TB SSD hai.

For ShopMax India, deploy the multilingual embedding model as a separate microservice with GPU acceleration for fast encoding. Cache query embeddings using the query string as the key to avoid re-encoding repeated queries. Test retrieval quality for each target language using a labeled evaluation set - cross-lingual retrieval accuracy for closely related languages like Hindi and Marathi is usually high, but for more distant languages like Tamil you may need a larger multilingual model like multilingual-e5-large. Monitor which languages your customers actually use and prioritize embedding quality testing for those first.

Send your comments, suggestions or queries regarding this site to [email protected].