AI Model Cards - Documenting LLM Capabilities and Limitations

An AI model card is a structured document that describes what a machine learning model does, how it was trained, where it performs well, and where it fails. As ShopMax India deploys LLMs for product recommendations, customer support, and fraud detection across Mumbai, Bangalore, and Delhi, stakeholders - from legal and compliance to product managers and auditors - need a clear, standardized record of each model's capabilities and limitations. Model cards make AI systems transparent and accountable, and are increasingly required by enterprise procurement processes and emerging AI regulations.

A well-structured model card covers: model details (name, version, architecture, training date), intended use (tasks it is designed for, target users), out-of-scope uses (tasks it should not be used for), training data (sources, size, preprocessing steps), evaluation metrics (accuracy, latency, token usage), limitations (known failure modes, demographic biases, edge cases), and ethical considerations (privacy risks, potential harms). Model cards are stored alongside the model artifact in a version control system and updated whenever the model is retrained or fine-tuned.

The example below defines a Python dataclass for a ShopMax India model card and generates a formatted markdown report. It also demonstrates automated evaluation to populate the performance section, making the card self-updating when plugged into a CI pipeline.

It gives the following output,

# Model Card: ShopMax Product Recommender v2.1.0

## Description
GPT-4o fine-tuned on ShopMax India catalog data to recommend accessories and related products.

## Intended Uses
- Recommending accessories for electronics purchases
- Answering product compatibility questions
- Cross-sell suggestions for orders above Rs 10,000

## Out-of-Scope Uses
- Medical or safety-critical advice
- Legal or financial decisions
- Customer identity verification

## Training Data
ShopMax India product catalog (2M listings), 500K customer Q&A pairs...

## Performance Metrics
- Recommendation relevance (human eval): 87%
- Average latency: 340ms
- Token cost per request: Rs 0.02

## Limitations
- May suggest out-of-stock products if catalog not refreshed within 24 hours
- Lower accuracy for niche product categories with fewer than 100 training examples
- Does not understand regional language queries (Hindi, Tamil, Telugu)

## Ethical Considerations
- Model may reflect historical purchasing bias toward premium products
- Do not use to recommend based on inferred demographic data

Store model cards in your Git repository alongside the model weights or API configuration, and treat them as living documents - update the metrics section on every evaluation run and the limitations section whenever a new failure mode is discovered in production. For ShopMax India's AI governance process, require a completed model card as a gate before any new LLM goes to production. Link the model card to the incident log (see AI Incident Response) so auditors can trace every production issue back to a known limitation or an unanticipated failure mode, making compliance reviews significantly faster.