Karya is evaluating conversational AI systems in real-world Indian contexts, in partnership with Anthropic.
10,000
community-sourced queries
30,000+
evaluation tasks
7
Indian languages
Hybrid human and expert evaluation
The work focuses on how models perform in high-impact domains such as agriculture, law, and education, using 10,000 community-sourced queries across seven Indian languages. Rather than relying on abstract benchmarks, evaluations are designed to reflect real usage conditions and decision-making contexts.
The approach combines large-scale public evaluation with expert review. Over 30,000 evaluation tasks assess model outputs for correctness, fluency, and trustworthiness, alongside detailed input from domain experts, including agronomists and legal professionals. This enables a deeper understanding of how AI systems behave in practice, where accuracy alone is not sufficient.
Until recently, Bhili, a tribal language spoken by millions of people across western and central India, had limited representation in the country's digital infrastructure. Karya, working with the Bhil community, has helped build a Bhili language system that now lives on two national-scale platforms.
Samiksha is a rigorous, community- and expert-driven evaluation framework for benchmarking AI models in real Indic-language contexts, developed by Karya in collaboration with the Collective Intelligence Project and Microsoft.