Evaluating AI Across Agriculture and Law Using Community and Expert Input
Karya is evaluating conversational AI systems in real-world Indian contexts, in partnership with Anthropic.
Building a tribal language voice assistant co-built with communities
Until recently, Bhili, a tribal language spoken by millions of people across western and central India, had limited representation in the country's digital infrastructure. Karya, working with the Bhil community, has helped build a Bhili language system that now lives on two national-scale platforms.
400+
contributors
30
days
30,000+
translation pairs
120
hours of speech data
Karya built and deployed the system in partnership with the District Administration of Nandurbar, Government of Maharashtra, with contributions from more than 400 people across the district. In 30 days, this work produced an end-to-end AI pipeline in Bhili, drawing on over 30,000 translation pairs, 120 hours of speech data, and 6,500 glossary terms generated by community members.
Through MahaVISTAAR, a platform used by more than 13 million farmers across Maharashtra, this system gives farmers access to agricultural services in Bhili. Through BHASHINI, it enabled the live broadcast of Mann Ki Baat, the Prime Minister's monthly radio address, in Bhili for the first time.
Related
Building Real-World AI Evaluation with Samiksha
Samiksha is a rigorous, community- and expert-driven evaluation framework for benchmarking AI models in real Indic-language contexts, developed by Karya in collaboration with the Collective Intelligence Project and Microsoft.