NVIDIA, in collaboration with IIT Madras and its AI4Bharat lab, has announced the creation of a massive synthetic pretraining corpus for Indian languages, totaling approximately 2.8 trillion tokens. This significant development, detailed in an announcement on Hugging Face dated October 16, 2026, also includes the release of an open, GPU-accelerated data generation pipeline that developers can use to create similar datasets.
The newly introduced corpus, named Nemotron-pretraining-Indic-v1, covers 22 Indian languages that are constitutionally recognized in India, representing languages spoken by nearly 1.2 billion citizens. This initiative addresses a critical data gap in the development of artificial intelligence for Indian languages, a challenge that has historically lagged behind English due to a scarcity of high-quality training data. The open pipeline provides the complete generation recipe, including prompts, Large Language Model (LLM) workflows, and quality assurance checks, enabling developers to adapt it for their specific needs and source materials.
Information was available with The Chenab Times that the corpus comprises 2,601.0 billion native-script tokens and 224.6 billion romanized tokens, as measured by the Gemma-4 tokenizer. The counts reflect the retained corpus, including any accepted rewrites, rather than the total token budget of a training run. Across 34 language-script configurations, the corpus offers native-script data for all 22 languages and romanized text for twelve. Twelve languages, including Bengali, Gujarati, Hindi, Kannada, Malayalam, Marathi, Nepali, Odia, Punjabi, Tamil, Telugu, and Urdu, feature both native-script and romanized data. Ten languages—Assamese, Bodo, Dogri, Kashmiri, Konkani, Maithili, Manipuri, Sanskrit, Santali, and Sindhi—have native-script data exclusively.
The creation of this extensive dataset involved translating high-quality English web data at scale, with each translation being scored by an LLM judge. Only translations scoring 75 out of 100 or higher were retained, while lower-scoring translations were repaired and rechecked to ensure quality. The Nemotron-pretraining-Indic-v1 corpus is built upon 15 subsets of Nemotron-CC-v2, which includes data from web content, question-and-answer pairs, mathematics, code, and specialized domains, all translated from English.
This project is part of a broader effort to advance AI development in India, a market with over 700 million internet users and a burgeoning developer ecosystem. NVIDIA has previously released Sovereign AI datasets for the United States and Japan, and Nemotron-Personas-India, an open synthetic dataset of Indic personas, was developed using NVIDIA’s NeMo Data Designer microservice. This new Indic pretraining corpus complements existing Hindi evaluation datasets, supporting a comprehensive pipeline from synthetic data generation to model evaluation for Indian AI systems.
❤️ Support Independent Journalism
Your contribution keeps our reporting free, fearless, and accessible to everyone.
Or make a one-time donation
Secure via Razorpay • 12 monthly payments • Cancel anytime before next cycle
(We don't allow anyone to copy content. For Copyright or Use of Content related questions, visit here.)
The Chenab Times News Desk
