Announcing Vidyut-Prakriya Benchand SGram-14B for verifiable Sanskrit tasks
for verifiable Sanskrit tasks
Today, we’re excited to announce Vidyut-Prakriya Bench, a benchmark measuring LLMs on verifiable sanskrit grammar tasks generated from the vidyut-prakriya rust derivation engine, and SGram-14B, our own model which tops this benchmark with statistical significance. In this post, we cover our methodologies and results.
At a high level, our contributions are as follows:
- A verifiable deterministic benchmark testing Sanskrit grammar ability across four domains: tinantas, krdantas, subantas, and taddhitantas measuring skills in verb, noun, and adjective derivation and inflection.
- A post-trained model we call SGram-14B which beats frontier LLM contenders Claude Opus 5.5 and Gemini-4-Argon with statistical significance
This work would not be possible without gracious advising of Rohan Pandey and Lambda Labs compute from 1517. Special thanks to early readers Aryaman Arora, Mars Xiang, Marc Melyikan, and Kento Nishi.
Overview
Sanskrit grammar is largely algorithmic in its grammar, and is codified in 4,000 concise rules in the Aṣṭādhyāyī from between 6th and 4th centuries BCE. Given the rise in RLVR for LLMs, the natural step is to explore creating high-fidelity morphology Sanskrit tasks with verifiable answers. This is made possible through Vidyut-Prakriya, a Rust library that derives Sanskrit word forms with step-by-step derivations.
Task types
We now present the four task types we test in this benchmark, each of which are supported by one of Vidyut-Prakriya's core morphology functions1.
TiṅantaVerb Derivations
A tiṅanta is a conjugated verb form, which is derived from a specified verbal root (i.e. a dhatu) and a chosen lakāra, prayoga, puruṣa, and vacana. For a given dhatu, our full 4-D Cartesian product contains 10 × 3 × 3 × 3 = 270 candidate combinations (of which 190 are eligible after restricting bhāve to third-person singular.)
Our dhatus for this task and the next are sourced from Vidyut-Prakriya's Dhatupatha inventory, which provides 2,229 dhatus.
KṛdantaVerb Derivations II
A kṛdanta is formed from a dhatu verbal root and a chosen kṛt suffix. It is used to express ideas such as the doer of an action, the action itself, or its result (e.g. "teach -> teacher", "write -> written"). Our dataset covers 14 suffixes across 2,229 roots.
SubantaNoun and Adjective Forms
A subanta is a noun or adjective whose form reflects its gender, number, and role in a sentence. Subantas are derived from a given nominal stem (the word's base form, known as a prātipadika), along with a chosen liṅga, vibhakti, and vacana. The model must produce the correct word form.
Our nominal stems for this task and the next come from MWlexnorm, which extracts grammatical information from the Monier-Williams Sanskrit–English Dictionary. We use the genders listed in the dictionary and, after resolving their stem forms, removing duplicates, and filtering Vidyut-Prakriya unsupported entries, yield 74,193 stem–gender pairs located here. We then take each pair and vary vibhakti and vacana which gives 8 * 3 = 24 tasks.
TaddhitāntaNoun and Adjective Derivations
A taddhitānta is formed from a nominal stem and a chosen taddhita suffix. It expresses a related idea, such as possession or a quality (e.g. "wealth -> wealthy", "kind -> kindness"). We use the same 74,193 stem–gender pairs as before and attempt 175 suffixes for each, ignoring all those which Vidyut-Prakriya fails to parse.
Task examples
Our tasks are hydrated from carefully designed prompt templates. Our benchmark consists of 5,000 tasks per category. See here for tasks and results.
Post-training SGram-14B
We fine-tune Qwen3-14B against these set of tasks as well using standard SFT. We need to state in this section:
- initial pass rates of this model (0)
- the accuracy we achieve like pass@{1, 16}
- amount of tokens seen? (4.7M loss-bearing tokens in 25% across each category)
Results
The overall score summarizes 20,000 tasks, but it does not tell us how much output a model generates or how much each trial costs. We examine those tradeoffs below. All single-response comparisons use the same tasks and the original exact-answer scoring rule.
How much output buys a correct answer?
Our efficiency plot uses one point per model, calculated from every task in the selected category. A single trial is either correct or incorrect; a small random sample would obscure accuracy differences near the top of the benchmark. Switch between total output tokens, including provider-reported reasoning, and recorded API cost per trial. The horizontal axis shows the median per trial across all 20,000 tasks, or across the 5,000 tasks in a selected category. Accuracy is calculated over that same full task set.
Longer outputs do not reliably correspond to higher accuracy across these models. This is a descriptive comparison: model capability, task difficulty, tokenizers, prompts, and decoding settings all differ. SGram is shown on the token plot, but omitted from the API-cost plot because its inference cost was not measured. Missing cost is not zero cost.
Scoring, source data & figure methods
Each API model and SGram is evaluated on the same 20,000 tasks, with 5,000 in each family. Main comparisons use one saved response per task. SGram uses the preselected sample 0. Accuracy is exact membership in the reference set; incomplete responses are failures. Prompts and decoding differ between API and SFT evaluations.
Efficiency shows median total output tokens (including reported reasoning) or median recorded input + output cost per trial, across all selected tasks including failures. Accuracy uses the full selected task set. Missing measurements are omitted, not replaced with zero. Token counts use different tokenizers; cost reflects the recorded pricing tiers before credits. No claim of a causal effect of token use is made.
Download figure data (JSON) · Statistical report · Error audit