The variance btween Indo-Aryan - the language of the Rg Veda - and the evolving Prakrit that grew after the Vedas increased almost imperceptibly, until a couple of centuries before 600 BC or so, grammarians had written extensively on the differences, specifically differences in the tenses used.
By then, the centre of gravity for the culture of the immigrants had shifted from the north-west, to the heart of the Punjab, to the Delhi region, and progressively into the Gangetic Doab. Essentially the grammar codification you refer to was needed to create some order due to the differences between the Rg Vedic (and almost Avestan) language and the language that was being spoken, early forms of Prakrit.
Panini's codification of the Indo-Aryan language eliminated the 'alien' forms, and codified the language strictly, making it clear which forms and usages were 'legitimate', without quite declaring the others 'illegitimate'. This codification is the reason why there is a gap between the sacerdotal language that Doc has been showing us, and classical Sanskrit; Sanskrit was specifically streamlined to eliminate the similarities with Avestan, the sacred language of the Iranians, a variation of the broader group spoken by the eastern Iranians, the Sakas, known in the Mahabharata as the Parama Kamboja, a valiant tribe of horsemen from the extreme edges of consciousness of the immigrants of the Punjab and upper Doab. Note that this occurrence in the Mahabharata is just before the appearance of the codified Sanskrit, as the Mahabharata was probably last 'enhanced' around 800 BC or so.
So simultaneously, there existed the descended language, Prakrit, and the cleaned up language, Sanskrit. It is important to know that Sanskrit was before then, 600 BC, a cleaned up, codified version of the archaic language of the Vedas, while the language that was spoken in those areas that the immigrants' language had prevailed over the older, pre-existing ones (that is, between the Indus and the mountains to the west to the Rajgir mountains to the east, and between the Himalayas to the north and Vindhya mountains to the south) was Prakrit.
Today's north Indian languages are descendants essentially of Sauraseni Prakrit, from which is derived most of the languages of the west of India, including Punjabi, Marathi, Gujarati, Konkani, or of the more glamorous Magadhi Prakrit, the cultural centre of gravity having shifted to Magadha, that gave rise to a number of languages in the east.
Hindi doesn't figure because Hindi was an artificial creation of the 18th and 19th centuries, with the weaponisation of language by the Hindus of the UP as a reaction to their resentment at a perceived domination. Effectively, these cultural guerrillas were rejecting the earlier lingua franca that everyone in the north spoke, Hindustani, that was often written in Urdu script. Hindi rejected those parts of Hindustani that were 'imported' from Persian, Turkish, Arabic, and created words from Sanskrit to replace some of them, and also rejected the Urdu script in favour of Nagari, soon to become Devnagari. So, ironically, we have a repeat of the north Indian rejection of the Persian, first, due to the codification of Sanskrit excluding Avestan and its secular cognates, second, due to the promotion after codification of Hindustani, excluding Farsi.
Hindi was an adaptation of one of the
eight seven dialects that prevailed in today's UP, the heart of the Gangetic Doab, Khadiboli, that describes the language spoken in the extreme west, including some part of Haryana. There were six others, Braj Bhasha, Bundeli, Kanauji, Awadhi, Bagheli and Bhojpuri, besides three others from Bihar, not including Maithil, and the four or five dialects of Rajasthan.
