Great Elven kingdoms once spanned the world. Each with their own unique culture and history, the descendants of these kingdoms are the different Elven subraces. Their long lives allowed for deep study of magics, according to the needs of the kingdom. Magic became common and integrated deeply into their culture, the likes of which has never been seen before or since.
They traded magic for raw materials with the various Dwarven kingdoms under the mountains, and nearly drove Dragonkind (their only rivals in magic) and Orcs (whom they considered barbaric) to extinction. All other races, including humans at the time, were too few in number, too tribal, or too scattered to be anything more than exotic oddities and lesser beings.
And then almost overnight by the span of the Elves, the human population -which had been burgeoning but still mostly unnoticed by the Elvenkind- coagulated into their own kingdoms and revolted against the Elven powers with help from the remaining Dragonkind. None will ever know what the result of the wars would have been, because it was about that time the Apocalypse happened. 80% of life in the world was wiped out. It was an irrecoverable blow to the Elves, as they could never hope to recover their population faster than the other races, and their great cities and kingdoms were mostly lost or abandoned. Millenia later, they yearn for the old days as they fade toward irrelevance, watching the the world pass them by.
Even now however, on average, Elven magic remains only equaled by Draconic Sorcery.