You are right. It doesn’t make much sense when put like that.
Thankfully, this question has been answered in a recent interview.
Question: “If they come from a small group of Silvermoon Exiles, how are they an Allied Race? Where do the numbers come from?”
Moorgard answer:“They start out as a small group,” Danuser continues. “But it’s natural as other elves have found out about them — elves who are interested in new sources of magic, power, or opportunities — would see if they could undergo a similar process. They’re not … recruiting, necessarily, but they’re open to those who show a similar interest.”
Not to mention there are High Elves accompanying Void Elves in Stormwind.
And the High AND Blood Elves present in Tel’rogus learning to wield the void.
And the Voidcrafted Ravasaurs which the Void Elves create during the Alliance invasion which shows that they can infuse other living beings with the void. Please note this is not the Undead dinosaur they later reanimate, but living beings they corrupt in the name of the Alliance.
So, Void Elves can convert other Elves into Void Elves which is why their numbers have expanded.