Like any society there’s going to be someone on top and someone on the bottom. The Highborne who went into exile might have taken X Y Z amount of household staff with them - or over the X amount of time since their arrival on the EK and the establishing of the High Kingdom some houses/families either outshone the others or died. It is mentioned, IIRC (it’s been a long time since I’ve played Belves) that they suffered losses in the cold northern traversal before they settled. Maybe some of the nobility died, and all that was left to fill the void were household staff. Maybe those staff were absorbed. Then instead of (I’m simplifying here) 5 houses with 10 staff each, they’re left with 4 houses with 12 staff each. Etc. etc.
btw Drathir did nothing wrong