I guarantee you there would be people doing that. If it’s possible to vote as a group to leave, how many would you require, 4? What if it’s two people doing something like that to try and mess with people? There could even be 4 people doing it to trap just one other person in there with them. In that kind of scenario, reporting would be completely useless as one report doesn’t do anything.
You could also have people who want to leave because of something, but don’t want the ban, just start ninjapulling stuff to make sure the group wipes another few times, to make people want to vote to leave as a group.
If it’s possible to abuse something, that is what is going to happen.