Well, technically speaking you’re only “fighting” against the ones on your realm in the honor brackets, in only your faction.
But the reason so many people don’t care is because of the anonymous nature of the matchmaking. The more this anonymity is enforced through repeated exposure to it, the less people will care as time moves on. The same effect could be seen in retail as well.
It turns people apathetic over time.