This goes vise versa. Yes one could do that. However not everyone is thinking about that in his daily life. And judging by looks is actually part of our nature. Also people could not get upset if they are by accident misgendered if the person afterwards is doing it correctly. If the person however utterly refuses for whatever reason then ye one can complain (or ignore the person since its not worth ones time)