Should We Judge Infidelity By Who Cheats?

We treat infidelity differently depending upon the gender of the cheater. How can we hold a uniform idea of justice across genders then?