Who was exempt from the penal code in the scarlet letter?

The first thing that comes to mind is men (in particular the dude that knocked up Hester). They can impregnate the women but only the woman gets punished. It has been a good 12 years since I read this so I could be completely wrong.